Chromium Embedded continued

Previously I got the Chromium Embedded (CEF) sample application to compile (and run). That was only a tiny step towards what I needed to do.

Suneido has a DLL interface, but it can only handle straightforward interfaces. Although CEF has a DLL with a C interface, it's fairly complicated. So I decided I needed to write a small DLL that Suneido could call that would talk to CEF.

But I wasn't sure what the minimum functionality I'd need. The sample application is actually quite large and complex because it is demonstrating a bunch of features like extensions and plugins. So I started by whittling down the sample to the bare minimum. That process helped me identify the core functions I'd need.

It took me a certain amount of struggling to recall / figure out how to build a Windows DLL. (It's not something I do very often.) Eventually I got to the point where Suneido could see the DLL functions. But calling them didn't work (it gave some weird stack errors). Now what?

Then I remembered the issues with building single-threaded versus multi-threaded. The main version of Suneido we use is built single-threaded (/ML), but CEF is built multi-threaded (/MT). I tried it with a multi-threaded build of Suneido and it worked! (Suneido is still single-threaded, it's just linked with the multi-thread runtime libraries.) Here is Chromium Embedded running in a Suneido window:

Coincidentally, the single/multi-thread version issue has also come up in another area. I'm looking at upgrading my work computer to Windows 7. But before I take the plunge I want to make sure the software I need is going to run under Windows 7. One of the "oldest" packages I use is Visual C++ 2003 (VC7). I would have moved to a more recent version long ago, but VC7 still generates the smallest, fastest executable. I think the main reason for this, is that VC7 was the last version that supported building with the single-threaded libraries. (/ML) I'd prefer to use GCC but it doesn't handle the interface with the Internet Explorer browser component. (It's also a bit slower, but not too bad.) So using CEF instead of IE might also allow switching to GCC.

Updating ASM

jSuneido uses ASM to generate Java byte code. I was using 3.1 and the current version is 3.3

I wasn't sure if this would be easy or hard. It turned out to be somewhere in between.

First I tried to update my Eclipse plugin. That was a little confusing because I had three update sites showing ASM - eclipse.org, asm.ow2.org, and andrei.gmxhome.de. I'm guessing the eclipse.org one is something internal to Eclipse. I thought asm.ow2.org would be the correct location, but it didn't have the latest version.

Next I downloaded asm-3.3.jar but that didn't work, I needed to download asm-3.3.zip and extract asm-all-3.3.jar  This wasn't entirely a surprise because previously I was using multiple asm jar files. I'm not quite sure why the asm-3.3.jar is available as a separate download, but not the asm-all-3.3.jar

Of course, tests failed. The first message was:


java.lang.IndexOutOfBoundsException: Trying to access an inexistant local variable 0
at org.objectweb.asm.tree.analysis.Frame.setLocal(Unknown Source)
at org.objectweb.asm.tree.analysis.Analyzer.analyze(Unknown Source)
at org.objectweb.asm.util.CheckMethodAdapter$1.visitEnd(Unknown Source)
at org.objectweb.asm.util.CheckMethodAdapter.visitEnd(Unknown Source)
at org.objectweb.asm.tree.MethodNode.accept(Unknown Source)
at org.objectweb.asm.commons.TryCatchBlockSorter.visitEnd(Unknown Source)
at org.objectweb.asm.MethodAdapter.visitEnd(Unknown Source)

[I thought "inexistant" was a typo, but it appears to be the French version of nonexistent.]

I searched for other people having this problem and found:

When visitMaxs returns an ArrayIndexOutOfBoundsException, more information would be nice

and

ClassWriter.COMPUTE_FRAMES causes java.lang.ArrayIndexOutOfBoundsException

I didn't resolve this. As a workaround I use CheckClass(visitor, false) to turn off the data flow checks. I am probably doing something wrong, but the byte code passes Java's verification and it runs ok, so presumably it's nothing too serious? 

The next problem was that you now have to call visitTryCatchBlock before defining any of it's labels. At first I thought that was easy to fix, just move the calls from the end of the try-catch block to the beginning. But one test failed - with nested try-catch blocks. The problem is that the JVM searches the exception table in order. So inner blocks must come before outer ones. By emitting at the end of the blocks I got the right order. But now that I was forced to emit at the beginning of the blocks, the order was wrong. 

This stumped me for a while. The problem is that jSuneido (and cSuneido) compiles in a single pass, generating code as it parses. This avoids building a syntax tree. But it doesn't give much flexibility in the order of generating code. I was having scary thoughts of having to make two passes, or switching over to building an intermediate tree. Of course, I could just go back to ASM 3.1, but in the long run I'd probably need to upgrade (e.g. for invoke dynamic support).

Again I searched for other people having this problem and didn't find anything. I was going to post on the ASM Tracker, but before I did I figured I'd better make sure there wasn't an existing post. As it turned out there was: Provide exception table sorting functionality (I'm not sure why my Google searches didn't find it.) Not only had someone else had the same problem, but they'd come up with a solution, and it was now included in ASM. All I had to do was add TryCatchBlockSorter. It's actually quite simple, once you know that the table is accumulated in a way that allows it to be sorted.

Still, I'm not sure why they added this requirement. It seemed to work fine without it in the previous version.

After that, I just had to fix a few tests to take into account the slight differences in the generated try-catch code and I was back in business.

Chromium Embedded

Our Suneido applications use the Internet Explorer browser component included in Windows.

This works pretty well, but we are running into a few problems:
- people have a wide variety of IE versions
- some people have Javascript turned off
- security settings are getting more and more restrictive

And in the long run I'd like to run on Mac and Linux which don't have IE built in for some reason :-)

So I've been thinking about using another browser component that we can have better control over. Initially I was looking at Gecko (the Mozilla Firefox engine). There's even a Mozilla ActiveX control with the same interface as IE, but it requires registry changes which isn't always feasible for our clients. I tried to find a workaround for the registry issue but couldn't get it working (and no one on Stack Overflow had an answer).

The recent trend seems to be to use WebKit - like in Safari, Chrome, Google Earth, etc. I found the Chromium Embedded project that seems like it would do the trick.

As a first step, I thought I should try to build the example application.Four hours later I think I have achieved this. Since it wasn't easy, I thought I'd recount the saga in case it might help someone else.

First, the standard build environment seems to be Visual Studio C++ 2005. I had various other version installed but not that one. Unfortunately, it's been replaced by VS 2008 which I tried building with, but it gave a lot of errors. (Although maybe the fixes below would have worked?)

A Stack Overflow post led me to someone's page with links to old versions. When I installed, I got compatibility warnings which suggested some updates. I downloaded the updates but when I tried to install it failed, saying the software wasn't installed. (Even though I could run it.) Instead, I was able to get the updates by running Windows Update.

When I tried to build it couldn't find atlbase.h  That's because Visual Studio Express (the free version) doesn't come with the Windows SDK. I already had a Windows Server 2003 SDK installed so I updated Tools - Options to point to it.

But I still got errors about shlwapi.lib missing. It turns out this is part of "Microsoft Web Workshop (IE) SDK", an optional part of the SDK. So I tracked down the right SDK installer and installed it.

Next, I had to patch atlbase.h and atlwin.h as explained in a Code Project post. This cleaned up a bunch more errors. But I was still getting errors about _Module which I fixed based on yet another post.

Then I got errors that I was missing opengl32.lib which stumped me for a while till I found someone else with a similar problem, and ended up at instructions for adding the SDK libraries to Visual Studio. Since you don't actually need opengl32.lib, I'm not sure why the project specifies it.

Initially the Debug version built. It ran, but it had a bad tendency to crash. The Release version seems to be fine.

Another issue is that it's big, about 20mb for the required dll's. I guess that's not that big these days, but it's still a fair size to deploy over the internet.

This all seems way harder than it should be. I realize it's Windows and Visual Studio and C++ and open source, but still! Maybe I've been spoiled by working with Java lately. Java does have benefits.

Azul's Better GC

InfoQ: Azul Systems To Open Source Significant Technology in Managed Runtime Initiative

Interesting. The more cores you have, the more garbage collection (GC) you need - a result of the shared memory model. So even if you avoid killing your multi-threaded performance with locking overhead, you still have scaling issues with GC.

This may also put pressure on Java to do more to avoid heap allocation (e.g. escape analysis and fixnums).

Responsive Web Design

A List Apart: Articles: Responsive Web Design

Nice techniques for handling todays huge variation in screen size - from mobile phones to 30 inch monitors.

Email Lives

Hello Lego World

Very cool!



via John Nack

Phone Companies Antics

I had a phone call from Rogers yesterday trying to sell me a "Rocket Stick". I asked if I could use my existing data plan. "Oh, do you have a data plan?" they replied.

I wish these companies would get their act together and use the information they have. They say they're calling me because I'm a "valued customer" but they don't even have my account information. Of course, it's probably a different department or outsourced or one of a dozen other excuses. But it makes them look stupid. One of the first rules of selling is to know your customer.

So they asked me what my plan was ($30 per month for 6gb) and told me "Yes, you can switch to a shared plan". I was pleasantly surprised because I assumed the answer would be "no".

The new plan would be $45 per month for 1gb. Wait a minute, you want me to pay 50% more for 1/6 the service? "Yes, but it would be shared."  No thanks, doesn't sound like a good deal to me. (I don't use 6gb and 1gb would probably be fine, but I still don't want to pay more for less! And with 6gb I don't have to worry about my usage, which is a big plus.)

Actually, what I'm really interested in is the Mobile MiFi Hotspot. Then I could use it with my laptop and Shelley would use it with her iPod Touch or laptop at the same time. (Plus any future wifi devices.)  I'd looked this up before, but it just said "Out of Stock". I asked about it and they told me they used to have them, but there was some problem with the batteries. That sounded a little lame to me - they're still selling them in the US.

Note: In theory I can share my data plan by tethering my MacBook to my iPhone but it hasn't worked too well when I tried and I haven't spent the time to figure it out. And I don't think that would help with other devices.

see also: AT&T’s cynical act

Impossible Things

"Why, sometimes I've believed as many as six impossible things before breakfast." Alice in Wonderland

I was testing jSuneido on a Windows system and I got:

java.io.IOException
        at java.nio.MappedByteBuffer.force0(Native Method)
        at java.nio.MappedByteBuffer.force(MappedByteBuffer.java:154)

So I went to add a try-catch to log this error. But Eclipse told me MappedByteBuffer.force doesn't throw IOException

I check the documentation but it didn't mention IOException either.

This is a "checked" exception, meaning you can't throw it without declaring that you throw it, and you can't call something that may throw it without catching it.

But I guess all that goes out the window when you get down to native methods.

I just caught the more general base Exception instead.

PS. I wasn't able to recreate the error, I just hope it won't come back to haunt me.

Apple Overtakes Microsoft in Market Value

Apple Overtakes Microsoft in Market Value: End of an Era?: "At the close of Wednesday’s trading, Apple was valued at $222 billion, while Microsoft was worth $219 billion. Apple’s shares ended the day at $244.11, while Microsoft’s finished at a seven-month low of $25.01. And it isn’t only Cupertino’s successes, but also Redmond’s failures that are responsible for the new power dynamic between the two companies. Overall, Microsoft stock is down 20 percent compared to 10 years ago, while the value of Apple’s has grown tenfold over the same period."

Motivation

A great animated video for a TED talk by Daniel Pink related to his recent book Drive.

Third Time is the Charm

I decided that for what I needed Netty (or Mina) was overkill.

So I rewrote my server code using thread-per-connection and blocking io. I still used NIO Channels because I wanted to be able to use gathering writes (and because it was closer to the previous code).

And I still had the same problem!

That tells me it probably wasn't a mistake in my NIO non-blocking Selector version.

And the same fix "worked" for this version. (synchronizing the SocketChannel write)

That seems to indicate the problem is really in SocketChannel.write (or ByteBuffer)

I posted a question on the network forum on Sun (or should I say Oracle) but so far no response.

So I rewrote the server code a third time, using just old style java.net Socket, no NIO, no channels.

This time it seems to work!

On the positive side, the blocking, thread-per-connection versions (with channels or sockets) seem to be about 15% faster than the non-blocking Selector version.

Assuming I don't find any problems, and depending on what (if any) response I get to the forum question, I'll probably go ahead with this version.

Not Making Sense

Two more days working on my concurrency bug. I'm growing less and less confident that I'll "figure it out". The more I narrow it down, the less frequently it happens, which makes it more and more difficult (and time consuming) to know whether I've "fixed" it.

I'm really reluctant to say it, but I'm starting to wonder if there's a bug in Java. I have it narrowed down to where I check the data on the server immediately before writing it and it's always ok. But every so often the client will receive garbage. (Which usually leads to a timeout because it's waiting for a newline.) I'm pretty sure it's not a problem on the client side because it only happens when the server is multi-threaded.

If this was C or C++ I'd assume I had a bad pointer and was overwriting memory or that I was referencing memory that was prematurely free'd. But that's not supposed to be possible in Java.

There is no concurrent access in my code to the data that is being sent. I don't reuse the buffer or anything like that. I've tried using a read-only buffer, and I've also tried allocating a fresh buffer every time.

I have found a "fix". If I synchronize the channel.write on a global object, so only one thread can write at a time, then it seems to work. At least I've run it for about 10 times as long as it usually takes to get a failure.

But this "fix" makes no sense. First, channel.write is supposed to be thread safe. Second, concurrent writes (when they happen) are to separate channels, channels are not shared by multiple threads.

It could be a concurrency bug in Java or the OS, but why would I be the only one to encounter it?

Of course, this change may not be directly related to the problem. It may just be altering the timing enough to make the bug not happen, or to happen so rarely I'm not seeing it.

I've gone back and re-read all the NIO material I can find on the web. Despite a certain amount of conflicting advice I think I have a pretty good idea how it works, how to use it, and the common errors. I even read the arguments against using NIO (which I'm sympathetic to, given my problems).

Although it seems to be working, I'm not comfortable leaving it as it is. Since I don't know the source of the problem, or why/how the "fix" works, the bug could reappear at any time. But I'm also running out of things to try. It's narrowed down to only a few hundred lines of fairly straightforward code, most of which is identical to examples found on the web.

So my options are to abandon NIO and use a simple thread per connection model. According to some reports, this would be faster for the number of connections I expect (up to several hundred).

Or I could use someone else's framework. The contenders seem to be Apache Mina and JBoss Netty. (see Netty vs Mina)

I think I'll probably give Netty a try and see how it goes. But it's frustrating not being able to figure it out - I hate "giving up"!

Be Precise

Guest Post by
Penny McKinlay

I’m sure all of us have used the Help feature on a software program and been frustrated and annoyed because it didn’t make sense or didn’t answer our question.

One of my goals as a technical writer is to develop a good relationship with the customer. I want them to turn to the documentation feeling confident that it will help them complete their task quickly and successfully.

I don’t want to confuse or antagonize them.

Avoid Confusion

I prepare the online documentation for customers of an integrated trucking and accounting software package. I used to use the verb “select” when telling customers to pick the appropriate option on a pull-down menu.

Fortunately, one of the trainers pointed out to me that this was really confusing for the customers because there was a Select button at the bottom of many of the screens. Now, I only use the word “select” if I am referring to this button.

The Help carefully distinguishes between “placing a checkmark in the checkbox” and “generating a check.” And we deliberately use the American spelling of “check” so as not to irritate or confuse American customers.

We consistently refer to “Owner Operators” rather than switching back and forth between “Owner Operators,” “Leased Carriers,” and “Third-Party Contractors.”

Don’t Annoy the Customer

I try and write the Help from the user’s perspective: “you will not be able to invoice your Order until you do this” or “use this report to review outstanding trucking transactions.”

We used to say that the software would “allow” customers to do certain things, but we realized that the word “allow” was really condescending. Now, we use phrases such as “this screen will help you to . . . .”

We also try very hard to avoid jargon because it’s intimidating and makes readers feel stupid. Software programmers talk about “parameters” and “locking down.” The Help talks about “options” and “protecting.”

Writing Tips

Be precise, be consistent, consider your audience – good things to keep in mind whatever we’re writing.

For more tips on business writing, visit my web site.

Penny McKinlay
pennymckinlay.axonsoft.com
www.wanderlustandwords.blogspot.com
twitter.com/PennyMcKinlay

enTourage eDGe™

enTourage eDGe™

Another interesting alternative to the iPad or Kindle. I'm not sure it's ready for prime time though - it sounds like it's slow, heavy, limited battery life, and no apps. Other than that, it looks great :-)

I'd really like to go for something more "open" than the iPad or Kindle, but the problem is the apps and books. No one has all the apps like Apple, or the books like Amazon.

If we can move toward HTML 5 apps and get away from DRM, then we'll have a lot more freedom with hardware.

Which still leaves the question whether anyone can match Apple in terms of design and user experience. Even people who promote Android admit that it's not as polished as Apple. The software might get there eventually - Linux (e.g. the latest Ubuntu) is starting to rival Windows and OS X. But Apple's hardware design and polish is tough to match.

iPhone / Touch Apps

I gave my sister Penny an iPod Touch for her birthday and figured I'd better give her some recommendations for apps. Here's some of what I have on my iPhone:

Evernote (free) - save notes, photos, documents, works with web, Windows, and Mac versions
BlogPress ($2.99) - blog post writing tool, you can write offline
Note Taker ($1.99) - hand write notes with your finger (free Lite version)
Kindle (free) - ebook reader, works with Kindle devices or Windows and Mac software
Tripit (free) - trip itinerary, syncs with web site
iKeePass ($0.99) - password manager, works with web, Windows, and Mac versions
Remote (free) - remote for iTunes on Mac computers
Facebook (free)
Echofon for Twitter
Encyclopedia ($8.99) - offline copy of Wikipedia
iBird (free to $29.99) - bird guide (search for iBird)
GoodReader ($0.99) - PDF reader (free Lite version)
Instapaper (free) - save web pages for later offline reading (Pro version $4.99)
Outliner ($4.99) - syncs with web site
Google Earth (free)
oMaps ($1.99) - open maps
Fugawi iMap ($4.99) - topo maps of Canada and USA, can download maps for offline
Copilot Live Directions (free) - offline street maps of Canada and USA
Stanza (free) - ebook reader
Eucalyptus ($9.99) - ebook reader, access to 20,000 free books
Dropbox (free) - works with Windows and Mac software and web site
Gnu Go (free)

Perseverance

Another day working on my bug but at least I've made some progress, at least as far as narrowing it down.

I managed to simplify my test to just repeating the same request over and over (instead of the semi-realistic mix of operations in the original test) This greatly reduces the amount of code where the bug could be.

Next I wrote a test client in Java to do that same request. But it succeeded. (Of course, you never know if one more run would run into the bug.)

Doing more testing with the cSuneido client I realized that the bug occurred more often (or only) when the client and the server were on separate machines. So I tried running my Java client on a separate machine. And now it failed too.

So now I know the problem isn't with cSuneido, or Parallels, or Windows.

I narrowed it down even further by commenting out all the real work of the request (the database querying stuff) leaving only the framework of request handling. It still fails. This reduces the amount of code where the bug could be even more.

But I'm still puzzled. My simple test client plus test server has yet to fail, even running on separate machines. But the test client plus the real server does fail, even though the real server is cut down to where it's doing little more than the test server. There must be some difference. Or else it's just that the probability of the error is less and I haven't run it enough times.

Just to give an idea, when it's failing, it fails about once every million requests. Gotta love that kind of bug!

So I still haven't found the problem, but I might be getting closer.

This is what I was afraid of!

I just spent another day trying to find my jSuneido concurrency bug, with no concrete progress. That makes four days I think, definitely the trickiest bug on this project (so far). It's this kind of bug that makes me afraid of concurrency in the first place.

I made a test server and test client and at first I thought I had recreated the problem, but the fix I found only worked for the test program, not jSuneido :-( Since then my test server and client have worked perfectly, despite extending them to be more like the real jSuneido code.

I've tried synchronizing here and there and making defensive copies of this and that but the bug still happens. I've read and re-read the code. I've drawn diagrams of the sequence of events.

I start to wonder if the bug is in the cSuneido client rather than the jSuneido server. But that doesn't really make sense because we have thousands of people using the cSuneido client and we don't see this problem.

My next step is to write a simple Java client for the jSuneido server. That will eliminate the cSuneido client. I think the best bet is to try to gradually narrow down the problem and isolate it.

I don't doubt I'll find the problem eventually. The question is how many days I'll need to bang my head against the wall. Hopefully the wall crumbles before my head!

People Hate Change

My company sells specialized software for trucking companies. Often companies that we sell to have been using QuickBooks. And the people in those companies are very reluctant to leave QuickBooks and switch to using our software. And even after they do switch they continue to complain that our software does not work like QuickBooks.

QuickBooks is a good program and no question there are QuickBook features that are our software lacks. But for trucking companies our software does a lot more than QuickBooks ever will. QuickBooks doesn't handle the specialized needs of trucking companies, our software does. But to the people being faced with change, none of that matters, they want the familiar.

This was brought home recently by a long time customer of ours. They're not a trucking company and the software they have is very old. (It's DOS text mode software!) Recently they wanted to set up another company and wanted another copy of this ancient software. We suggested they just use QuickBooks. They came back and said they hated QuickBooks and that their current software is much better! This is so obviously not true that it was funny. It was a good reminder of how strongly biased people are to what they are used to.

jSuneido Concurrency Bug

I spent all day yesterday working on my concurrency bug in jSuneido. (see previous post)

It didn't seem like I made any progress, but I guess that's not true - I eliminated a few possibilities.

Originally I assumed it was a deadlock since it was "freezing". So I managed to make the problem happen (that takes a while), connected JConsole, and ran the deadlock detector. It didn't find any deadlocks. Thinking about it, that makes sense since it's really only the client that gets "stuck" - the server thinks everything is fine. There are no worker threads running for that client.

So much for that assumption. I tried to narrow it down, without much success. I added a bunch of assertions, which failed, but I found it was my assumptions that were wrong, not the code.

The client appears to be waiting for a response to a request. But the server is not in the process of handling a request - it either didn't get the request, or it thinks it finished it. The read and write buffers for that client are empty.

Sometimes you have to run test for a long time before the problem shows up. So you try something, and the longer the tests run successfully, the more you think you've got it fixed. But eventually it has always shown up. I guess I should be thankfully this is on the order of minutes rather than hours! Even so, it's reminiscent of the "old" days when you had to have something to read while you waited for compiles. (And no internet to browse back then.)

On the positive side, the handling for inactive transactions and clients seems to be working well - when the client gets stuck, the server eventually aborts its transactions and if it stays stuck, closes the connection.

My current hypothesis is that it's related to a potentially overlap between handling requests. As an optimization, worker threads first attempt to directly write their response. This is in non-blocking mode so it may or may not write everything. Any remaining data is sent by the select thread when the socket becomes write ready. But if the worker does successfully write the entire response, and then there is a context switch back to the "select" thread, the next request can be read and another worker started, before the previous worker has a chance to finish. I'm not sure why this would be a problem, but my assumption when I was writing the code was that workers for the same client would not overlap, so I didn't worry about making the data structures thread safe. That's the next area I want to look at.