JVM Languages Summit Day 2

Another good day of talks starting with Doug Lea on parallel recursive decomposition using fork-join. Amazing how much subtle tweaking is require to get good performance.

This led into Joshua Bloch's talk on performance. There is so much complexity in all the layers from CPU to OS to language that performance varies a lot. He showed a simple example of a Java program that gave consistent times on multiple runs within a given JVM, but sometimes when you restarted the JVM it would consistently give quite different results! Cliff Click's theory was that it was caused by non-deterministic behavior of the JIT compiler since it runs concurrently. The behavior is still "correct", it can just settle into different states. The solution? Run tests over multiple (eg. 40) different JVM instances. That's on any given machine, of course you should also test on different CPU's and different numbers of cores. Easy for them to say. 

Neal Gafter talked about Microsoft's LINQ technology - pretty cool, although nothing to do with the JVM. 

Kresten Thorup talked about his Erlang implementation on the JVM using Kilim for coroutines. Erlang is an interesting language, and quite different from Java so it was interesting to see how he implemented it. He actually runs the byte code produced by the existing Erlang compiler. 

I talked to Remi Forax about whether I should use his JSR292 backport in jSuneido. This would let me use the new technology before Java 7 is released (who knows when). Of course, he said I should. But ... it means developing with the "beta" JDK 7 which still has bugs and is not supported by IDE's. And then it requires an extra run-time agent. I'm not sure I want to complicate my life that much!

JVM Languages Summit Day 1

My hotel is "behind" the Sun/Oracle campus so I had to circle around through the endless acres of parking lots to get to the right entrance. But it was pretty easy, and not as far as it looked on the map. I've never had much to do with giant organizations so it's still a little mind boggling when you use an automated kiosk (like the ones at the airport) to get your visitors badge.  


The sessions were a real mixture, from stuff where the nuances were pretty much over my head, to thinly veiled sales pitches with no technical details. It was pretty neat to be in the company of people you're used to thinking of as gurus, like Doug Lea, John Rose, Joshua Bloch, and Cliff Click. And nice to see they have their frustrations with the technology also.


It's naive of me, but unconsciously I expect really smart people to be "rational", and it's always a bit of a disappointment when that proves to be untrue. Smart people have egos, are insecure, or argumentative, or negative, or defensive, or obnoxious, just like everyone else. 


Maybe I'm just too soft, but I felt bad for one guy who basically got told he was doing it wrong. It seems like they could just have well asked "did you consider ..." or "what would you think about ...", rather than just "that's wrong, you should have done ..."


Mostly I just listened to the conversations. In the Java / JVM area I still feel like a relative novice and I'm not sure I have much to contribute yet. But for the most part I didn't feel too much out of my depth, so that's good.

OSCON Wrapup

The last two days of OSCON were good.

I got to hear Rob Pike talk about the Go language. Rob is a legend in software and Go is a cool language. In some ways Go is more like C or C++ in that it's compiled to machine code with no VM. But unlike C++ it compiles extremely fast. For what he called a party trick he showed it compiling every time he typed a character - and it kept up. It has features I miss in Java like pointers (but safe) and values on the stack (not always allocated). It also has "goroutines" - lightweight threads like coroutines. But its attractions aren't quite sufficient to tempt me away from Java. They don't even have a Windows version yet, let alone all the support libraries and frameworks that Java has.

I also got to hear another legend, Walter Bright, talk about his D language. I used Walter's C and C++ compilers for many years. D also has some very interesting features. Andrei Alexandrescu of Modern C++ Design fame is now working on D and has written The D Programming Language book.

One feature D has that is sorely missing from other languages is the ability to declare functions as "pure" (ie. no side effects) and have the compiler verify it. This (not lambdas) is the key to functional programming. And yet languages like Scala that claim to support functional programming don't have this.

I also went to a talk by Tim Bray whose blog I read. He was working at Sun but moved to Google after the Oracle buyout. His talk was on concurrency. He was very pro Erlang and Clojure but didn't mention Scala. When asked about it he said he thought the Scala language was too big. It does have a sophisticated type system, but the actual syntax is quite simple - smaller than Java. Scala's Actor implementation has been criticized but it's been improved in 2.8 and there are alternatives like Akka.

OSCON Another Day

After the less than thrilling key notes, I went to another Scala session. A lot of it was repetition, but I learn a bit more at each one. The recurring theme seems to be if you're going to use Scala, use Simple Build Tool (SBT).

Next I went to a talk on GPARS, a Groovy concurrency library. The talk was as much about concurrent programming patterns as about Groovy, which suited me.

Next, I was going to go to one of the database talks, but then I realized it was a vendor presentation and they're usually more sales spiel than technical. Looking for an alternative I noticed Robert (R0ml) Lefkowitz had a talk on Competition versus Collaboration. If you've never listened to one of his talks, it's well worth it. They are as much performances as presentations and always thought provoking. (search for Lefkowitz on the Conversations Network)

Next was a talk on Clojure (a Lisp that runs on the JVM). The title was "Practical Clojure Programming" which sounds like an intro, but it was actually about build tools, IDE's, and deployment. Like most of the audience, I would have rather heard more about Clojure itself. I guess we should have read the description closer.

Finally, I snuck into the Emerging Languages Camp to hear Charles Nutter talk about Myrah (formerly Duby) his close-to-the-JVM-but-like-Ruby language. I would have been tempted to go to the Emerging Languages Camp (it was even free) but by the time they announced it, I'd already signed up for the regular sessions.

All in all, a pretty good day. Lots of food for thought, which is, of course, the point.

One thing I forgot to mention yesterday is that a lot of the Scala people are also (or ex-) Ruby/Rails people.  Maybe that's simply because they're the kind of people that like to learn/adopt new things.

But a lot of people left Java and went to Ruby, so it's surprising to see them coming almost full circle back to Scala. Scala is better than Java, but it's still a statically typed language like Java, which is part of what people seemed to reject. Maybe Ruby wasn't the silver bullet they were hoping. Maybe it was performance issues. Maybe they realized that static typing does have advantages after all. Maybe they realized the advantages of running on the JVM (although jRuby allows that). Maybe Scala's improvements over Java are enough to win people back.

OSCON Part One - Scala

I just finished the first day and a half of OSCON - in the Scala Summit. And contrary to my anti-social nature, I even went to a Scala meetup last night.

As I've written before, I'm quite intrigued with the Scala programming language. It runs on the JVM and interfaces well with Java. It combines functional and object-oriented programming. It has actors for concurrency. And it has a rich type system that is probably Turing complete, like C++ templates. 

Scala seems to be attracting a lot of attention. Of course, the Scala Summit attendees are all keen, but there were also several other language developers (like Charles Nutter of jRuby) interested in borrowing ideas from Scala.

Programming in Java is like driving a truck. It's not fast or fuel efficient or fun to drive or comfortable. But it gets the job done without a lot of problems and it can haul just about anything. 

Why do people like to drive a fast car? It's not like they regularly need go from 0 to 60 in 5 seconds, or hit 200 km/hr. But they like to know that if they "need" to, they could.

Similarly, I think people like to know their language "has a lot of power under the hood", even if they never use it. And it gives lots of potential for books and conference talks that keep people excited.

And like you could use C++ simply as a better C, you can use Scala as a better Java. This lets less sophisticated users get involved as well. 

The funny part is that it's very reminiscent of C++.  There were all kinds of talks and articles and books about all the cool stuff you could do with templates and other fancy C++ features. Now, all you hear are negative comments about C++, too complicated, too hard to use, etc. I start to think I'm the only person that liked C++. 

But there are differences too. Scala didn't try to be backwards compatible with Java the way C++ was backwards compatible with C. And Java already had object-oriented programming, so Scala isn't as big a jump as C++ (in that respect). 

If nothing else, it's made me really want to spend some more time with Scala. It will sound crazy, but I'm very tempted to rewrite jSuneido in Scala. Thankfully, that's something that could be done gradually. I think it would let me make the code much smaller and cleaner. I think it would also make it easier to switch back and forth with Suneido coding, since Scala has things like named and default arguments, optional semicolons, and type inference that make the code much more similar to Suneido code.

Time Capsules

I use an Apple Time Capsule for my home wireless router and backup storage.

Time Capsules have a bad reputation for dying and I've had mine for quite a few years so I was a little nervous about it. If it died I wouldn't have a backup of my iMac. Which would be ok unless it happened to die at the same time. This seems unlikely, but it's surprising how often you do get simultaneous failures. For example, a power surge due to lightning. I can also keep the external drive at work so I have an offsite backup in case the house burns down.

I had a 500gb external drive that I'd used for backups, but it's not big enough to do a complete backup of my 1tb iMac. So I went and bought a Lacie 2tb external drive and used SuperDuper to make a backup. I used the free version, but I'll probably get the paid version so I can update the backup without redoing it.

Then I decided I should also backup my MacBook and my MacMini. I didn't have much critical stuff on them, but a backup would save hassle if I needed to restore. But SuperDuper takes over the whole drive so how could I backup additional machines? The answer seemed to be to partition the drive, but I didn't want to have to redo my iMac backup (600gb takes a while to backup, even with Firewire 800). I searched on the web and found various complicated ways to resize partitions. Finally, I found that with recent OS X you can resize right from the Disk Utility. All the complicated instructions were for older versions of OS X.

The "funny" part of this story is that a few days later I went to use wireless and it wasn't working. I went and checked on the Time Capsule and it was turned off. Strange, because I leave it running all the time. I turned it on and about 10 seconds later it turned itself back off.

My computer was still fine, so I didn't actually need the external backup, but I was glad to have it nonetheless.

I phoned the local Apple dealer (Neural Net). The receptionist wanted me to bring it in and they would look at it in the next few days. I didn't want to be without internet for days so she let me speak to the technician. When I described the symptoms he said the power supply had died. But Apple doesn't let them repair them and doesn't supply any parts. Apple has been promoting their environmentally friendly products, but no matter how they're built, a non-repairable "disposable" product isn't very environmentally friendly.

I would have preferred to give Neural Net the business but they didn't have any Time Capsules in stock so I picked up a 2tb Time Capsule from Future Shop. 10 minutes after opening the box I was back up and running. (Although the initial Time Machine backups took considerably longer.)

One nice thing is that Time Machine backups seem to be a lot less intrusive. Before, when Time Machine kicked in it would really bog down my machine. If I was trying to work I'd often stop it. But now I don't even notice when it runs. I'm not sure how a new external device would change the load on my computer, but it's nice anyway.

My old Time Capsule ran quite hot, even when it wasn't doing anything. I was hoping the new ones would be better, but it seems much the same. I haven't measured it, but I assume heat means power consumption. I'm not sure why it can't go into a low power mode when it's not active. The other reason they run hot is that they have no ventilation or heat sink. Apparently there is an internal fan but all it does is stir the air around inside a small sealed box. You'd think they could come up with some better heat management without compromising their design. I would guess the heat is one of the big reasons they have a reputation for dying. Electronic components tend to have a shorter life at higher temperatures.

Rather than throwing out the old Time Capsule, I passed it on to one of the guys at work that tinkers with hardware. I thought he could at least extract the 1tb drive. But he managed to repair the power supply and is using the whole unit. I guess the hardest part was getting the case open! I'm glad it was saved from the landfill for a while longer.

Now that I have a bigger drive I thought I might as well backup Shelley's Windows machine as well. I have it set up with Mozy for on-line backups but just with the free 2gb account so I'm only backing up selected things. I'd seen Mozy will now backup to external drives so I thought I'd set this up. Unfortunately, it only backs up to directly attached drives (i.e. internal or USB) not to network drives. I'm not sure what the logic is behind that choice. I could use different software, but I think what I'll do (I haven't got round to it yet) is to use the old 500gb external drive.

iPhone Multi-tasking

A lot of people made a big thing about how the iPhone didn't have multi-tasking. But personally, I never saw any of the reasons as very compelling. Given limited cpu power and memory and a tiny screen, it made perfect sense to single task.

Sure, there are probably a few reasonable uses of multi-tasking. But there are way more potential abuses.

I look at my Windows machine, which is continually thrashing away when I'm not doing anything. Every piece of software I install wants to run stuff in the background. Just to check for updates once a week they want to have a process running constantly. That's not a benefit, that's abuse.

Two days after I installed iOS 4 with its long awaited multi-tasking, I went to use my iPhone in the morning and the battery was totally dead - it wouldn't even turn on. It had plenty of power the evening before. I can't be 100% sure of the culprit, but I assume it must have been some application left running by the new multi-tasking. I suspect it was a mapping app that I'd been playing with, probably running the GPS constantly or something like that.

On the other hand, according to Apple, most apps do not actually run in the background, they just get suspended and resumed. If that's really true in all cases, then I'm not sure what drained my battery overnight.

So now I find myself regularly "killing" all the active apps because I'm paranoid about this kind of scenario. Great, a "feature" has imposed a large manual overhead on me.

Hopefully the situation will improve once apps are updated to work better with the multi-tasking. But there will always be poorly behaving apps.

What I find strange is that Apple went from not allowing multi-tasking, to making it the default, with no way to switch it off. It seems like an option that you had to explicitly turn on (or at least some way to disable it) would have still silenced the critics, but wouldn't have imposed the cost on those of us that didn't want it.

Chromium Embedded continued

Previously I got the Chromium Embedded (CEF) sample application to compile (and run). That was only a tiny step towards what I needed to do.

Suneido has a DLL interface, but it can only handle straightforward interfaces. Although CEF has a DLL with a C interface, it's fairly complicated. So I decided I needed to write a small DLL that Suneido could call that would talk to CEF.

But I wasn't sure what the minimum functionality I'd need. The sample application is actually quite large and complex because it is demonstrating a bunch of features like extensions and plugins. So I started by whittling down the sample to the bare minimum. That process helped me identify the core functions I'd need.

It took me a certain amount of struggling to recall / figure out how to build a Windows DLL. (It's not something I do very often.) Eventually I got to the point where Suneido could see the DLL functions. But calling them didn't work (it gave some weird stack errors). Now what?

Then I remembered the issues with building single-threaded versus multi-threaded. The main version of Suneido we use is built single-threaded (/ML), but CEF is built multi-threaded (/MT). I tried it with a multi-threaded build of Suneido and it worked! (Suneido is still single-threaded, it's just linked with the multi-thread runtime libraries.) Here is Chromium Embedded running in a Suneido window:

Coincidentally, the single/multi-thread version issue has also come up in another area. I'm looking at upgrading my work computer to Windows 7. But before I take the plunge I want to make sure the software I need is going to run under Windows 7. One of the "oldest" packages I use is Visual C++ 2003 (VC7). I would have moved to a more recent version long ago, but VC7 still generates the smallest, fastest executable. I think the main reason for this, is that VC7 was the last version that supported building with the single-threaded libraries. (/ML) I'd prefer to use GCC but it doesn't handle the interface with the Internet Explorer browser component. (It's also a bit slower, but not too bad.) So using CEF instead of IE might also allow switching to GCC.

Updating ASM

jSuneido uses ASM to generate Java byte code. I was using 3.1 and the current version is 3.3

I wasn't sure if this would be easy or hard. It turned out to be somewhere in between.

First I tried to update my Eclipse plugin. That was a little confusing because I had three update sites showing ASM - eclipse.org, asm.ow2.org, and andrei.gmxhome.de. I'm guessing the eclipse.org one is something internal to Eclipse. I thought asm.ow2.org would be the correct location, but it didn't have the latest version.

Next I downloaded asm-3.3.jar but that didn't work, I needed to download asm-3.3.zip and extract asm-all-3.3.jar  This wasn't entirely a surprise because previously I was using multiple asm jar files. I'm not quite sure why the asm-3.3.jar is available as a separate download, but not the asm-all-3.3.jar

Of course, tests failed. The first message was:


java.lang.IndexOutOfBoundsException: Trying to access an inexistant local variable 0
at org.objectweb.asm.tree.analysis.Frame.setLocal(Unknown Source)
at org.objectweb.asm.tree.analysis.Analyzer.analyze(Unknown Source)
at org.objectweb.asm.util.CheckMethodAdapter$1.visitEnd(Unknown Source)
at org.objectweb.asm.util.CheckMethodAdapter.visitEnd(Unknown Source)
at org.objectweb.asm.tree.MethodNode.accept(Unknown Source)
at org.objectweb.asm.commons.TryCatchBlockSorter.visitEnd(Unknown Source)
at org.objectweb.asm.MethodAdapter.visitEnd(Unknown Source)

[I thought "inexistant" was a typo, but it appears to be the French version of nonexistent.]

I searched for other people having this problem and found:

When visitMaxs returns an ArrayIndexOutOfBoundsException, more information would be nice

and

ClassWriter.COMPUTE_FRAMES causes java.lang.ArrayIndexOutOfBoundsException

I didn't resolve this. As a workaround I use CheckClass(visitor, false) to turn off the data flow checks. I am probably doing something wrong, but the byte code passes Java's verification and it runs ok, so presumably it's nothing too serious? 

The next problem was that you now have to call visitTryCatchBlock before defining any of it's labels. At first I thought that was easy to fix, just move the calls from the end of the try-catch block to the beginning. But one test failed - with nested try-catch blocks. The problem is that the JVM searches the exception table in order. So inner blocks must come before outer ones. By emitting at the end of the blocks I got the right order. But now that I was forced to emit at the beginning of the blocks, the order was wrong. 

This stumped me for a while. The problem is that jSuneido (and cSuneido) compiles in a single pass, generating code as it parses. This avoids building a syntax tree. But it doesn't give much flexibility in the order of generating code. I was having scary thoughts of having to make two passes, or switching over to building an intermediate tree. Of course, I could just go back to ASM 3.1, but in the long run I'd probably need to upgrade (e.g. for invoke dynamic support).

Again I searched for other people having this problem and didn't find anything. I was going to post on the ASM Tracker, but before I did I figured I'd better make sure there wasn't an existing post. As it turned out there was: Provide exception table sorting functionality (I'm not sure why my Google searches didn't find it.) Not only had someone else had the same problem, but they'd come up with a solution, and it was now included in ASM. All I had to do was add TryCatchBlockSorter. It's actually quite simple, once you know that the table is accumulated in a way that allows it to be sorted.

Still, I'm not sure why they added this requirement. It seemed to work fine without it in the previous version.

After that, I just had to fix a few tests to take into account the slight differences in the generated try-catch code and I was back in business.

Chromium Embedded

Our Suneido applications use the Internet Explorer browser component included in Windows.

This works pretty well, but we are running into a few problems:
- people have a wide variety of IE versions
- some people have Javascript turned off
- security settings are getting more and more restrictive

And in the long run I'd like to run on Mac and Linux which don't have IE built in for some reason :-)

So I've been thinking about using another browser component that we can have better control over. Initially I was looking at Gecko (the Mozilla Firefox engine). There's even a Mozilla ActiveX control with the same interface as IE, but it requires registry changes which isn't always feasible for our clients. I tried to find a workaround for the registry issue but couldn't get it working (and no one on Stack Overflow had an answer).

The recent trend seems to be to use WebKit - like in Safari, Chrome, Google Earth, etc. I found the Chromium Embedded project that seems like it would do the trick.

As a first step, I thought I should try to build the example application.Four hours later I think I have achieved this. Since it wasn't easy, I thought I'd recount the saga in case it might help someone else.

First, the standard build environment seems to be Visual Studio C++ 2005. I had various other version installed but not that one. Unfortunately, it's been replaced by VS 2008 which I tried building with, but it gave a lot of errors. (Although maybe the fixes below would have worked?)

A Stack Overflow post led me to someone's page with links to old versions. When I installed, I got compatibility warnings which suggested some updates. I downloaded the updates but when I tried to install it failed, saying the software wasn't installed. (Even though I could run it.) Instead, I was able to get the updates by running Windows Update.

When I tried to build it couldn't find atlbase.h  That's because Visual Studio Express (the free version) doesn't come with the Windows SDK. I already had a Windows Server 2003 SDK installed so I updated Tools - Options to point to it.

But I still got errors about shlwapi.lib missing. It turns out this is part of "Microsoft Web Workshop (IE) SDK", an optional part of the SDK. So I tracked down the right SDK installer and installed it.

Next, I had to patch atlbase.h and atlwin.h as explained in a Code Project post. This cleaned up a bunch more errors. But I was still getting errors about _Module which I fixed based on yet another post.

Then I got errors that I was missing opengl32.lib which stumped me for a while till I found someone else with a similar problem, and ended up at instructions for adding the SDK libraries to Visual Studio. Since you don't actually need opengl32.lib, I'm not sure why the project specifies it.

Initially the Debug version built. It ran, but it had a bad tendency to crash. The Release version seems to be fine.

Another issue is that it's big, about 20mb for the required dll's. I guess that's not that big these days, but it's still a fair size to deploy over the internet.

This all seems way harder than it should be. I realize it's Windows and Visual Studio and C++ and open source, but still! Maybe I've been spoiled by working with Java lately. Java does have benefits.

Azul's Better GC

InfoQ: Azul Systems To Open Source Significant Technology in Managed Runtime Initiative

Interesting. The more cores you have, the more garbage collection (GC) you need - a result of the shared memory model. So even if you avoid killing your multi-threaded performance with locking overhead, you still have scaling issues with GC.

This may also put pressure on Java to do more to avoid heap allocation (e.g. escape analysis and fixnums).

Responsive Web Design

A List Apart: Articles: Responsive Web Design

Nice techniques for handling todays huge variation in screen size - from mobile phones to 30 inch monitors.

Email Lives

Hello Lego World

Very cool!



via John Nack

Phone Companies Antics

I had a phone call from Rogers yesterday trying to sell me a "Rocket Stick". I asked if I could use my existing data plan. "Oh, do you have a data plan?" they replied.

I wish these companies would get their act together and use the information they have. They say they're calling me because I'm a "valued customer" but they don't even have my account information. Of course, it's probably a different department or outsourced or one of a dozen other excuses. But it makes them look stupid. One of the first rules of selling is to know your customer.

So they asked me what my plan was ($30 per month for 6gb) and told me "Yes, you can switch to a shared plan". I was pleasantly surprised because I assumed the answer would be "no".

The new plan would be $45 per month for 1gb. Wait a minute, you want me to pay 50% more for 1/6 the service? "Yes, but it would be shared."  No thanks, doesn't sound like a good deal to me. (I don't use 6gb and 1gb would probably be fine, but I still don't want to pay more for less! And with 6gb I don't have to worry about my usage, which is a big plus.)

Actually, what I'm really interested in is the Mobile MiFi Hotspot. Then I could use it with my laptop and Shelley would use it with her iPod Touch or laptop at the same time. (Plus any future wifi devices.)  I'd looked this up before, but it just said "Out of Stock". I asked about it and they told me they used to have them, but there was some problem with the batteries. That sounded a little lame to me - they're still selling them in the US.

Note: In theory I can share my data plan by tethering my MacBook to my iPhone but it hasn't worked too well when I tried and I haven't spent the time to figure it out. And I don't think that would help with other devices.

see also: AT&T’s cynical act

Impossible Things

"Why, sometimes I've believed as many as six impossible things before breakfast." Alice in Wonderland

I was testing jSuneido on a Windows system and I got:

java.io.IOException
        at java.nio.MappedByteBuffer.force0(Native Method)
        at java.nio.MappedByteBuffer.force(MappedByteBuffer.java:154)

So I went to add a try-catch to log this error. But Eclipse told me MappedByteBuffer.force doesn't throw IOException

I check the documentation but it didn't mention IOException either.

This is a "checked" exception, meaning you can't throw it without declaring that you throw it, and you can't call something that may throw it without catching it.

But I guess all that goes out the window when you get down to native methods.

I just caught the more general base Exception instead.

PS. I wasn't able to recreate the error, I just hope it won't come back to haunt me.

Apple Overtakes Microsoft in Market Value

Apple Overtakes Microsoft in Market Value: End of an Era?: "At the close of Wednesday’s trading, Apple was valued at $222 billion, while Microsoft was worth $219 billion. Apple’s shares ended the day at $244.11, while Microsoft’s finished at a seven-month low of $25.01. And it isn’t only Cupertino’s successes, but also Redmond’s failures that are responsible for the new power dynamic between the two companies. Overall, Microsoft stock is down 20 percent compared to 10 years ago, while the value of Apple’s has grown tenfold over the same period."

Motivation

A great animated video for a TED talk by Daniel Pink related to his recent book Drive.

Third Time is the Charm

I decided that for what I needed Netty (or Mina) was overkill.

So I rewrote my server code using thread-per-connection and blocking io. I still used NIO Channels because I wanted to be able to use gathering writes (and because it was closer to the previous code).

And I still had the same problem!

That tells me it probably wasn't a mistake in my NIO non-blocking Selector version.

And the same fix "worked" for this version. (synchronizing the SocketChannel write)

That seems to indicate the problem is really in SocketChannel.write (or ByteBuffer)

I posted a question on the network forum on Sun (or should I say Oracle) but so far no response.

So I rewrote the server code a third time, using just old style java.net Socket, no NIO, no channels.

This time it seems to work!

On the positive side, the blocking, thread-per-connection versions (with channels or sockets) seem to be about 15% faster than the non-blocking Selector version.

Assuming I don't find any problems, and depending on what (if any) response I get to the forum question, I'll probably go ahead with this version.

Not Making Sense

Two more days working on my concurrency bug. I'm growing less and less confident that I'll "figure it out". The more I narrow it down, the less frequently it happens, which makes it more and more difficult (and time consuming) to know whether I've "fixed" it.

I'm really reluctant to say it, but I'm starting to wonder if there's a bug in Java. I have it narrowed down to where I check the data on the server immediately before writing it and it's always ok. But every so often the client will receive garbage. (Which usually leads to a timeout because it's waiting for a newline.) I'm pretty sure it's not a problem on the client side because it only happens when the server is multi-threaded.

If this was C or C++ I'd assume I had a bad pointer and was overwriting memory or that I was referencing memory that was prematurely free'd. But that's not supposed to be possible in Java.

There is no concurrent access in my code to the data that is being sent. I don't reuse the buffer or anything like that. I've tried using a read-only buffer, and I've also tried allocating a fresh buffer every time.

I have found a "fix". If I synchronize the channel.write on a global object, so only one thread can write at a time, then it seems to work. At least I've run it for about 10 times as long as it usually takes to get a failure.

But this "fix" makes no sense. First, channel.write is supposed to be thread safe. Second, concurrent writes (when they happen) are to separate channels, channels are not shared by multiple threads.

It could be a concurrency bug in Java or the OS, but why would I be the only one to encounter it?

Of course, this change may not be directly related to the problem. It may just be altering the timing enough to make the bug not happen, or to happen so rarely I'm not seeing it.

I've gone back and re-read all the NIO material I can find on the web. Despite a certain amount of conflicting advice I think I have a pretty good idea how it works, how to use it, and the common errors. I even read the arguments against using NIO (which I'm sympathetic to, given my problems).

Although it seems to be working, I'm not comfortable leaving it as it is. Since I don't know the source of the problem, or why/how the "fix" works, the bug could reappear at any time. But I'm also running out of things to try. It's narrowed down to only a few hundred lines of fairly straightforward code, most of which is identical to examples found on the web.

So my options are to abandon NIO and use a simple thread per connection model. According to some reports, this would be faster for the number of connections I expect (up to several hundred).

Or I could use someone else's framework. The contenders seem to be Apache Mina and JBoss Netty. (see Netty vs Mina)

I think I'll probably give Netty a try and see how it goes. But it's frustrating not being able to figure it out - I hate "giving up"!