I decided that for what I needed Netty (or Mina) was overkill.
So I rewrote my server code using thread-per-connection and blocking io. I still used NIO Channels because I wanted to be able to use gathering writes (and because it was closer to the previous code).
And I still had the same problem!
That tells me it probably wasn't a mistake in my NIO non-blocking Selector version.
And the same fix "worked" for this version. (synchronizing the SocketChannel write)
That seems to indicate the problem is really in SocketChannel.write (or ByteBuffer)
I posted a question on the network forum on Sun (or should I say Oracle) but so far no response.
So I rewrote the server code a third time, using just old style java.net Socket, no NIO, no channels.
This time it seems to work!
On the positive side, the blocking, thread-per-connection versions (with channels or sockets) seem to be about 15% faster than the non-blocking Selector version.
Assuming I don't find any problems, and depending on what (if any) response I get to the forum question, I'll probably go ahead with this version.