Third Time is the Charm

I decided that for what I needed Netty (or Mina) was overkill.

So I rewrote my server code using thread-per-connection and blocking io. I still used NIO Channels because I wanted to be able to use gathering writes (and because it was closer to the previous code).

And I still had the same problem!

That tells me it probably wasn't a mistake in my NIO non-blocking Selector version.

And the same fix "worked" for this version. (synchronizing the SocketChannel write)

That seems to indicate the problem is really in SocketChannel.write (or ByteBuffer)

I posted a question on the network forum on Sun (or should I say Oracle) but so far no response.

So I rewrote the server code a third time, using just old style java.net Socket, no NIO, no channels.

This time it seems to work!

On the positive side, the blocking, thread-per-connection versions (with channels or sockets) seem to be about 15% faster than the non-blocking Selector version.

Assuming I don't find any problems, and depending on what (if any) response I get to the forum question, I'll probably go ahead with this version.

Not Making Sense

Two more days working on my concurrency bug. I'm growing less and less confident that I'll "figure it out". The more I narrow it down, the less frequently it happens, which makes it more and more difficult (and time consuming) to know whether I've "fixed" it.

I'm really reluctant to say it, but I'm starting to wonder if there's a bug in Java. I have it narrowed down to where I check the data on the server immediately before writing it and it's always ok. But every so often the client will receive garbage. (Which usually leads to a timeout because it's waiting for a newline.) I'm pretty sure it's not a problem on the client side because it only happens when the server is multi-threaded.

If this was C or C++ I'd assume I had a bad pointer and was overwriting memory or that I was referencing memory that was prematurely free'd. But that's not supposed to be possible in Java.

There is no concurrent access in my code to the data that is being sent. I don't reuse the buffer or anything like that. I've tried using a read-only buffer, and I've also tried allocating a fresh buffer every time.

I have found a "fix". If I synchronize the channel.write on a global object, so only one thread can write at a time, then it seems to work. At least I've run it for about 10 times as long as it usually takes to get a failure.

But this "fix" makes no sense. First, channel.write is supposed to be thread safe. Second, concurrent writes (when they happen) are to separate channels, channels are not shared by multiple threads.

It could be a concurrency bug in Java or the OS, but why would I be the only one to encounter it?

Of course, this change may not be directly related to the problem. It may just be altering the timing enough to make the bug not happen, or to happen so rarely I'm not seeing it.

I've gone back and re-read all the NIO material I can find on the web. Despite a certain amount of conflicting advice I think I have a pretty good idea how it works, how to use it, and the common errors. I even read the arguments against using NIO (which I'm sympathetic to, given my problems).

Although it seems to be working, I'm not comfortable leaving it as it is. Since I don't know the source of the problem, or why/how the "fix" works, the bug could reappear at any time. But I'm also running out of things to try. It's narrowed down to only a few hundred lines of fairly straightforward code, most of which is identical to examples found on the web.

So my options are to abandon NIO and use a simple thread per connection model. According to some reports, this would be faster for the number of connections I expect (up to several hundred).

Or I could use someone else's framework. The contenders seem to be Apache Mina and JBoss Netty. (see Netty vs Mina)

I think I'll probably give Netty a try and see how it goes. But it's frustrating not being able to figure it out - I hate "giving up"!

Be Precise

Guest Post by
Penny McKinlay

I’m sure all of us have used the Help feature on a software program and been frustrated and annoyed because it didn’t make sense or didn’t answer our question.

One of my goals as a technical writer is to develop a good relationship with the customer. I want them to turn to the documentation feeling confident that it will help them complete their task quickly and successfully.

I don’t want to confuse or antagonize them.

Avoid Confusion

I prepare the online documentation for customers of an integrated trucking and accounting software package. I used to use the verb “select” when telling customers to pick the appropriate option on a pull-down menu.

Fortunately, one of the trainers pointed out to me that this was really confusing for the customers because there was a Select button at the bottom of many of the screens. Now, I only use the word “select” if I am referring to this button.

The Help carefully distinguishes between “placing a checkmark in the checkbox” and “generating a check.” And we deliberately use the American spelling of “check” so as not to irritate or confuse American customers.

We consistently refer to “Owner Operators” rather than switching back and forth between “Owner Operators,” “Leased Carriers,” and “Third-Party Contractors.”

Don’t Annoy the Customer

I try and write the Help from the user’s perspective: “you will not be able to invoice your Order until you do this” or “use this report to review outstanding trucking transactions.”

We used to say that the software would “allow” customers to do certain things, but we realized that the word “allow” was really condescending. Now, we use phrases such as “this screen will help you to . . . .”

We also try very hard to avoid jargon because it’s intimidating and makes readers feel stupid. Software programmers talk about “parameters” and “locking down.” The Help talks about “options” and “protecting.”

Writing Tips

Be precise, be consistent, consider your audience – good things to keep in mind whatever we’re writing.

For more tips on business writing, visit my web site.

Penny McKinlay
pennymckinlay.axonsoft.com
www.wanderlustandwords.blogspot.com
twitter.com/PennyMcKinlay

enTourage eDGe™

enTourage eDGe™

Another interesting alternative to the iPad or Kindle. I'm not sure it's ready for prime time though - it sounds like it's slow, heavy, limited battery life, and no apps. Other than that, it looks great :-)

I'd really like to go for something more "open" than the iPad or Kindle, but the problem is the apps and books. No one has all the apps like Apple, or the books like Amazon.

If we can move toward HTML 5 apps and get away from DRM, then we'll have a lot more freedom with hardware.

Which still leaves the question whether anyone can match Apple in terms of design and user experience. Even people who promote Android admit that it's not as polished as Apple. The software might get there eventually - Linux (e.g. the latest Ubuntu) is starting to rival Windows and OS X. But Apple's hardware design and polish is tough to match.

iPhone / Touch Apps

I gave my sister Penny an iPod Touch for her birthday and figured I'd better give her some recommendations for apps. Here's some of what I have on my iPhone:

Evernote (free) - save notes, photos, documents, works with web, Windows, and Mac versions
BlogPress ($2.99) - blog post writing tool, you can write offline
Note Taker ($1.99) - hand write notes with your finger (free Lite version)
Kindle (free) - ebook reader, works with Kindle devices or Windows and Mac software
Tripit (free) - trip itinerary, syncs with web site
iKeePass ($0.99) - password manager, works with web, Windows, and Mac versions
Remote (free) - remote for iTunes on Mac computers
Facebook (free)
Echofon for Twitter
Encyclopedia ($8.99) - offline copy of Wikipedia
iBird (free to $29.99) - bird guide (search for iBird)
GoodReader ($0.99) - PDF reader (free Lite version)
Instapaper (free) - save web pages for later offline reading (Pro version $4.99)
Outliner ($4.99) - syncs with web site
Google Earth (free)
oMaps ($1.99) - open maps
Fugawi iMap ($4.99) - topo maps of Canada and USA, can download maps for offline
Copilot Live Directions (free) - offline street maps of Canada and USA
Stanza (free) - ebook reader
Eucalyptus ($9.99) - ebook reader, access to 20,000 free books
Dropbox (free) - works with Windows and Mac software and web site
Gnu Go (free)

Perseverance

Another day working on my bug but at least I've made some progress, at least as far as narrowing it down.

I managed to simplify my test to just repeating the same request over and over (instead of the semi-realistic mix of operations in the original test) This greatly reduces the amount of code where the bug could be.

Next I wrote a test client in Java to do that same request. But it succeeded. (Of course, you never know if one more run would run into the bug.)

Doing more testing with the cSuneido client I realized that the bug occurred more often (or only) when the client and the server were on separate machines. So I tried running my Java client on a separate machine. And now it failed too.

So now I know the problem isn't with cSuneido, or Parallels, or Windows.

I narrowed it down even further by commenting out all the real work of the request (the database querying stuff) leaving only the framework of request handling. It still fails. This reduces the amount of code where the bug could be even more.

But I'm still puzzled. My simple test client plus test server has yet to fail, even running on separate machines. But the test client plus the real server does fail, even though the real server is cut down to where it's doing little more than the test server. There must be some difference. Or else it's just that the probability of the error is less and I haven't run it enough times.

Just to give an idea, when it's failing, it fails about once every million requests. Gotta love that kind of bug!

So I still haven't found the problem, but I might be getting closer.

This is what I was afraid of!

I just spent another day trying to find my jSuneido concurrency bug, with no concrete progress. That makes four days I think, definitely the trickiest bug on this project (so far). It's this kind of bug that makes me afraid of concurrency in the first place.

I made a test server and test client and at first I thought I had recreated the problem, but the fix I found only worked for the test program, not jSuneido :-( Since then my test server and client have worked perfectly, despite extending them to be more like the real jSuneido code.

I've tried synchronizing here and there and making defensive copies of this and that but the bug still happens. I've read and re-read the code. I've drawn diagrams of the sequence of events.

I start to wonder if the bug is in the cSuneido client rather than the jSuneido server. But that doesn't really make sense because we have thousands of people using the cSuneido client and we don't see this problem.

My next step is to write a simple Java client for the jSuneido server. That will eliminate the cSuneido client. I think the best bet is to try to gradually narrow down the problem and isolate it.

I don't doubt I'll find the problem eventually. The question is how many days I'll need to bang my head against the wall. Hopefully the wall crumbles before my head!

People Hate Change

My company sells specialized software for trucking companies. Often companies that we sell to have been using QuickBooks. And the people in those companies are very reluctant to leave QuickBooks and switch to using our software. And even after they do switch they continue to complain that our software does not work like QuickBooks.

QuickBooks is a good program and no question there are QuickBook features that are our software lacks. But for trucking companies our software does a lot more than QuickBooks ever will. QuickBooks doesn't handle the specialized needs of trucking companies, our software does. But to the people being faced with change, none of that matters, they want the familiar.

This was brought home recently by a long time customer of ours. They're not a trucking company and the software they have is very old. (It's DOS text mode software!) Recently they wanted to set up another company and wanted another copy of this ancient software. We suggested they just use QuickBooks. They came back and said they hated QuickBooks and that their current software is much better! This is so obviously not true that it was funny. It was a good reminder of how strongly biased people are to what they are used to.

jSuneido Concurrency Bug

I spent all day yesterday working on my concurrency bug in jSuneido. (see previous post)

It didn't seem like I made any progress, but I guess that's not true - I eliminated a few possibilities.

Originally I assumed it was a deadlock since it was "freezing". So I managed to make the problem happen (that takes a while), connected JConsole, and ran the deadlock detector. It didn't find any deadlocks. Thinking about it, that makes sense since it's really only the client that gets "stuck" - the server thinks everything is fine. There are no worker threads running for that client.

So much for that assumption. I tried to narrow it down, without much success. I added a bunch of assertions, which failed, but I found it was my assumptions that were wrong, not the code.

The client appears to be waiting for a response to a request. But the server is not in the process of handling a request - it either didn't get the request, or it thinks it finished it. The read and write buffers for that client are empty.

Sometimes you have to run test for a long time before the problem shows up. So you try something, and the longer the tests run successfully, the more you think you've got it fixed. But eventually it has always shown up. I guess I should be thankfully this is on the order of minutes rather than hours! Even so, it's reminiscent of the "old" days when you had to have something to read while you waited for compiles. (And no internet to browse back then.)

On the positive side, the handling for inactive transactions and clients seems to be working well - when the client gets stuck, the server eventually aborts its transactions and if it stays stuck, closes the connection.

My current hypothesis is that it's related to a potentially overlap between handling requests. As an optimization, worker threads first attempt to directly write their response. This is in non-blocking mode so it may or may not write everything. Any remaining data is sent by the select thread when the socket becomes write ready. But if the worker does successfully write the entire response, and then there is a context switch back to the "select" thread, the next request can be read and another worker started, before the previous worker has a chance to finish. I'm not sure why this would be a problem, but my assumption when I was writing the code was that workers for the same client would not overlap, so I didn't worry about making the data structures thread safe. That's the next area I want to look at.

Dig Deep

In the last few days I've questioned a number of changes that I saw going through version control. None of them was "incorrect". The recurring theme was that they were fixing problems at too "shallow" a level.

For instance a dialog was using a feature to save its size after being changed by the user. This wasn't working properly, so the feature was disabled. There's a couple of problems with this. One is that we lose a feature from that dialog that we presumably wanted. But more importantly, that feature is used in many places and if there's a bug in it, it's going to show up elsewhere. Are we going to disable it everywhere? How much time are we going to spend dealing with each new instance that turns up?

Of course, sometimes you have to fix a bug the most immediate way. That's fine as long as you go back and look for the real problem.

Here's another example: we got an error about a missing database table. This was after wiping out all the data so the initial thought might be to ignore it. But this table is supposed to be created automatically. So we dug a bit deeper and we found one place where it didn't create it when necessary and fixed that. But access to this table is supposed to be encapsulated in one class. This class handled creating the table if necessary. The spot we fixed was accessing the table directly. So we moved this access into the class to restore encapsulation. But then we realized that it shouldn't even need to use this class/table. It turns out we had used it to work around yet another bug. So we removed the use of the class/table and fixed the original, previous bug. The end result was less code and simpler.

This process of digging deeper into a problem can take time. It can take orders of magnitude more time than a quick fix to the immediate problem. Which is why we often apply the quick fix and move on. But the long term result of that approach is technical debt - your code gets more complex, less understandable, less consistent, hard to change. And in the long run the quick fix ends up costing your more time than if you had fixed it properly in the first case.

This is a bit like the "lean" idea of finding and fixing "root causes". One of the techniques is to ask five (more or less) "whys". Why did we get a missing table error? Because the code wasn't creating it on demand. Why? Because it was bypassing the access class. Why? Because it was doing something non-standard. Why? To work around another bug.

Is It All About Sales?

It always bugs me when people (usually sales people) say "It's all about sales. Everyone in the company needs to be selling all the time, not just the sales and marketing department."

Before anyone gets the wrong idea and starts jumping on me - I am not disagreeing with this. But I don't think it's the whole picture, and I don't think it's the most useful way to think about it.

Take my company - we have sales and marketing, product development, and customer support. To me, they're like three legs of a stool. Does it make sense for one leg to say it's all about that leg? All three legs are essential. Take away any one and the stool will fall over.

Part of the reason sales is seen as primary is that they bring in the money. It's pretty obvious that's an important role. They have the most direct link to the success of the company. But just because the other roles have a less direct link to success doesn't make them less important.

Think of fighting a fire. The fireman holding the nozzle of the hose has the most direct link to spraying water on the fire. But without the rest of the hose, and the source of water, he would be out of business.

Take Apple for example, obviously great at sales and marketing. But are they "all" about sales? Where would they be without great products?

A good marketing and sales group sometimes think it can sell anything. (And there is some truth to that.) But when my company has tried to sell third-party products it has become pretty clear that if the product doesn't have good development and good support then in the long run, it's pretty hard to sell.

You don't usually hear "it's all about customer support" or "it's all about product development". To me, those views would be just as incomplete. For example, 37 Signals claims they have no marketing people. That may be true in the sense that they don't have anyone whose only job is marketing. But get real, those guys do more marketing than most marketing departments. They're certainly not "just" programmers.

Perhaps another reason you hear "everyone needs to sell" more often is that non-sales people tend to avoid selling. Whereas sales and support are usually more than happy to talk about where the product should be developed. Another weak spot is that programmers tend to avoid customer support (I know I did!).

While equally incomplete, I think there's value in the idea that everyone needs to do customer support and everyone needs to do product development, just like there is value in the idea that everyone needs to sell.

I think what it boils down to is that everyone needs to keep the big picture in mind, not just any one particular role. That's easier and more obvious in a small company. Most of the time I think my company does fairly well at this. Sales and support contribute to product development, programmers and sales help with customer support, and yes, hopefully everyone helps sell.

Cultures and Attitudes

You would think that peoples' attitudes and behaviour would be an individual thing with a range of variation in any organization.

But attitudes and behaviours are greatly affected by the culture of an organization, biasing people within the organization to act a certain way. There is still variation, and you can find good people in a bad organization (or vice versa). But even this is reduced by a tendancy for good people to leave a bad organization, and bad people to be removed from a good organization.

Amtrak (US passenger rail) seems to have a culture of small minded bureaucracy.

On the way to Portland there were only a handful of people in our car. So one person moved to a seat with more room. No big deal, until the steward came along: Is this your seat? You must remain in your designated seat. It is not possible to switch seats. The passenger wisely kept quiet. Eventually the steward wound down his littany of pronouncements. The passenger looked at him. Then the steward said "Ok, here's what we can do. I can let you stay in this seat as long as you agree to move if someone else is assigned this seat.". I had to shake my head. This was exactly what the passenger had said he would do right from the start. "But you must leave your seat assignment card on your assigned seat."

But that wasn't the end of it. A bit later a second steward came around: Is this your assigned seat? No, but the other steward said I could sit there. You could tell he was frustrated at having his exercise of power cut short and looked for another outlet. "Ok, but you must have your seat assignment card on the seat you are sitting in, not the seat you were assigned." (The exact opposite of the first steward.)

This kind of bureaucratic attitude is often exemplified by people saying "you must" or "you cannot" or "I cannot", even (or especially) when this is obviously untrue.

For example, when I went to get tea I was told "I cannot put hot water in your cup.". Obviously he "could", presumably what he meant was that he wasn't supposed to. So why not say that? It would be a lot less frustrating if he said "sorry, I'm not allowed to do that". Or just bend the rules. The previous time, the person had put the hot water in a paper cup, as he was required to do, and then poured it into my cup. This satisfied his requirement, while at the same time fulfilling my desire to not waste a cup.

I think creatively interpreting (bending) the rules (and not being punished for it) is a sign of a good organizational culture. Whereas, presenting the rules as "cannot's" is a sign of a bad one.

The problem is that saying "you cannot" makes you feel more powerful. While saying "I'm not allowed" makes you feel less powerful. And of course, most people would prefer to feel more powerful.

I worry about where my own company's culture fits. I hope it's clear that people should understand the reason for "rules" and use their judgement in interpreting them. And at the very least be truthful and say "I'm not allowed to do that" rather than "you cannot".

Building the Wrong Thing

"The biggest cause of failure in software-intensive systems is not technical failure; it's building the wrong thing." - Leading Lean Software Development

(Presumably that assumes a certain minimal level of technical expertise.)

I spend a lot of time thinking about how to reduce the errors (bugs) in our software. We spend a big percentage of development time on fixing errors. Just to be clear, most of these errors are small internal issues that aren't even noticed by the majority of users. I think as far as our users are concerned, our quality control is good.

But if we could reduce those errors, we would have more time to get stuff done.

But ... if the biggest cause of failure is building the wrong thing, then having more time won't necessarily help. Or at least it's not the best way to improve. By this line of thinking, we'd be better off working on building the right thing, rather than less errors. (Of course, you still want to reduce errors as well.)

It reminds me of the old saying "if it doesn't have to work, you can make it as small/fast/etc. as you like", if you interpret "work" as "building the right thing".

Of course the trick is how to do better at building the right thing. In many ways that's tougher than reducing errors, and that's tough enough.

If you're not involved in the software business at this level, this talk of "building the right thing" may strike you as strange. Why wouldn't you build the right thing? The problem is that no one knows what the "right thing" is. And no, the customers don't know either. As Ford said, "If I'd asked my customers what they wanted, they'd have said a faster horse."

None of this is really new, but it's good to be reminded. Definitely food for thought.

Books and More Books

If you like books and you're in Portland, don't miss Powells. And if you like technical books (eg. computer stuff) don't miss their technical book store a few blocks away.

The technical store even has a display of old computer hardware - an Apple II, an Osborne, an original Mac, etc. I can remember quite clearly when those machines were the hot new thing. That's dating myself, but there's something to be said for perspective.

One of the nice things about Powells is that they mix new and used books. That means you see older books and books that are out of print. They had several copies of Software Tools by Kernighan and Plauger. I remember how excited I was about that book. It came out in 1976, when I was 16. A few years later, a friend and I wrote versions of some of those tools (in C, compiled with Whitesmiths compiler) and sold them. Not that we made any money at it.

That's the exception though. Most computer books don't age very well. They become useless a few years after they're published. I should know, I have piles of them!

Between the two Powells stores, and a Borders I visited on the way there, and between computer, science, environment, and science fiction, I made note (on Evernote) of 30 books I wanted to read. I didn't buy any, which in a way was easier - it would have been tough to choose! (I've already bought one on my Kindle).

Oh well, I guess there's worse things to be addicted to!

Hands on the iPad

I got to touch an iPad today at the Apple store in Portland.

As others have mentioned, it is heavier than you expect. I'm not sure why that is when you've never felt one before. Maybe because the ads make it look so thin. You expect it to be as thin and light as an iPod Touch but it's thicker and heavier like an iPhone.

Beyond that it was like a bigger, faster, fancier iPhone. Cool, but nothing earth shaking.

Do I want one? Sure. Am I desperate to buy one this minute? Not really. My current iPhone plus Kindle combo works pretty well for me.

It was interesting to listen to people's reactions to the iPad. People still aren't quite sure where it fits.

As usual, the Apple store was packed. There always seems to be an undercurrent of excitement.

I fondled the new MacBooks, but externally they haven't changed much. It's that i7 heart I lust after. And I gazed longingly at the 26" iMac with quad core i7. Someday. It's not like there's anything wrong with my current MacBook or iMac. Just seductive marketing.

Location:SW Naito Pkwy,Portland,United States

A Plea to Evernote

Evernote is a popular app for the iPhone and iPod Touch. (There are also PC and Mac versions.)

You can write notes and record photos from the iPhone camera. Notes are synced to Evernotes servers and to all your devices. You can organize and search your notes. Evernote even does OCR (optical character recognition) on images so you can search for text in images.

It's a great product and many people swear by it.

But ...

The software is buggy. It freezes. It crashes. You have to reinstall it periodically to get it working again. Check out the problem reports on GetSatisfaction for more examples.

A tool like this is only worthwhile if it's dependable. I put all my travel information in it and it died almost as soon as I left home. That doesn't make a happy customer. (thankfully reinstalling got me going again)

I'm not the only one with a love hate relationship with Evernote. People give up on it, come back thinking it's fixed, only to discover some time later (after entering a bunch more notes) that it's still got bugs, just a different set from before.

I'm not sure where the problem lies. But if they want to be successful in the long run they have to do better.

Quit adding new features until it's more stable. Add more automated tests. Set up alpha and beta testing progroms. Review the code. Refactor the code. Throw out problem code and rewrite it. Whatever it takes. I can't see anything more important.

It's a good product but it doesn't matter how many great features it has if it's not reliable.

Not Sold on Multiple Monitors

Jeff Atwood recently posted about using three monitors. He's a big fan of multiple monitors.

Personally, I'm not so sure. A few years ago we upgraded almost everyone at our company to dual monitors. I was one of the few hold outs.

At that point, one of my arguments was that I preferred one large monitor to two smaller ones. Even if the surface area is larger, two monitors isn't so good if you want one large window (e.g. Eclipse or Lightroom). Of course, if money isn't an issue, you could just have two (or three) large monitors.
To be clear, I'm not saying there aren't good, valid uses for multiple monitors. Studies have shown they can increase productivity. But I'm not sure that translates into normal usage. As the saying goes, "In theory, theory and practice are the same. In practice, they are not."

I think the key issue is how people use multiple monitors. I suspect that many people use them in ways that are counter productive, that just add distraction and interruptions to their environment. I think you should be closing your email and chat and Twitter and Facebook - not using multiple monitors to keep them constantly in your face.

Of course, it's virtually impossible to wean people away from keeping email etc. open all the time. It doesn't matter how much you talk about the cost of distractions and interruptions. Regardless of the excuses they give, I think the problem is that a lot of people welcome these distractions. After all, it's hard work to focus on one task for extended periods.

If you're going to focus on one thing at a time, then do you still need multiple monitors? Occasionally, I would like to have documentation or the program I'm working on open as well as my IDE. But I find virtual desktops to work pretty well for this.

I do agree that ending up with one monitor straight ahead is an advantage of three monitors over two.

Spoke Too Soon

Good think I'm not superstitious or I'd be certain I jinxed myself writing my previous blog post.

Right after posting it I started to test limiting the maximum number of threads and ran into a problem I'd actually seen earlier in the day but written off as nothing because I was still getting my test program working.

For some reason, occasionally one of the clients will get "stuck". The server appears to have completed the request, but the client appears to still be waiting for a response. Eventually the client times out.

And, Murphy's Law, it seems to not happen (or at least nowhere near as often) when the debugger is attached. Which happens to be how I was running the tests all day, just in case I ran into any problems that I would need to inspect. When I resumed testing after writing the blog post, I didn't attach the debugger, since I was just wrapping up.

Of course, not being able to make it happen easily, and not at all when the debugger is attached is going to make it hard to track down. Exactly what I was afraid of. Wish me luck!

Putting jSuneido Through Its Paces

I've spent the day exercising jSuneido. I planned to do more of this sooner, but I kept finding other parts to work on. Partly I'm scared of finding bugs, which at this stage in the game are liable to be obscure and hard to fix. Wimp!

But so far, no new bugs. I've had three computers, each with multiple instances of clients (from 4 to 40), all hammering constantly on the jSuneido server. So far so good - it appears to be solid. I've been using JConsole to monitor it. It's fun to watch the memory grow and then shrink after garbage collection, and threads come and go in the thread pool.

My iMac is just dual core. I also want to run some tests on a quad core machine at the office. More cores is a better test of concurrency since there is more going on literally at the same time, rather than just with context switching. Also, at the office I've got even more machines to run clients on. And of course, I want to see how it scales with more cores, which is to a large degree the point of this whole project.

One thing I'm not sure about is whether I should be limiting the size of the thread pool. Currently I'm just using a standard Executors.newCachedThreadPool which isn't limited. So if I fire up 40 clients making constant requests, I get 40 threads. But 40 threads on 2 cores presumably means a lot of context switching overhead. I'll have to experiment. (Note: in normal operation there wouldn't be one thread per client because the clients wouldn't be making continuous requests. There would only be as many threads as concurrent requests.) The minimum context switching would be to have the same number of threads as cores. But you want to service clients concurrently, i.e. a long request shouldn't block other requests. So you have to sacrifice some context switching overhead to gain more concurrency.

Slow Gmail on iPhone

Lately Gmail on my iPhone has been really slow. (This is the web version, not the mail app.) The screen pops up right away, but it takes a long time to finish checking for new mail - maybe 30 seconds.

I don't have a lot of messages in my inbox, often none. And it happens with a good wifi connection.

I didn't really think too much about it, but I let my sister check her gmail and it was much faster. Very strange.

I searched on the internet but didn't find too much. Most of what I found was for the mail app.

So I just did the usual "kick the machine" - I cleared the history, cache, and cookies. That seemed to do the trick - it was fast again (less than 5 seconds).

So I wrote this blog post and then went back and tried it again and it was slow!!!

Next, I tried deleting the databases for gmail. Still slow.

Try again, delete the databases and this time also clear the history, caches, and cookies. Log back in and it's fast again. And it seems to be staying fast this time, although I'm not totally confident that it'll stay that way.

If you're having the same problem and want to try this, go to Settings > Safari, scroll down to the bottom. Click on Databases, then Edit, then delete any gmail entries. I always seem to have two entries which appear identical. Then go back to the Safari settings and click on Clear History, Clear Cookies, and Clear Cache (one at a time).

NOTE: Clearing cookies will log you out of sites, so don't do this if you don't have your passwords handy to log back in.

A Welcome iTunes Feature for the iPhone

For smaller iPod's like the Shuffle, iTunes has for a long time had an option to convert audio to lower quality to fit more on the device. But for some reason this feature was not available on the iPhone until recently.

As I accumulate more music, and at higher quality, and therefore larger files, my iPhone has been getting full, despite being a 32gb model. I have a few apps that take up more space, like Wikipedia and CoPilot Live but the majority of the space is audio.

Recently I noticed that this option had appeared for my iPhone. I think it probably came in the last update, but it may have been there for a while and I just missed it. It freed up over 10gb of space on my phone! Very nice. Audiophiles will no doubt disapprove, but in the places where I listen to music on my iPhone, I doubt I'll notice the difference.

NOTE: Because this requires updating all the music files on your phone, the next sync after setting this will take a while. Don't do it when you're short of time.

Geek Envy

I was sitting reading my Kindle over lunch the other day at Innovation Place. Someone walking by did a double take and asked "Is that an iPad?!"  I said no, it was "just" a Kindle. He was relieved and explained, "My boss is getting the first iPad in Saskatoon and he'd be really upset if someone else got one first." I don't think he saw me shake my head and roll my eyes as he walked away.

Eclipse Keyboard Shortcuts

I don't tend to use a lot of keyboard shortcuts, but I use these Eclipse ones quite regularly:

• F3 - Open Declaration

• Shift + Command + G - References

• Alt + Command + M - Refactor - Extract Method

• Alt + Command + R - Refactor - Rename

• Command + F11 - Run

Even if you're only an occasional Eclipse user,  it's probably worth learning a few of the shortcuts.

Related posts:

Template Shortcuts in Eclipse

Eclipse Tip for Windows and Mac Users

Template Shortcuts in Eclipse

One of the features of the Eclipse IDE is that you can define "template" shortcuts. For example, you can expand "main" into the complete method definition. I don't use a lot of these, but they are handy once your fingers learn them.

Eclipse comes with a bunch of pre-defined templates and you can add your own. For example, I added a template that expands "sop" into "System.out.println".

Ctrl+Space brings up the choices. In theory if you select Auto Insert and there is only one matching choice it will automatically fill in the template. In practice there are usually multiple choices so you have to pick from the list. That's usually just a matter of hitting Enter (as long as your choice is the first one).

You can find the templates in Preferences - Java - Editor - Templates.

I also edited the "test" template to generate a JUnit 4 test (instead of the older style):

@Test
public void ${name}() {
    ${cursor}
}

A Faster, Cleaner Database Design for Suneido

Current design:

• entire database is kept in a single file
• the database file is accessed via memory mapping
• data records are immutable, updates add new versions
• indexes are mutable
• data records are used like a "log" for crash recovery
• indexes are not recovered, they are rebuilt

For concurrency, it would be really nice if the database was a persistent immutable data structure. But this is difficult. For example, you don't want to write a new copy of a btree index node to disk every time you update it. (At least, I'm assuming that the resulting database growth would be impractical.)

In theory Suneido happily accesses databases bigger than physical memory. In practice, performance drops off rapidly when this happens due to swapping (page faults). You would think there would be a "working set" of data that would be a small portion of the database. But users tend to run things like historical reports or searches that access large portions of the data.

The other issue is that memory mapping the database makes reads fast (once the file is mapped into memory). But writes still have to go to disk (albeit somewhat asynchronously). And the index update writes are essentially random which is the worst case for disk performance. (Maybe SSD drives would help with this?)

Which led me to think, why not just keep the database in memory. But...

• Despite the performance issues, I'm not sure it's reasonable to limit database size to physical memory.
• Loading the database into memory on startup and saving it to disk on shutdown would be slow.
• You still need to write something to disk for crash recovery.

Next iteration, why not memory map the database, write data records to disk (so they can still be used as a "log" for recovery), but keep index changes in memory and only write them out when you close the database.

This has some nice features:

• Similar start up speed (although slower shutdown)
• Less disk writes (only data, not indexes) - faster
• Disk writes are sequential  (index updates were random) - much faster
• Allows databases bigger than memory

Suneido's crash recovery rebuilds the indexes anyway, so it's not a problem that they're kept in memory. Theoretically, if indexes are now immutable you could take the last good state and roll forward from there for faster recovery.

You could also periodically update the on-disk indexes, presumably during idle time on a background thread.

This seems pretty good, but there's more. Now that index updates are not being written to disk immediately, it becomes feasible to make the indexes persistent immutable data structures.

jSuneido already sort of does this, but in a complicated round-about way. Each transaction maintains an "overlay" of updates to indexes. When the transaction commits, these are merged into the real indexes on disk and other concurrent transactions are given a copy of the old index nodes so they don't see the changes. It works, but it's ugly and error prone.

With this new scheme, each transaction has, in effect, it's own copy of the indexes. Since it's a persistent immutable data structure, updates by transactions have no effect on each other. When a transaction commits, you merge it's indexes with the master. I think this would make the code quite a bit simpler.

It's hard to say how much faster this would be. Updates would be considerably faster, but reading would be much the same. So the overall speedup would depend on the mix of updates and reads. However, in a multi-user scenario (the current target for jSuneido) if you can reduce the load, that benefits everyone, including reads.

I'm not going to drop everything and implement this, but it's an interesting idea. I'll have to mull it over some more - there may be flaws that I haven't thought of yet. It might be something to work on once jSuneido is "finished" and deployed.

As with any "bright idea" like this, I can't help wonder if I'm reinventing the wheel. I wouldn't be surprised if someone was already using this approach. Of course, that doesn't mean that there's any way to reuse their code, or even learn from them. The internet might give us access to the world's knowledge, but it's still hard to find this kind of stuff.

Related posts:
Language Connoisseur
A Java Immutable Persistent Map
A Java Immutable Persistent List
jSuneido Progress

Blogger + Picasa Tweak

When I reference photos on Picasa from my Blogger blog I use the HTML under "Link to this album" and then "Paste HTML to embed in website" (using the Edit HTML function of Blogger)

However, when people click on the result they get taken to the album page of thumbnails. Some of my less technically savvy friends don't realize or notice that they can get a slideshow so they end up clicking on the photos one at a time to look at them.

You can fix to go directly to the slideshow by changing:

?feat=embedwebsite

to:

#slideshow

 in the HTML. There are two occurrences - one for clicking on the photo and one for the caption.

You can see examples in my Sustainable Adventure blog.

Note: Picasa also has an "Embed Slideshow" option under "Link to this Album" but I don't normally use this because it only shows small images and I'd prefer to have people view my photos larger. I do use it when I'm not as concerned about image size. (You can still click on it to go to the Picasa album, but most people won't bother.)

Visibility

Now that jSuneido is more or less functional, I've been working on some of the peripheral functions like checking and rebuilding databases. (i.e. crash recovery)

The checking part went quite smoothly and quickly. The rebuilding/recovery part has been slow. I've been working on it for several days and it seems like I've made no progress. The only code I've written is tests and debugging! But whether it feels like it or not, I guess I was making progress because it "suddenly" started working today.

Recovery is hard for a number of reasons. For starters you're assuming the database has been corrupted, so you have to code a lot more defensively. Second, because you're working outside the normal operation of the database, you can't use the normal functionality. You're not doing transactions etc.

One lesson I (re)learned was how important "visibility" is. By that I mean being able to "see" the data you're working on. That's a large part of why debuggers can be valuable - they let you inspect the data. Often, inserting the right "print" statement in the "right" place is all it takes to figure out a bug. Of course, finding that right place is not so easy. In this case visibility meant writing a dump utility so I could see exactly what was inside the database. Obviously, I have a pretty good idea in general how it's structured, but not the details of exactly which types of blocks of data are in which sequence. And dumping to text files meant I could use tools like diff to compare before and after recovery.

PS. In hindsight, yes, it would have made sense to write the database checking much earlier so I could verify that databases weren't getting corrupted by operations. On the other hand, that was usually pretty obvious -  it crashed!

Readability

How often do you go to a web site to read an article and find that the page is plastered with distracting ads and other material, the text is too small, and the lines have way too many characters for comfortable reading?

Here's a solution - install the Readability bookmarklet. It will re-format the page you're on for easier reading - less distractions, bigger text, and shorter lines.

Before:

After:

Buy O'Reilly eBooks Direct

When I wrote my last post - Buy Pragmatic Programmer eBooks Direct - I should have mentioned that O'Reilly has a similar deal for their eBooks - ePub, Mobi, and PDF formats, DRM free, and free updates. Not all of their books are available as eBooks (some are print only) but most of the newer ones are.

They even have a 3 for 2 sale on right now.

Buy Pragmatic Programmer eBooks Direct

If you're buying Pragmatic Programmer books for your Kindle you have the choice of buying from Amazon or buying direct.

Buying from Amazon is the path of least resistance since you can buy right from your Kindle.

But if you do that, you'll end up with a proprietary format with DRM that only works with your Kindle. (Although you can read them on your PC or iPhone or iPod Touch with the Kindle app.)

Instead I'd recommend buying direct. That way you can download the book in multiple formats - PDF, ePub (e.g. for iPhone or iPod Touch using apps like Stanza, or for Sony readers), or Mobi (for Kindle). The books are DRM free, and as a bonus, you can download updated copies when they come out.

I've bought a few eBooks from Pragmatic Programmers in the past, either on their own or together with a paper copy. It was very nice to be able to go back and download the Mobi version so I could have them as a reference on my Kindle.

The only very minor hassle is that you have to buy and download the books on a separate computer and transfer them to the Kindle. In Canada that means by USB since emailing to your Kindle is not supported here.

Thanks to Pragmatic Programmers, at least a small part of my eBook library is not locked to the Kindle. I hope more publishers follow their example.

jSuneido Speed Issue Fixed

After some digging, I found the speed issue on Windows.

It didn't turn out to be Parallels - it was just as slow on my Windows machine at work.

It didn't turn out to be anti-virus - it was just as slow with anti-virus turned off.

I thought it might be Nagle problems again, although last time it was the opposite problem - slow on OS X and fast on Windows. But nope, not that either.

I narrowed it down to sending the response. Debugging showed it was fast receiving the request and executing it.

I found I had two methods for sending responses. One was doing selector.wakeup after changing the interestOps (like you're supposed to) and the other wasn't. Of course, the incorrect code was the method being used. I fixed the method being used and got rid of the other one. Problem solved.

Now, the speed is reasonable. 75 seconds to run the test suite with the cSuneido server, 100 seconds with jSuneido.  I'm pretty happy with that considering the cSuneido version is C++ code that has been tweaked and optimized for years, whereas jSuneido is my first Java program and has barely been debugged, let alone optimized.

I think jSuneido can probably be faster, but even if it can't, its ability to scale to multiple threads/cores should allow it to outperform cSuneido when there are multiple users. I say "should" because I've barely started testing this. One very preliminary test took about 40 seconds with one client, 50 seconds with two simultaneous clients (on two cores), whereas cSuneido would be pretty much linear i.e. 80 seconds with two clients. Of course, it would be nice to have a quad-core machine to test with, but I haven't quite convinced myself that's excuse enough to buy a new Core i7 iMac :-)

Removing Borders from Images on Blogger

This may be due to the template I'm using, or maybe even my browser, but in case other people are having the same problem, I thought I'd post my workaround. (Besides, it gives me a handy reference when I forget what to do!)

When I insert images into my blog posts I get a gray border around them. Like this:

Sometimes that's ok, but often, like with this image, I don't want that border.

NOTE: The border doesn't show up while composing, or in preview, only (annoyingly) after you publish.

To remove it, I edit the HTML and add style="border: none;" to the img tag. (I'm not sure why this is necessary because it already has border="0" - maybe something in the CSS.) The result should be the image without the gray border, like this:

Note: If you're reading this in something like Google Reader you may not see the border.

Back to Work on jSuneido

I fixed a couple of bugs and now all our application tests pass, both standalone on jSuneido, and client-server (with the Windows cSuneido client).

But quite slowly - about 4 times slower running client-server with jSuneido as the server instead of cSuneido. I'm not surprised it's slower, but 4 times seems like a lot. Strangely, when I run the tests under jSuneido standalone (i.e. not client-server) they run about the same speed as cSuneido client-server (which is still about half the speed of cSuneido standalone). So presumably the difference is the client-server networking.

I've been testing with the server running on OS X (since that's where I develop) and the client running under Parallels. Maybe the networking slows down the Java test so I decided to try running the Java server on Windows as well as the client.

First I had to remember how to package up jSuneido as a jar file. I've done this before, but not for a while. In Eclipse you use Export > Runnable Jar File.

Next, I found I didn't have Java on my current Windows 7 Parallels VM. I downloaded and installed the JRE (Java Runtime Environment).

I got some weird errors until I went back and rebuilt my jar file with "Extract required libraries into generated JAR" (instead of "Package required libraries into generated JAR"). I'm not sure exactly what the difference is, or why it makes a difference to running my code. I made a half-hearted attempt to find it in the Eclipse Help but didn't get anywhere.

Finally I got it running, but it was soooooo slow! Instead of the IDE starting more or less instantly, it took more like a minute. Ouch.

I figured it might help to run it with the -server option (that's the default on 64 bit Java, which is what I'm running on OS X). But I discovered the JRE doesn't include the server version. Grrrr. It comes with the JDK (Java Development Kit). That seems odd to me, but that appears to be the way it is.

So I installed the JDK, but the default Java that's on the path is still the JRE one. So now I have two separate installations of Java with the "wrong" one on the path. Sigh.

I ignored that problem and used the absolute path to run the JDK one to get the server version. But it's still just as slow. Argh!

The only thing I can think of is that jar files are zip files and working with zip files (or other compressed archives) has always seemed painfully slow under Parallels. I'm not sure why. Maybe decompression normally takes advantage of CPU features that aren't accessible under Parallels? Or the anti-virus program slowing it down? (I'm using Microsoft's anti-virus.)

The next step is to try it on my Windows machine at work to eliminate the Parallels issue. Or to make a simple Java client so I can try it client-server all on OS X without Windows or Parallels.

And if it's really that slow, then it's time to do some more profiling and see what I can do to speed it up.

Getting Past “Good Enough” eBooks

Getting Past “Good Enough” eBooks: Liza Daly | Digital Book World

Now I have a Kindle I'm noticing the same kind of things, but I have to shake my head - this stuff is so basic!

Another example is that section headings frequently show up at the very bottom of the page. We were fixing this problem in nroff 30 years ago!

Of course, software is no better, people keep making the same mistakes we've known were bad for decades.

My New Kindle eBook Reader

I just got an Amazon Kindle (the 6" global). Again I'm amazed by the speed of delivery these days. I ordered Friday afternoon and it was delivered Monday morning. (Despite an email that said delivery would be delayed by "extreme" weather.)

Several people have asked why I didn't wait for the Apple iPad. Of course I want an iPad. I love my iPhone and I'd love a similar device with a bigger screen. But ...

• The iPad is supposed to have 10 hours battery life - that's great for a laptop, not so good for an eBook reader. The Kindle, with it's E Ink display will last a couple of weeks. That's critical for travel, especially off the beaten path and away from power (e.g. trekking).
• The iPad isn't available yet, and it's unclear when it will be available in Canada. I could be waiting forever if I keep waiting for the next great thing (I've been waiting a while for the Kindle).
• As far as I can tell, Amazon has the best selection of books. That's what stopped me from buying the Sony eBook reader, which was available sooner in Canada, and in some ways is nicer than the Kindle. Apple hasn't been saying much about their books  - how much selection they'll have and what they'll have in Canada.

I'm a committed reader. In the past I've bought tons of books. Now I have a basement full of literally 1000's of books, most of which I'll never look at again, but which I still have a hard time parting with. My office at work has stacks of books everywhere. And most computer books are out of date and worthless after a year or two. I have nowhere to put more books!

On top of that, it doesn't seem environmentally friendly to be consuming all those trees and all the resources that go into converting them to physical books.

I've been trying to use the library more, but it's frustrating. Their software sucks. (They just upgraded to a new system and it sucks almost as bad as the old system.) They don't get new books very quickly. They don't get most computer books at all. And you never know when you'll get a book you reserve. It might be tomorrow, it might be next year. And it's no good for books you might want to refer back to. I'll continue to use the library, but mostly for fiction and general non-fiction where I don't care too much when I read it.

Sony has an interesting feature where you can borrow eBooks from your library, but only certain libraries support this (not ours) and there are various limitations. Sony also lets you lend your books to other people with Sony readers - I would really like this feature.

Kindle positives:

• nice display, very easy to read, good for reading outside in bright light
• small, light-weight
• great battery life
• ability to buy books right on the device (don't need to download to your computer and then sync like the Sony)
• ability to create bookmarks and add notes
• ability to read free content in pdf or mobi format
• much more convenient to carry around the Kindle rather than a bunch of books
• free 3G wireless (cell phone) access (but only to buy books from Amazon)

Kindle negatives:

• no back-light (Shelley loves being able to read on her iPod Touch at night)
• slow display - ok for turning pages, but not so good for user interface
• no touch screen (I'm so used to my iPhone I keep trying to tap links on the Kindle)
• no wifi
• no Wikipedia access in Canada
• no web browsing in Canada
• no pictures in magazines in Canada
• no color (black and white display)
• limited memory and no way to expand it (e.g. for an offline copy of Wikipedia)
• you have to buy books from amazon.com (US), but my wish-list and gift cards are on amazon.ca (Canada)
• too easy to buy books ;-)

How "green" is a Kindle? That's a tough question that depends on a lot of factors. Manufacturing the Kindle isn't green. And it's another device that will likely be out of date and discarded within a few years. But if I avoid buying several hundred books a year, that's a fair saving. Of course, the library is the greenest option, but not a very functional one.

I debated whether to buy the small or large (DX) Kindle. The large one would be a lot better for reading PDF's that you can't reformat, but I went with the smaller one for ease of carrying.

It's too bad there isn't a better display that combines the battery life and readability of E Ink with the color and speed of a laptop display. The One Laptop Per Child computer had an interesting display with a low power black and white mode (like E Ink) but also a color mode. But no one else seems to be pursuing this kind of display.

The other big negative to all this is that the books you buy are tied to a specific device/company. If I buy books for my Kindle I can't switch to the Sony reader and vice versa. This is mostly because of the DRM copy protection. Imagine if music was tied to a specific player instead of being able to play your mp3's anywhere. (However, Amazon has software that lets you access your Kindle books from your PC, Mac, iPhone, or iPod Touch.)

Thanks to the Weave Team!

I just installed the 1.0 release of Mozilla Weave and I was excited to see this in the Release Notes:

If you use a master password, Weave Sync will automatically connect after you enter in your master password. Weave Sync will stay disconnected until you enter your master password or you choose to manually connect.

It drove me crazy to have the request for the master password come up every time I started Firefox, and it totally baffled anyone else using my computer (you can just cancel, but they would think they couldn't use Firefox without the password)

I had actually disabled Weave on most of my computers because of this. But that was a pain because then my passwords weren't synced. Now I can re-enable it.

Now if only it would just sync my add-ons and work with my iPhone ...(users are never satisfied!)

The Magic Mouse

I finally got the new Apple Magic Mouse. It came out before I went on holidays in mid December, but no one had them in stock. As you'd expect, it's very slick looking.

So far I'm pretty happy with it. My biggest complaint about the previous Apple Mighty Mouse was that the right click was really erratic. Not a big deal on OS X since it doesn't use right click too much, but a major issue when running Windows (under Parallels). Right click on the Magic Mouse seems much better. The other less serious, problem with the Mighty Mouse was that the scroll ball would get dirty and quit working, but it became second nature to clean it (by running it around on your hand or pants).

My only complaint is with the left and right swiping for horizontal scrolling - it's a little too easy to trigger by mistake. Again, that doesn't matter most of the time because there's nothing to horizontally scroll. But in Adobe Lightroom I keep switching images by mistake. And it's touchy enough that it zips through half a dozen images, making it a nuisance to get back to the one you were working on.

I tried installing Better Touch, a free program that lets you tweak all kinds of Magic Mouse and touchpad stuff. But for some reason it doesn't seem to work with Lightroom. Maybe Lightroom is using a different interface to the mouse. There doesn't seem to be any way to adjust the behavior in Lightroom either. I'd be happy just to turn off horizontal scrolling by left and right swiping - it's not something I need a lot.

I'm still getting used to the scrolling with momentum. It's a bit like the iPhone/Touch. I've been catching up in Google Reader and it seems to work well there.

Another minor issue was the lack of a middle button. On the Mighty Mouse you could click on the scroll ball to get a middle click. I had this configured to trigger Expose since I had the function keys set to be function keys (for Suneido on Windows on Parallels). Better Touch solves this since it lets you add a middle click.

Why Git is so fast (and JGit isnt)

Some interesting comments on performance in Java

'Re: Why Git is so fast (was: Re: Eric Sink's blog - notes on git,' - MARC

More jSuneido Slogging

Another long tedious day working on getting more of our application tests to run on jSuneido.

These are large scale functional tests, so when one fails it's not easy to figure out why. It could take 5 minutes, it could take 5 hours.

I've found a couple of small bugs in jSuneido. One was because BigDecimal differentiates 1 from 1.0 from 1.00, which makes sense from a scientific precision viewpoint, but not when you're dealing with money. And the problem was actually even more obscure - it was because it differentiates 0 from .0 from .00

But the rest of the bugs (the majority) have been in our application code, either in the tests or in the code itself. Nothing serious, most of them were inadvertent dependencies on the order of unordered things.

But it's frustrating. It would be tedious enough doing all this testing to find bugs in jSuneido. But when I'm doing it to find other people's bugs it's annoying. And of course, as with any large body of code, a lot of it is confusing, hard to understand, and could be improved. (Don't get me wrong, I tend to think the same about my own code.)

Oh well, it's got to be done. Hopefully it doesn't take me too much longer.

Systems that Never Stop

An interesting (and entertaining) talk by Joe Armstrong (the principal inventor of Erlang) about writing fault tolerant systems. Well worth watching.

InfoQ: Systems that Never Stop (and Erlang)

A Debuggers Life

Another day of debugging, although with a twist - I found as many bugs in our application code as I did in jSuneido. Just minor stuff - there's nothing like multiple implementations of a language to flush out the edge cases.

It seems like a slow process, but the jSuneido bugs do seem to be getting smaller and more obscure, which gives me a certain amount of confidence that the main stuff is ok. Most stuff just works, which is a vast improvement over not long ago.

Software That Fixes Itself

Technology Review: Software That Fixes Itself

Cool but a little scary - will the software start to evolve?

jSuneido Socket Server

Up till now I've been using Ron Hitchens NIO socket server framework. It has worked pretty well, but it's primarily sample code. As far as I know it's not in production anywhere and not really maintained.

The first problem I ran into with it was that it didn't use gathering writes so it was susceptible to Nagle problems. I got around that with setTcpNoDelay, although that's not the ideal solution.

Another problem I ran into was that the input buffer was a fixed size. And worse, it would hang up in an infinite loop if it overflowed. To get around this I made the buffer big, but again, not an ideal solution.

And lastly, everything sent or received had to be copied in or out of buffers maintained by the framework. (rather than used directly)

So I decided to bite the bullet and write my own. It took me about half a day to write. It's roughly 180 lines of code. It's not as flexible as Ron's but it does what I need - gathering writes, unlimited input buffering, and the ability to use the buffers directly without copying. It's fairly easy to use - there's a simple echo server example at the end of the code. I wouldn't want to have to write it with just the Sun Java docs to go by, but with the examples in Ron's book, Java NIO, it's not too bad.

Of course, there may still be bugs in it, but it seems to work well so far.

jSuneido Back on Track

After my last post I spent a full day chasing my bug with very little progress. Around 7pm, just as I was winding down for the day I found a small clue. It didn't seem like much, but it was nice to end the day on any sort of positive note.

This morning, using the clue, I was able to find the problem. It didn't turn out to be a low level synchronization issue, it was a higher level logical error, although still related to concurrency. That explained the consistency in the error. I had missed one type of transaction conflict, and that meant under certain circumstances one transaction would overwrite another. The fix was easy (two lines of code) once I figured it out.

Even with the clue, it wasn't exactly easy to track down. I ended up digging through a 100,000 line log file. Luckily I wasn't just looking through it, I was searching for particular things. It was a matter of finding the particular 50 lines where the error happened. After that it was fairly obvious.

Since fixing the bug I've run millions of iterations of a variety of scenarios for as long as 30 minutes with no problems. This evening I'll let it run for a couple of hours. I'll also think up some additional testing scenarios - there are still a few things that I'm not exercising.

Cleaning up the code before sending it to version control I found an entire data structure (a hash map of transactions) that wasn't being used! I was carefully adding and removing from it, but I never actually used it. I must have at some point. So I removed it and everything worked the same. Humorous.

I don't want to be overly optimistic, I'm sure there are still bugs (there always are), but it's starting to feel like I'm over the worst of it.

Offsite Sync and Backup

I have a large amount of music (~30gb) and photo files (~300gb). I back them up to my Time Capsule but that wouldn't protect me if my house burnt down. (Photo files from my Pentax K7 are 20mb each and I might take 10,000 in a year - that's 200gb added per year.)

So for an off-site backup, and so I can access them I keep a "mirror" copy on my computer at work. Currently, I update this mirror manually periodically, by copying new files to a portable hard drive and carrying that to work. But this is an awkward solution, and I don't update as often as I should.

There are a variety of backup and sync products out there, but none of them seem to handle this scenario.

I have been using Dropbox to sync my jSuneido files between home and work and laptop and it works really well. But their biggest account is 100gb.

Google's storage is getting cheaper, but Picasa won't let me store my big DNG (raw) photo files.

Jungle Disk has no limit storage, but at $.15 per gb that's roughly $50 per month, which isn't cheap.

Apart from the cost, the big problem with online storage is that uploading 300gb takes a long time. I signed up for Jungle Disk but it estimated 60 days to upload my files! Obviously, after that I'd only have to upload new files, but even a few thousand photos from a long holiday will take days or weeks to upload. Maybe I need a faster internet connection!

CrashPlan has a really interesting approach of letting you backup to other machines, either your own or your friends. This avoids the cost of storage. The upload speed may be better since the machines are local and aren't servicing other users. But CrashPlan doesn't sync, so I'd have an off-site backup, but I couldn't access the files (without restoring them). Another problem with CrashPlan is it requires both machines to be turned on at the same time. But to be environmentally friendly, I try to turn off my computers when I'm not using them.

Note: Jungle Disk only recently added sync and from their forum it sounds like it has problems.

A Proposed Solution

Here is an idea for a new service.

I don't really need a copy of my files in the cloud. If I could sync between my home and work computers that would be sufficient. I don't really want to be paying $50 per month just to store my files in the cloud.

All I really need to store in the cloud is a "summary" of my files (e.g. file names, dates, sizes, maybe hashes) plus any new or modified files. Once the files have propagated to my computers they can be removed from the cloud. If you used a clever hash scheme you keep even do partial updates of large files. (Although for music and photos this isn't that important since the files don't usually change.)

This would require far less storage than keeping a complete copy in the cloud.

You'd still have the problem of the initial syncing. But that could either be done by a different method e.g. a portable hard drive like I've been using, or by requiring both computers to be running at the same time for the initial sync. This is similar to Amazon allowing you to send them physical media to load data into S3. And if you had a big addition of files (like the photos from a long holiday) you could use an alternate method to move them around, and the sync could recognize that you already had the same files on each computer.

The businesses that make money from selling storage probably wouldn't be crazy about this idea, but it seems like a natural addition to CrashPlan since they aren't charging for storage, and charging for the sync service would be additional revenue. And presumably it could be cheap since the storage and bandwidth needs are minimal. (The actual data would be transferred peer to peer.)

You could even borrow some ideas from Git - their "tree" of hash values would work well for this, and also provides security and error checking.

If I had some spare time it would be a fun project. If anyone out there wants to implement it, you can count me in as your first customer :-)

Immutable and Pure

More and more I find myself wanting a programming language where I could mark classes as immutable and functions as pure (no side-effects) and have this checked statically by the compiler. Being able to mark methods as read-only (like C++ const) would also be nice.

This is coming from a variety of sources:
- reading about functional languages like Haskell and Clojure
- working on concurrency in jSuneido (immutable classes and pure functions make concurrency easier)
- problems in my company's applications where side-effects have been added where they shouldn't

I have been using the javax annotation for Immutable, which in theory can be checked by programs like FindBugs and that's a step in the right direction.

There are a lot of new languages around these days, but so far I haven't seen any with these simple features. Of course, in a "true" functional language like Haskell, "everything" is pure and immutable (except for monads), so this doesn't really apply. But I think for the foreseeable future most of us are going to be using a mixture.

To Laugh or To Cry?

I sat down this morning to write more concurrency tests for jSuneido, fully expecting to uncover more bugs. Amazingly, everything worked perfectly. I have to admit I was feeling pretty darn good, I was almost ready to claim victory. But as the saying goes, pride goes before a fall.

It was time for coffee so I figured I might as well let the tests run for a longer period. I came back to find ... the exact same error I've been fighting for the last week or more! I wouldn't have been surprised to uncover different bugs, but I could have sworn I had squashed this one.

It's bizarre that I keep coming back to this exact same error. I would expect concurrency errors to be more random. Even for a single bug I would expect it to show a variety of symptoms. I guess I shouldn't be complaining, consistency is often helpful to debugging.

I've obviously reduced the frequency of occurrence of the error. I just hope I can get the error to occur in less than 10 minutes of testing. Otherwise it's going to be a very slow debug cycle and I'll have lots of time to review the code!

So am I laughing or crying? Mostly laughing at the moment, but ask me again after I've spent a bunch more hours (or, heaven forbid, days) struggling to find the problem.

How Can This Be Acceptable?

I recently downloaded the latest version of the Scite programming editor. And subsequently, every time I ran it I got Windows security warnings. There's a check box that implies it will let you stop these warnings, but as far as I can tell it has no effect. I have no idea why the previous version ran without any warnings.

I eventually got these instructions to work:

1.. Right-click the file and select Properties.
2.. Click on the Security tab.
3.. Click Advanced in the lower right.
4.. In the Advanced Security Settings window that pops up, click on the Owner tab.
5.. Click Edit.
6.. Click Other users or groups.
7.. Click Advanced in the lower left corner.
8.. Click Find Now.
9.. Scroll through the results and double-click on your current user account.
10.. Click OK to all of the remaining windows except the first Properties window.
11.. Select your user account from the list up top and click Edit.
12.. Select your user account from the list up top again and then in the pane below, check Full control under Allow, or as much control as you need.
13.. You'll get a security warning, click Yes.
14.. On some files that are essential to Windows, you'll get a "Unable to save permission changes. access is denied" warning and there's nothing that you can do about it to the best of my knowledge.
15.. Reconsider why you're using Windows.

At my count, that's 7 levels of nested dialogs. And my name didn't show up in the list for step 12 so I had to Add "APM\andrew" (obviously, users would know to type that). Who designs this stuff? Who reviews it? Microsoft is supposed to hire all these really smart people, but they still seem to produce a lot of stupid stuff.

jSuneido Success

As I hoped, once I had a small failing test it didn't take too long to find the problem and fix it. It didn't make me feel too stupid (at least no more than the usual when you figure out a bug) since it was a fairly subtle synchronization issue. Have I ever mentioned that concurrency is hard?

The funny (in a sick way) part was that after all that, I still had the original problem. Ouch.  Obviously, the problem I isolated and fixed wasn't the only one.

Pondering it more I realized that the bugs I'd been chasing were all originating from a certain aspect of the design. And I realized that even if I managed to chase them down and squash them, that it was still going to end up fragile. Some future modification was likely to end up with the same problem.

So I reversed course, deleted most of the code I wrote in the last few days, and took a simpler approach. Not quite as fast, but simplicity is worth a lot. It only took a half hour or so to make the changes.

Amazingly, all the tests now pass! It took me a minute to grasp that fact. What does that mean when there are no error messages? Oh yeah, that must mean it's working - that's weird.

I'll have to write a bunch more tests before I feel at all confident that it's functional, but this is definitely a step in the right direction. I feel a certain amount of reluctance to start writing more tests - I'd like to savor the feeling of success before I uncover a bunch more problems!

The Joy of a Small Failing Test

Up till now I could only come up with small tests that succeeded and large scale tests that failed.

What I needed was a small test that failed. I finally have one. And even better, it actually fails in the debugger :-)

It's not so easy to come up with a small failing test because to do that you have to narrow down which part of the code is failing. Which is half the challenge, the other half is to figure out why it's failing.

At least now I feel like I have the beast cornered and it's only a matter of time before I kill it.

The test is simple enough that I look at it and think "this can't fail". But it is failing, so obviously I'm missing something. I just hope it's not something too obvious in hindsight because then I'll feel really stupid when I find it.

jSuneido Progress

The good news is that I've fixed a number of bugs and came up with a reasonable (I think) solution for my design flaw. The solution involved the classic addition of indirection.^[1] Of course, it's not the indirection that is the trick, it's how you use it.

The bad news is that after I'd done all this, I was still getting the original error! It only occurs about once every 200,000 transactions (with 2 threads). (Thank goodness for fast computers - 200,000 transaction only takes about 5 seconds.) Frustratingly, it doesn't happen in the debugger. With this kind of problem it's not much use adding print statements because you get way too much irrelevant output. A technique I've been finding useful is to have each transaction keep a log of what it's doing. Then when I get the error I can print the log from the offending transaction. It's not perfect because with concurrency problems you really need to see what the other thread was doing, but it's better than nothing.

It was also annoying because it was the end of the day so I had to leave it with a known error :-(

Thinking about it, I realized I had rushed coding some of the changes, hadn't really reviewed them, and hadn't written any tests. Not good. When I went back to it this morning, sure enough I had made mistakes in my rush job. Obviously, that self imposed pressure to get things resolved by the end of the day is not always a good thing.

So now I'll go back and review the code and write some tests before I worry about whether I've fixed the original problem.

1. A famous aphorism of David Wheeler goes: All problems in computer science can be solved by another level of indirection;. Kevlin Henney's corollary to this is, "...except for the problem of too many layers of indirection." - from Wikipedia

jSuneido Multi-Threading Issues

It didn't take much testing to find something that worked single-threaded but failed multi-threaded.

I was expecting this - I figured there'd be issues to work out.

But I was expecting them to be hard to track down and easy to fix and it turned out to be the opposite - easy to track down but hard to fix.

The problem turned out to be more a design flaw than a bug. I've thought of a few solutions but I'm not really happy with any of them.

Oh well, I knew all along this wasn't going to be easy. It'll come.

IntlliJ IDEA Goes Open Source

I recently learned that IntelliJ has released a free, open source community edition of their IDE.

IntelliJ is one of the main IDE's along with Eclipse and Netbeans. I hadn't looked at it much because the other two are free, but it does get some good reviews. (Apparently they did offer free licenses to open source projects but I wasn't aware of that.)

I tried downloading it and installing it and had no problems. It comes with Subversion support "out of the box" and I was easily able to check out my jSuneido project. That's more than I can say for Eclipse where it's still a painful experience to get Subversion working (at least on a Mac). IntelliJ proves that it is possible to do it smoothly.

I haven't had time to play with it much yet. My first impression was that the UI was a little "rougher" than Eclipse. I can probably tweak the fonts to get it a bit closer. Maybe it's due to Eclipse using SWT. (I'm not sure what IntelliJ is using.)

IntelliJ is known for their strong refactoring. To be honest, I only use a few basic refactorings in Eclipse (like rename and extract method) so I don't know if this would be a big benefit. I should probably use more...

IntelliJ is also supposed to have the best Scala plugin. I'll have to try it. I tried the Eclipse one but wasn't too impressed with where it's at so far.