Monday, August 12, 2013

A Recurring Lack of Assertiveness

We recently ran into a problem with jSuneido - when you loaded a certain customer's dumped database it would load successfully but then fail the consistency checks.

After looking at where and how it was failing (a negative size value), I figured the problem was probably that the database had a single table larger than 2 gb and I was using a 32 bit int to store the size of commits so it was wrapping around. Normally commits wouldn't be anywhere near that large, but when bulk loading each table is written as a single commit.

I felt pretty good about finding the problem so quickly and easily.

I put asserts into the code to confirm that this was the problem. But they didn't fail. Hmmm... maybe I put them in the wrong place. I added more asserts in other places. They still didn't fail.

So I fell back on the age old debugging method of inserting print's. Of course, it's a big database with a lot of tables so there was lots of output. I skimmed through it and couldn't find a table bigger than 2gb.

So much for finding the problem quickly and easily!

The next day, at home, I continued working on it, partly just trying to remember how the database code works! On the positive side, I updated some comments and diagrams while I was at it.

Eventually, I ended up full circle, finding that there was indeed a table bigger than 2gb and my original guess about the problem was correct! Argh! (I'd missed it when I skimmed through the prints.)

The problem was that I didn't have assert's enabled, which is why the asserts I added didn't fail. I've been burnt by this before - see Don't Forget to enable Java assert and Burnt by Java assert Again. You'd think I would learn.

Part of the problem is the way Eclipse works. You can set options on JRE's, but when you update to a new version of Java, then you have to remember to set the options again (which I had forgotten to do, both at work and at home). It's too bad there isn't a way to set options that are common to all JRE's.

You can also set options in Eclipse launch configurations, but I have a ton of them, and again (AFAIK) there isn't a way to set default options that are common to all launch configurations.

I thought I had good defenses in place for this. I have a test which confirms that assert is enabled. But I'm using Infinitest to run my tests automatically and it must enable asserts itself. So unless I run the test manually, it's useless for confirming that I have asserts enabled.

I also enable asserts programmatically in the start-up code. But while I was testing I was running specific classes and bypassing the start-up code.

I'm not sure what else I can do to defend against this. Any suggestions?

No comments: