Programmers Need Supervision

I was using Windows on my Mac for the first time in a while. The first frustration is that when you haven't used Windows for a while it wants to download and install massive updates, in my case include Vista Service Pack 1. Not everyone leaves their machine running all the time! And when you start it up to do some actual task, the last thing you want to do is wait for updates to install.

Next I tried to open a PDF help file and Adobe Reader wanted to update itself. Except the update failed saying I didn't haven't sufficient permissions, even though I was logged in as Administrator. That wouldn't have been so bad except the failed install left me without a working Reader, neither old nor new.

So I downloaded the latest Adobe Reader 9 and installed that (after rebooting in hopes of cleaning up whatever mess the last install might have left)

This install succeeded (I guess I did have sufficient permissions after all!) But the process was somewhat amusing. For example one of the stages was "Creating Duplicate Files". That seems like a strange thing to do, let alone advertise. Most of the messages it displayed looked more like debugging than anything else. Do I really need to see every registry key it's creating? Even if I did want to see, it flashes by too fast to read anyway. Write it to a log file if you really feel someone might want to look at it.

Then it sat on 99.90 percent complete for a long time. That's typical, but the funny part is that it wasn't 99% done or even 99.9% done, it was 99.90% done. Even more amusing because that 0.10% took about 10% of the install time. Why even display a number? It's such a rough estimate that the progress bar is more than sufficient.

And this is from Adobe who probably have some huge number of people who architect, design, review, and test this stuff. How come none of them questioned all this debugging output?

Someone really needs to supervise us programmers!

Mac TV and iTunes movie rentals

After I got my new iMac I decided to try using my old Mac Mini as my DVR & DVD player.

I bought an Elgato eyetv 250 cable tuner/converter. The tuner/converter works well, but the remote that came it seems cheap and isn't very reliable. You have to hit the buttons multiple times and point it exactly right, even though I'm only about 6 feet from the TV. And many of the buttons on the remote are either labelled very cryptically, or not labelled at all. (just colored!)

One of the reasons I've resisted digital cable and set-top boxes is that you end up (in most cases) with a single "tuner" and the only way to change channels is through the box. With my old fashioned analog cable I have a tuner in the tv and one in the eyetv so I can record one channel and watch another. And since eyetv has it's own tuner it can change channels to record the right one. (Much more difficult with a separate set-top box.)

I'd heard good things about Elgato's eyetv software. I can't say I was too impressed. It works, but the user interface is the usual confusing maze. But I haven't used any other software so it could still be better than the others.

Although the form factor of the Mac Mini is ideal for this, it soon becomes obvious that it isn't what the system is designed for. I don't want update notices popping up in the middle of a movie. And although you can do just about everything without a keyboard, you tend to get pop up warnings about no keyboard connected. And although a wireless mouse and keyboard are pretty much essential, the battery life sucks. So your movie also gets interupted by low battery warnings!

I wanted to use the new tiny wireless aluminum keyboard, but it was quite a struggle to get it to work with the mini. Eventually I got it connected.

Meanwhile, Shelley has yet to try using this setup. I don't blame her - booting up the Mac, getting into the software, etc. is not trivial. At least she can still use the tv as usual.

There are other advantages to having an internet connected tv. It's nice to be able to watch videos from the internet on tv instead of sitting at my computer. And I have access to my music and photo library. Although, I don't have it connected to my stereo so the sound quality is crappy. But if I connect it to the stereo then I'd have to turn that on to watch tv.

Another motivation for all this was iTunes movie rentals. I was excited about this when it was announced, but then it was US only and it was slow coming to Canada. Recently I went to check and found it had come to Canada back in June. Strange they didn't make a bigger deal out of it - I hadn't heard anything and I was looking for it.

My first experience with iTunes movie rentals was pretty much perfect. I went into iTunes, picked a movie, clicked Rent, and started watching. iTunes already has my credit card into so it was a painless process. The quality was great, indistinguishable from a DVD (on my ancient tv), and there were no pauses, lags, or choppiness. The price is about the same as local video rental ($3.99 or $4.99) but I don't have to pick it up or return it. I was sold.

Unfortunately, my second experience was not so good. For whatever reason, the download rate was incredibly slow. And since the download was slower than watching speed it was impossible to watch and download concurrently. It still worked and I watched the movie the next night, but the instant gratification was gone.

I'll try again and see which experience is typical. The nice part is that I don't have to sign up or make any commitment, I can just use it when/if I feel like it.

Although Mac Mini's are relatively cheap, it's still an expensive DVR/DVD player. It's definitely a place for a special purpose device. The Apple TV box is close, but it has no DVD player and it has no facility to connect to regular TV. I'm sure these were deliberate choices, but it means you still need other devices as well. I want a single device.

I have mixed feelings. It's definitely not the ideal setup, and it pretty much requires a geek to operate. But at the same time I've become hooked on some of the features it provides.

jSuneido Progress

I have to admit my sense of urgency for this project is fading a little bit. Don't get me wrong, I'm still committed to it and working steadily on it. But it's just a little too big to push through to completion on that initial excitement.

On the positive side, I pretty much have the database query optimization working. That's a big step forward since it's one of the most complex parts of Suneido. It's gone better than I expected. But I think my expectations are a little more negative than they should be, given that I'm just porting working code.

As I suspected might happen, I've found a few bugs in the existing C++ code. A few times I thought I'd found major bugs but it turned out I was just not understanding how it was working. The few real bugs I've found haven't been very serious, which makes sense given that the current version has been pretty heavily tested and used in production.

The next step is query execution. This is a lot simpler than the optimization and should go fairly quickly. It'll be interesting to run some benchmarks and see how performance compares.

What's left after that? I have to hook up the server to the database and do the crash recovery - that should just about complete the database server part. Wrong - I still have to do the multi-threading - ouch. Then comes the fun part of implementing the language. This is only needed in the database server for rules and triggers - the bulk of what the database server does doesn't need the language at all.

It would be nice to get the bulk of this done before I go on holidays for a month in September. Not that I'd want to deploy it just before going on holidays, but the more complete it is, the less I'll have to try to remember where I was at when I get back.

Of course, the next obvious project is to port the client side. The big challenge here is the user interface. This is much less well defined and intertwined with Windows so it will be a much messier job. But also in some ways more rewarding since UI is a lot more visible than the behind the scenes server. So I won't have any shortage of challenges for the foreseeable future!

Inline PDF for Mac Firefox 3

One of my last annoyances with Firefox on the Mac was that it didn't display PDF's "inline" - in the browser window. Instead, you had to download and open it in another program.

It wasn't just Firefox either, originally Safari had issues with this as well.

It always seemed odd to me because OS X is otherwise very PDF "friendly".

It bugged me again today so I searched to see if there was a solution yet and I found the firefox-mac-pdf plugin.

Of course, the first time I tried it I got a blank screen :-( but the next one I tried worked. The one that wouldn't work is landscape - maybe that's the problem, although I couldn't see any known problems with landscape.

Strangely, Gmail still forces you to download PDF's to view them. (I think by setting the Content-Disposition header.) I'm not sure why they do this. Maybe because otherwise people can't figure out how to download because they don't think to right-click and save. You could always have separate download and view links, but that only makes sense if you know the browser can handle viewing, which is probably difficult to determine.

Still, it's nice to finally have this.

jSuneido - progress and Antlr

I now have the main database query language parsing to syntax trees (using Antlr). This is a good step because it includes all the expression syntax. Next, I'll work on the query optimization, and then query execution.

Antlr is working pretty well. One thing I don't like (but it's not unique to Antlr) is that the grammar gets obscured by the "actions". It's too bad, because a big reason for using something like Antlr is so the grammar is more explicit and easier to read and modify.

For example:

cols : ID (','? ID )* ;

turns into:

cols returns [List list]
    @init { list = new ArrayList(); }
    : i=ID { list.add($i.text); }
      (','? j=ID { list.add($j.text); }
      )* ;

(There may be easier/better ways of doing some of what I'm doing, but I think the issue would still remain.)

It would be nice if you could somehow separate the actions from the grammar, similar to how CSS separates style from HTML.

This might also solve another issue with Antlr. To write a tree grammar to process your syntax tree you copy your grammar and then modify it. Sounds like a maintenance problem to me.

Similarly, Suneido's language has pretty much the same expression grammar as the database query language. But I'm going to have to copy and modify because the actions will be different. (For queries I build a syntax tree, but for the language I generate byte code on the fly.) Which means if I want to change or add something, I have to do it in multiple places. Granted, that's no different than the handwritten C++ parsing code, but you'd like a better solution from a sophisticated tool.

Mylyn - task management Eclipse plugin

I just watched an interesting talk on the Mylyn plugin for Eclipse.

It looks great, which is funny because before now I didn't really know what it was, so I've been doing my best to hide/close/disable all the Mylyn stuff in Eclipse. I'll have to look at starting to use it.

As usual, when I see these great tools, I want to add them to Suneido. If only there were more hours in the day!

jSuneido - progress and premonitions

For "fun", I decided to hook up the database request parsing and execution to the server code. It was pretty easy. I could then telnet into the server and create/modify/delete tables in the database. Can't actually do anything with the resulting tables, but still a good milestone.

On a big project, it's great when you can hook up some pieces and have them actually do something. Most of the time you're down in the bowels of the code plugging away at little pieces of functionality. It's nice to get reminded that the whole system is actually going to do something!

I've made good progress in the last week or two. I've been averaging about 200 lines of code per day. Unfortunately, it's getting hard to measure that because the lines of code are exaggerated by things like the generated parsing and lexing code. The metrics tool I'm using doesn't seem to allow excluding some files.

However, in all this progress estimation I've been glossing over one big stumbling block - concurrency. A port of the existing implementation will not be particularly useful. I still have to make it run multi-threaded - arguably a tougher nut to crack.

I just finished "reading" The Art of Multiprocessor Programming. (I'm not sure "Art" belongs in the title of a book full of lemmas and proofs!) It's pretty dense but I found it fascinating. I haven't encountered so many novel algorithms in a long time. Of course, it only reinforced the fact that concurrency is hard, and often counter-intuitive. The chapter on Transactional Memory was really interesting. This sounds like the direction for the future. (check out a podcast from OSCON and a paper from OOPSLA)

The issue of making Suneido multi-threaded has been in the back of my mind throughout this project. Every so often I spend some time thinking about it.

Although it would be possible to just add some naive coarse grained locking to the existing code, that's not going to give good results. I'm pretty much resigned to having to do some redesign to accommodate concurrency.

Yesterday I came up with an idea for using copy-on-write for index nodes. The more I thought about it, the better I liked it. Last night I was lying in bed thinking about it. (I know, what a geek, I can't help it.) It just seemed a little too good to be true. Sure enough, I had been overlooking some critical issues. Luckily (?) I stayed awake long enough to think of another alternative. It was a good example of how concurrency (or any complex system) is hard to reason about correctly with our (mine, anyway) limited brainpower.

PS. In case it wasn't obvious, the "premonition" part of the title refers to the concurrency issue. The definition seems appropriate: "A warning of an impending event, experienced as foreboding, anxiety and intuitive sense of dread." Although I'm not crazy about the rest of the definition: "Premonitions tend to occur before disaster"!

Antlr Frustrations

I just spent a frustrating morning fighting to get Suneido's query grammar to work with Antlr.

I kept getting "no viable alternative" and "no start rule (no rule can obviously be followed by EOF)"

I searched on the web but didn't find any answers.

I made more or less random changes to my grammar and if I cut out the more complex stuff it would work but that wasn't much help.

I started trying small examples, like:

expr : m (('+'|'-') m)* ;
m : ID
| '(' expr ')' ;
ID : ('a'..'z')+ ;
WS : (' '|'\t'|'\r'|'\n')+ {skip();} ;

which is pretty much straight out of the book but exhibited the same problems.

I started to wonder if something was messed up in my copy of Antlr or something.

Eventually, I found a post that explained it. The problem is that Antlr identifies "start" rules (ones that can end with EOF) by looking for rules that are not used anywhere else in the grammar. When you have recursion on the "start" rule (expr in this case), it doesn't work. The above example could be fixed by adding:

prog : expr ;

It makes sense once you know the problem, but it certainly wasn't obvious at first. And I don't recall the book mentioning it anywhere. It seems like an obvious "gotcha". I can't be the only person to get caught by this. It would have been nice if the error message said something like "make sure your start rule is not used in another rule".

Or else just make it work. Obviously, the rule you start with should be endable by EOF, whether or not it's used elsewhere. That's probably a little tricky to handle, but compared to the complexity of Antlr it's nothing.

This kind of frustration is one of the downsides of using someone else's tools and libraries.

Various other stupid mistakes on my part added to the frustration, but eventually I got things starting to work.

Inside Lightroom

Lightroom is Adobe's digital photography software. I really like it. (And I don't "really like" much software!) It competes with Apple's Aperture, which I haven't tried. (but I'm getting tempted to get a copy to check it out)

Lately I've listened to a few podcasts from/with the Lightroom developers. Being in the software development business I find it fascinating to get a peek inside other development, especially for a product that is innovative like Lightroom. The podcast feed is at:

http://rss.adobe.com/www/special/light_room.rss

Or: http://www.mulita.com/blog/?cat=3

For another view, check out this video interview with Phil Clevenger, the main Lightroom UI designer.

HMAC SHA1, Amazon SQS, and My First C# Program

We're looking at using Amazon SQS (Simple Queue Service) for a project. One end of the queue will be a Ruby on Rails application, the other end will be our Suneido application. There is code we can use from Ruby, but not from Suneido. (The new Programming Amazon Web Services is quite helpful.)

As with S3, the issue is the authentication. Amazon web services use HMAC-SHA1. For S3 we ended up using a separate external command line program, but we couldn't find anything similar for SQS.

I looked at adding SHA1 to Suneido (it already has MD5) but using the Win32 crypto api's is very confusing.

We found a command line sha1sum and figured we could use it. In my ignorance I didn't realize the difference between straight SHA1 and HMAC-SHA1. So much for using sha1sum.

Writing (or using) something in Java was another possibility, especially since I'm in the middle of Java right now, but then we'd have to worry about whether or not our clients have the Java runtime, and which version.

With our clients, on Windows, .Net is a safer bet. I looked at Amazon's C# sample code and the HMAC-SHA1 part looked pretty simple - .Net has an HMACSHA1 class that makes it pretty easy.

So I downloaded the free Microsoft Visual C# 2008 Express and wrote my very first C# program. The source is only 25 lines long and I copied most of it from other sources. (I had some problems with installing Visual C# - it took forever and I had to restart the install several times. Not sure what was going on.)

One thing that impressed me was the size of the resulting exe - 4.5 kb! Of course, it's using the .Net framework, which is huge, but as long as our clients already have that, I don't really care. In the old days, you could write tiny programs like that, but bloated exe's have become pretty much standard.

jSuneido and Antlr

I still haven't decided whether to use Antlr to replace my handwritten lexers and parsers in Suneido, but I decided to take it a little further.

I've been working on the methods to create/modify/drop database tables, so the obvious grammar to try was the database "admin requests" (the simplest of the grammars in Suneido).

I'd already written the basic grammar when I first played with Antlr so the main task was to hook up the grammar to jSuneido. For small, simple grammars it is often more of a challenge to "hook them up" than it is to write the grammar.

My first approach was to use Antlr's built-in AST (abstract syntax tree) support. That wasn't too bad. It definitely helped to have AntlrWorks to test and debug.

But then I had to do something with the AST. For big grammars, you can write another grammar to "parse" the AST, but that seemed like overkill for this. I could have just manually extracted the information from the AST but this isn't covered in the book and there's not much documentation.

Instead, I decided to not use the AST support and just accumulate the information myself. It took some experimentation to figure out how to do this. Again, it's not an approach the book really covers. In hindsight, I'm not sure if it was any easier than figuring out how to use the AST.

One of the weaknesses with the handwritten C++ parsers is that I didn't uncouple the parsing from the desired actions. It would be really nice to be able to use the same parser for other things e.g. checking syntax without generating code. What I'm hoping to do with jSuneido is to have the parser call an interface that I can have different implementations of. So even though I don't really need this for the admin request parser I decided to try out the approach.

Once I got it working in AntlrWorks the next step was to get it integrated into my Eclipse project. I had a few hiccups along the way. One was ending up with different versions of the Antlr compiler and runtime (which leads to lots of errors).

But eventually I got it working. I have a few things left to implement but there's enough working to validate the approach.

One downside is that building jSuneido has gotten more complex. Now you need the Antlr compiler and runtime. I guess I could eliminate the need for the compiler if I distributed the java files generated by the Antlr compiler.

Deploying jSuneido will now require the runtime as well. I'm not totally happy about that, considering one of Suneido's goals is ease of deployment.

For just the admin request lexing and parsing it's probably not worth it. The next step will be the actual database query language. But the real test is how well it works for the main Suneido language. If that goes well, then it's probably worth the "costs".

If you're interested, the code is in Subversion on SourceForge. e.g. Request.g

Firefox, FTP, and Sync

I've used Filezilla for the last few years for FTP. It's got a few quirks, but it works pretty well. And it has a Mac version (and Linux) which I need.

Recently I started using the FireFTP Firefox add on. It's only at version .99 but it seems to work fine. And of course, like most Firefox add ons, it's also cross-platform.

It's another example of the "browser as a platform" trend.

The more I do on the browser, and on multiple computers (work, home, laptop), the more I want to sync my browser "environment" - mostly cookies and passwords.

Up until recently I was using Google Browser Sync to do this. Again, it had some quirks, but it mostly worked. But Google has dropped the project. They recommend using Mozilla Weave. At first, I couldn't even download Weave (maybe they got swamped with traffic?) I now have it installed (a somewhat painful process) but it takes a long time to sync - a lot longer than Google did.

It underscores how performance is a lot about perception. If they didn't display a "working" dialog it would be less painfully obvious how long it took. Unless I'm rebooting, I don't really care how long a program takes to shut down, as long as it doesn't make me watch the process. Although browser restarts (required by updates) would be slower.

Weave is only at version 0.2 so hopefully it will improve. I'll use it a while longer and see.