Write Your Own Compiler

I recently listened to a podcast of Scott Hanselman talking to Joel Spolsky.

One of the things they talked about was how Fog Creek (Spolsky's company) wrote their own compiler because it was easier than rewriting their application, which was written in an ancient version of VBScript.

I've always been a little defensive about how my company writes their applications in our own language (Suneido). At best people look at me funny, at worst they think I'm crazy. Our salesmen have to skirt around the issue because it scares people to hear you're not using big name tools. As the old saying goes, "no one gets fired for buying IBM". Nowadays you can substitute Microsoft or Oracle or Java.

So it was heartening to hear Spolsky talk about (aka defend) why they wrote their own compiler. One of his points was that writing a compiler is not that hard. People tend to think it's a big deal, but for a smallish language it's not that big a job. For example, Suneido's compiler makes up a tiny fraction (less than 1%) of the total source code in my company's main application.

I'm the first to admit that part of the reason is simply that I like developing languages (and database servers, and IDE's, and frameworks) better than I like writing applications.

But there are other reasons. Having your own platform has its costs, but it also has its benefits. It's a good way to insulate yourself from the fast pace of change in the computer industry. My company still has customers running software that we originally wrote on MS-DOS. The exact same application code is now running on Vista. There are not many commercial platforms that can claim that. Even things like Java that are supposed to be stable platforms change a lot over the years. And Microsoft changes languages and frameworks faster than you want to keep up with. (Spolsky's problem.)

Of course, that can mean you're not using the latest bleeding edge technology, but that's not such a bad thing for a business application.

More Concurrency

Coincidentally, right after I wrote my last blog post about concurrency Tim Bray started a series of blog posts about it. Here's the latest:

ongoing - Concur.next — The Laundry List

Language Connoisseur

It took me a few tries to come up with the right word. Addict? Fanatic? Polyglot? Aficionado?

I think "connoisseur" fits best - "A person of informed and discriminating taste e.g. a connoisseur of fine wines."

I love to read about different programming languages, but I wouldn't say I actually "learn" them. I've only written significant amounts of code in three languages, C, C++, and now Java. It takes years for me to feel like I'm getting "good" at a language.

But I can "taste" other languages, and like a wine connoisseur, critique them (at least in my own mind). Ah yes, made from JVM grapes, definite functional nose, aged in strong typing, goes well with concurrent meals. (Sorry, I'm stretching, but you get the idea.)

It's interesting to note that 4 of the 6 languages that I've read about recently are JVM based. That's partly because I'm interested in JVM based languages because of jSuneido. But I think it also indicates a trend. There are languages based on .Net as well but for some reason they don't seem to be as popular, or at least have as many books.

One of my big interests these days is concurrency, again partly because of having to face it with jSuneido. But it's also something we all have to face as cpu's no longer get faster, and instead we get multiple cores.

It's becoming pretty obvious that threads, mutable data structures, and locking are not the answer. They work, and with a lot of effort it's possible to write programs using them, but it's hard and error prone. I think they will become the "machine language" of concurrency. Present under the covers but not dealt with directly by most people.

So the alternatives in other languages are pretty interesting. Erlang and Scala have "actors". Clojure has software transactional memory.

Pure functions and immutable objects seem to be part of the answer. Transactional memory is promising. All three of these go together.

When I first started reading about Clojure I was confused by the references to "persistent data structures". I was thinking of persistence as saving things in a database. But that's not what they're talking about. Persistent data structures  in this sense are data structures that are optimized for making modified immutable "copies" that share representation. The classic and simplest example is a single linked list (like in Lisp). For example, when you add a new item to the front of a list the new list "shares" the old list.

Suneido strings are immutable and use persistent data structures. Unlike most languages, when you concatenate onto a string in Suneido it doesn't create a whole new string. Instead, behind the scenes, it creates a list of two strings. This is transparent - to the programmer it's just as if it created a new string.

This answered a question that had been lurking in the back of my mind. Isn't it really slow to use large immutable objects in functional languages? Now I can see how it can be done efficiently using some clever algorithms like Clojure's persistent hash map.

I'd been trying to figure out how to implement an efficient concurrent version of the schema cache in jSuneido. Suneido stores the database schema in tables in the database. But for speed it caches it in memory. The schema doesn't tend to change very often so it is well suited to making it immutable (and therefore not requiring locking). But it does change occasionally, and when it does, transactions have to see the appropriate version. (Suneido uses multi-version database transactions.) A persistent data structure is perfect for this because each transaction can have it's own immutable "copy" without physically having to copy the schema (which can be quite large).

However, there's a catch. cSuneido loads the schema lazily (on demand). This adds a twist - the schema cache is logically immutable, but physically mutable. That would require locking, which I'm trying to avoid.

Then I realized that lazy loading was to speed up start up. But jSuneido is primarily (at least initially) aimed at the server, where start up speed is not as important. So I can just load the whole schema at start up, eliminating the problem. Nice.

There is another catch. Java doesn't come with any persistent data structures :-(  and I don't particularly want to write my own.  Java doesn't even have a single linked list.

I did a quick search but I didn't find any third party libraries either.

One possibility is to use the Clojure data structures. (they're written in Java) I took a quick look at the source code and it seems like it might be feasible but I'm not sure how dependent they are on other parts of Clojure. If I'm lucky they'll be usable on their own.

Of course, adopting other code always has it's price. If there are bugs I won't know how to fix them. If Clojure fixes, updates, or improves the code I'll have to decide whether to switch to the new version.

Fun stuff! :-)

Updated Blogger Editor

If you're not using the new Blogger post editor I'd recommend switching. It's a lot nicer than the old one. I've been using it for a while and haven't noticed any problems with it.

An overview of the new post editor - Blogger Help

Burnt by Java assert Again

I was debugging a problem in jSuneido and I put in an assert that I was pretty sure would fail. It didn't.

I changed it to "assert false" just to be sure. It still didn't fail. WTF

I went back to my last blog post about this (Don't Forget to enable Java assert) because I couldn't remember how I'd fixed this. (Half the reason I blog is so I have a record for my own use!)

Oh yeah, I'd set the Default VM Arguments (to -ea). Why did that get lost?

Because I installed Snow Leopard, which finally came with a 1.6 JRE, which I'd switched to using.

Easy enough to add the setting again, but how can I avoid getting burnt by this again?

Of course, the same way you avoid a bug recurring - by writing a test.

My first attempt was the standard way to test that something throws:

        try {

            assert false;

            fail("assert not enabled");

        } catch (AssertionError e) {

        }

But that succeeded with or without the -ea setting!?  It took me a few minutes of head scratching to realize that fail also throws AssertionError so I had to do:

        try {

            assert false;

        } catch (AssertionError e) {

            return;

        }

        fail("assert not enabled");

At least now I'll know right away (or at least the first time I run the tests) if I lose assert again.

I still think the default should be to enable assert, at least for development. (The higher performance server VM could disable assert by default.) Even if Java itself didn't do this, Eclipse could.

How to Suppress Blogger Image Borders

For some reason Blogger often adds a gray border around images. Usually I don't want it. You can get rid of it by adding:

style="border: none;"

To the img tag (not the a or div tags) using Edit HTML.

Nice UI Feature on Amazon

Up till recently on amazon.ca the Add to Wishlist button would always add to your default list. If you wanted the item in a different wish list you then had to move it with a second step.

Now they've made the obvious improvement, you can pick which list to add the item to right from the start, avoiding the second step.

Buttons with pull down options are becoming quite common. I think they're a good way to handle this kind of thing.

Of course, I'm never satisfied! Next I'd like to be able to add to wish lists directly from the search results rather than having to go to the items page first.

Microsoft's Secret Tablet

Courier: First Details of Microsoft's Secret Tablet - Microsoft courier tablet - Gizmodo

The video makes it looks pretty cool. I like the ability to use a stylus as well as fingers. Some of the UI ideas look interesting.

via theAppleBlog

The Best Camera

Chase Jarvis Blog: The Best Camera: iPhone App Book Community

Marketing, but well done.

An interesting blend of iPhone app, book, and web site/service.

When Will We Get Computers That Don't Crash?

I just had Mac OS X crash totally, twice in a row. I suspect it was downloading some video podcasts in iTunes that did it (at least that's what's in common to the two crashes).

I can understand applications crashing. But surely to goodness by now we should have operating systems that can handle misbehaving applications. Don't we have hardware process isolation? Do I have to run my applications in VM's just to get sufficient isolation? (not that that is complete protection either!)

Similarly, people continue to give (and accept) misbehaving hardware as an excuse for systems crashing. Again, surely to goodness the operating system could be written to survive misbehaving hardware/drivers. Especially when most hardware interfaces in a standard way through USB. I've crashed my Mac importing photos. People say, "oh yeah, there are problems reading SD cards via USB". Why is that "acceptable"? Surely enough time should have been spent on the USB driver to make it crash proof?

And even if a device driver has obscure bugs (they always will), why is this able to take down the whole operating system?

I don't understand why the state of the art hasn't progressed further than this.

jSuneido Milestone

As of this morning, all the standard library tests run successfully in jSuneido!
(Other than a few that are not portable e.g. because they call Win32 functions.)

In some respects that's a huge milestone. In other respects it's just another arbitrary point in the progress of the project.

I'm pretty sure (I'd bet money on it) that if (when) I run other tests, e.g. the test suite from our accounting package, that I'll uncover more bugs. That's the next step but I'm procrastinating so I can enjoy the feeling of accomplishment a little bit longer.

Hopefully getting the other tests to pass won't take too long. Then I can finally tackle one of the main points of this project - multi-threading. jSuneido is currently multi-threaded but with no protection against threads interfering with each other. So using it multi-user will crash it pretty quickly.

One of the steps is to identify shared data structures and replace/convert them to concurrent versions.

But doing that naively is not going to work well because there will be too many bottlenecks that will end up serializing execution. So it'll be safe, but it won't scale.

So I'm going to have to redesign a few areas but I think I know what to do and it shouldn't be too hard.

Windows 7 on Parallels on Mac

I just installed Windows 7 (Professional 32 bit) on Parallels (4.0.3846) on my iMac (OS X 10.6.1)

The install went smoothly and relatively quickly.

You still can't use Aero (fancy effects like transparency) under Parallels but that doesn't bother me too much.

My first challenge was that I couldn't see the task bar in Coherence mode (mixed Mac and Windows). A quick Google search found other people with the same problem. Eventually I discovered that there is a menu option to show it (Applications > Show Windows Task Bar). I wonder if it wouldn't be better to show it by default and let the people who want to hide it go hunting for the menu option.

My network icon in the taskbar shows a yellow exclamation mark warning, with a tooltip of "No internet access", but I seem to be able to access the internet fine through Internet Explorer and Windows Update worked fine. When I run the troubleshooter it tells me everything is fine and asks what my problem is. I guess I just ignore the warning. Hopefully Parallels will fix this at some point.

Windows 7 has dropped the Quick Launch bar. Instead you can "pin" things to the task bar. Except that I couldn't. I tried dragging and using the right click context menu. No errors or anything, it just didn't work. More Google searching showed other people with the same problem but no clear solution. Someone suggested setting up a new user account. This didn't really make sense, but I tried it and it worked.

I guess the default initial Administrator account doesn't let you pin anything to the task bar. It would be nice if it gave you some kind of message.

If the default initial Administrator account is not a regular account, why doesn't the Windows install process create a regular account for you? Maybe because of installing via Parallels "unattended" method? I'll have to ask someone who has installed Windows 7 on a PC.

One of the few features that the Windows task bar had that the Mac OS X dock didn't was the ability to toggle between showing and hiding windows by clicking on the task bar icon. (A Windows Feature I'd Like on the Mac) Sadly, this feature seems to be gone in Windows 7. (Unless there is a way to enable it somewhere.)

I've been using Vista on Parallels, but I probably would have been better off with XP because it needs less resources. (That's why Netbooks come with XP.) I'm hoping Windows 7 will be less demanding than Vista. (It's supposed to be.)

Apple Updates

It's been a busy couple of days for me with Apple updates.

First, Apple released Snow Leopard (the new version of OS X) ahead of schedule, while I was on holidays. The day after I got back was a holiday but I headed for London Drugs who I knew would still be open. Unfortunately, they were all sold out. The next day I tried a few more places including Neural Net (our local Apple oriented computer store). They were all sold out!

Obviously the demand for Snow Leopard was higher than expected, even though there are no really big new features.

In a way the delay turned out to have a positive side. While I was waiting I decided I might as well get the "Boxed Set" which includes the latest iLife and iWork. I'd been thinking about buying iWork anyway and my Mini had an old version of iLife, so it seemed like a good deal. Even better, Neural Net had the Boxed Set Family Pack in stock :-)

Although there were some people recommending waiting to upgrade in case of problems, most people seemed to say it was ok. I updated my MacBook first and when that went smoothly, went ahead and updated my iMac and Mini. So far I haven't had any problems, but I haven't done too much.

OS X finally includes Java 6 :-) so I wondered if there'd be any glitches with Eclipse and jSuneido, but so far so good.

Coincidentally, iTunes 9 was released yesterday so I updated that on all my machines. iTunes finally has a Wish List :-) I always wondered why they didn't have this. Was it because they wanted people to buy right away? But then why would Amazon have a wish list?

The wish list is somewhat hidden. To add items you use the pull down menu attached to the Buy button. The annoying part about this design choice is that items that only have a Rent button (certain movies) can't be added to your wish list. To actually view the wish list, the only link I could find was at the very bottom of the iTunes home page under "Manage". The help describes a different location - under Quick Links on the right hand side - which seems like a better location. It's almost as if they still aren't sure about the feature so they're making it somewhat hidden.

Another major new feature in iTunes 9 is "Home Sharing" which lets you move your media between your different home computers. This should help me keep my living room Mini's music library up to date with purchases (which I mostly make on my main iMac).

You can only use Home Sharing between computers you have "authorized" for your iTunes account. (You're allowed to authorize up to 5 computers.) Originally authorization was for DRM protected music. Since I refused to buy any DRM protected music I never had to worry about authorization. Now I do. I find I have authorized 4 out of my allowance of 5 computers. At least one of those was a machine I no longer own (my old MacBook). I don't think there's any way to un-authorize a machine after the fact (you have to remember to do it before you get rid of the machine or reinstall the OS) As far as I know, the only solution Apple offers is that you can un-authorize all your machines, allowing you to re-authorize the ones you want. (But you can only do this once a year.)

After some searching I found the setting to automatically copy music purchased on other machines. I turned it on and waited. I knew I had purchased music since I last synced my library. Nothing seemed to be happening. I made sure iTunes was running on both machines. I left it for an hour in case it was a slow background task. Nope. I'm guessing that it only works for music you purchase after turning on this option. I guess it all depends how you interpret "automatic". No big deal, it was easy enough to view my iMac library, sort by date added, shift-select all the new stuff and drag it over. I'll have to wait till I purchase some new music to see if it actually syncs automatically then.

On top of all this, Apple released iPhone OS 3.1. I installed it, but there doesn't appear to be anything too exciting in it.

The other big announcement from Apple yesterday was the new version of the iPod Nano with video camera, microphone, speaker, FM radio, and pedometer (!?). I was surprised that the camera was video only, but according to comments by Steve Jobs, this was due to size/space limitations. The FM radio even has a "pause" feature like DVR's. It was nice to see Steve back up on the stage after all his health problems.

The iPod Touch (I keep wanting to call it the iTouch) is now being targeted as a game machine. I would never have predicted that, but then again, I very rarely play games so I tend to forget what a big market it is.

iPod Shuffle Won't Shuffle

I had an older model iPod Shuffle that I used for running. It started to get flaky and eventually died totally.

So I bought a new model, smaller and sleeker, and more memory.

But ... I listen to podcasts when I'm running, not music, and the new iPod Shuffle won't shuffle podcasts.

Even if it would sort by date I could live with it. But it sorts by source, and I don't want to listen to all the podcasts from one source all in a row.

And although I like the controls on the headphone wire, you have to double click to skip tracks and it is a frustratingly slow process to skip all the podcasts from one source just to get to the next source. Good luck if you want to find a particular podcast.

I started doing some research and I read that you could skip between playlists. Ok, I'll put each podcast source in a playlist and then I can skip through them. Except you can only put music in playlists, not podcasts, despite the fact that they're all just mp3 files.

Ok, I'll just move my podcasts over into my music section so I can put them in playlists. Except you can't. For some reason, iTunes goes to great lengths to prevent this. Even if you remove the file from iTunes and then try to import into the music section, it's too "smart" and puts them back in the podcast section. There are various work-arounds but I don't want to have to do this every time I get new podcasts.

Why stop you from shuffling podcasts? Sure, not everyone will want to shuffle, but that's no different than music. After all, the one and only control on the body of this iPod is whether to shuffle or not!

Why stop you from putting podcasts into playlists? Again, I can't think of any reason for blocking this.

It's probably a similar issue as with the K7 flaw - going overboard in trying to keep people on the correct path, refusing to accept that your (the designer's) idea of the "correct" path isn't necessarily the same as your users.

Judging from all the stuff on the web about this, it obviously annoys a lot of people. Come on Apple - listen to your users!

Pentax K7 Flaw (IMO)

I just traded in my Pentax K10D camera for the new Pentax K7. Overall I'm pretty happy with the upgrade, but there's one thing that really annoys me.

Both the K10D and the K7 have a "Green" mode where everything is automatic and many settings are ignored (forced to safe, default settings).

But in the K7, Green mode now forces saving the images as JPEG - RAW is not allowed.

I shoot in RAW 100% of the time - it gives me a lot more control over the final images, and using Lightroom (or Picasa) it's just as easy to handle RAW as JPEG - there are no extra steps or things to deal with.

This means I can't use Green mode on the K7. It's not the end of the world because "Program" mode can also be used fully automatically. You just have to be remember to put settings back to "normal" after changing them. On the K10D I'd use Program to do something different, but I could just flip back to Green mode without worrying about what settings I'd changed. I'd only have to worry about it when I went to Program mode. Now, staying in Program mode, I'll have to be more careful.

I'm sure Pentax had reasons for doing this, but I think they made the wrong decision. Beginners who can't deal with RAW are going to leave their camera set to JPEG. Anyone who is advanced enough to change their settings to RAW presumably did it deliberately (like me) and doesn't want it overridden by Green mode. Besides, given the cost of this camera, the market is not beginner newbies anyway.

It's a fine line between "protecting" users from shooting themselves in the foot, and being over-protective and stopping them from doing valid things. This time I think they went over the line.

Tethering iPhone to MacBook

It was so nice out today, after a less than stellar summer so far, that I decided to take my laptop and go sit outside somewhere for coffee. The spot I picked (Pacific Gallery & Cafe) doesn't have wireless so it seemed like a good time to figure out how to tether my MacBook (13" unibody) to my iPhone (3Gs) for internet access.

It didn't turn out to be so easy. First you have to enable bluetooth on both devices (I hadn't brought a cable or that might have been an easier approach). Then you pair the devices. This went ok other than a little searching to find the appropriate settings.

But after pairing successfully, you're still not connected. Pulling down the bluetooth menu from the menu bar showed the iPhone but Connect to Network was grayed out (disabled). My network preferences showed a Bluetooth PAN (Personal Area Network) but it said the "cable" (!?) was disconnected. Not very helpful. In the Bluetooth preferences the tools menu (the "gear" at the bottom) had a Connect to Network that wasn't grayed out, but it also didn't seem to do anything.

If I picked the MacBook on the iPhone the Bluetooth preferences on the MacBook would switch to connected and then immediately switch back to unconnected.

Of course, I googled for the problem. A lot of the results were about how to get around AT&T not allowing tethering. But I was on Rogers (in Canada) and they supposedly do allow tethering.

Apart from the AT&T results, there seemed to be quite a few people with similar problems, but no real consensus on a solution. Some people claimed if you simultaneously hit connect on both the MacBook and the iPhone then it would work. It didn't for me. Some people suggested removing the bluetooth devices from both the MacBook and the iPhone and re-pairing. That didn't seem to help either.

Finally, one person said to restart the MacBook. That worked! I had to laugh because when people ask me about computer problems one of the first things I always suggest is to restart. But I don't expect to have to do that on Mac OS X.

The sad part is that even after I got it working it was too slow to be usable. I couldn't even bring up Gmail because it would time out. Pinging the name server was giving a response time of 4 seconds (4000 ms)! The iPhone was showing 5 bars and a 3G connection, but obviously I wasn't getting a good connection. Browsing on the iPhone was also very slow so it wasn't just the tethering.

I'll have to try it again when I've got a better 3G connection. I'm not sure if it's going to work easily in the future or not. Some of the people reporting problems had it working for a while and then it quit so I'm not totally optimistic. Maybe using a cable will be simpler. I wonder if the dedicated USB cell "modems" work better. (I would hope I'd be able to use my existing data plan?)

New Camera with Projector

Nikon | Imaging Products | COOLPIX S1000pj

I'm not sure it's something I'd use a lot, but having a projector built in to a camera is a cool feature.

Reading the fine print, the projector is only VGA resolution which is not too impressive.

If you were using the camera to record your whiteboard, it might be handy to be able to redisplay it with the projector.

Anatomy of a feature

inessential.com: Anatomy of a feature

Interesting description of all the little details that are behind even the simplest feature, that most non-programmers have no idea of.

via

iPhone Competition

The Zii Egg looks like a pretty cool gadget.

• touch screen
• gps
• vga camera for video conferencing
• hd camera
• wifi and bluetooth
• sd card slot
• hd video output
• runs open source Google Android OS
  

So far it's not a phone, but that's probably coming.

via

Apple nabs 91% of "premium" computer revenue

Apple nabs 91% of "premium" computer revenue in June - Ars Technica

via