Lexer in TypeScript

One of the standard things I do when I'm investigating a new language is to implement Suneido's lexical scanner. Not because this is necessarily the best test of a language. It's more that I'm usually evaluating the language in terms of how it would work to implement Suneido. I realize that's a very biased view :-)

I wasn't planning to do this with TypeScript / JavaScript because suneido.js isn't intended to be an "implementation" of Suneido. It's a transpiler that runs on jSuneido (using the jSuneido lexer and parser) that translates Suneido code to JavaScript. The only manually written TypeScript / JavaScript code is the runtime support library. (And that's partly a bootstrapping process. Once the transpiler is functional much of the support library could be written in Suneido code.)

But as I implement the runtime support library, obviously I need tests. And I'm getting tired of writing the same tests for each version of Suneido. (Or worse, different tests for each version.) That's why I started developing "portable tests" some time ago. So I decided I should implement the portable test framework in TypeScript. At which point I remembered that this required a Suneido lexical scanner.

So I decided to write yet another version. I took about a day, which is about what I expected. And some of that time is still learning TypeScript and JavaScript. I based it mostly on the C# version. The Go version is more recent but Go has more language differences.

Overall, I was pretty happy with how it went. I find the repetition of "this." a little ugly. In Suneido you can abbreviate this.foo as just .foo, which is a lot more compact and less "noise" in the code, while still making it clear it's an instance variable.

I find JavaScript's equality testing somewhat baroque, the recommendation is to use === most of the time, but that won't always work for strings (because of primitive strings and object strings). And then there's the quirks of -0 versus +0, and NaN.

ES6 arrow functions and the functional methods like map are nice. And spread / rest (...) and for-of are good. I only specified types on function parameters and returns, but this worked well and gave me very little grief.

I was pleasantly surprised to find that TypeScript enum's automatically create a reverse mapping from name to number. Although the gotcha is that if you make them const you don't get this because they are inlined. Using a simple numeric enum doesn't let me attach extra information for parsing, but I'm not planning to write a parser so that's not a problem. The other issue is debugging, where all you see is a number.

Removing const from my token enum exposed another issue. I was relying on Atom's auto-compile but it only compiles the file you are editing, not the other files that may depend on it. So I went back to running tsc -w in a separate terminal window. (Open question - does anyone know why tsc lists some of my files as relative paths and some as absolute? It's consistent about which files, except that gradually more are switching to absolute. It's not causing any problems, I just can't figure out why.)

Although it can mean shorter code, I'm not a fan of "truthy" and "falsey" values. It was a source of problems 30 years ago in C. Now I got caught by it with a function that returned a number or undefined. I expected undefined to be "falsey" but I forgot that zero is also "falsey". With Suneido I took a stricter approach that things like if and while only accept true or false and throw an exception for any other type of value.

Now that I have a lexical scanner I can implement the portable test framework, which also shouldn't take me too long.

Versions, in order of age. cSuneido and jSuneido are production code and more full featured than the other more experimental versions.

• TypeScript tokens.ts, lexer.ts, lexer_test.ts
• Go tokens.go, lexer.go, lexer_test.go
• C# Tokens.cs, Lexer.cs (tests at the end)
• D token.d, lexer.d (tests at the end)
• Java (jSuneido) Token.java, Lexer.java, LexerTest.java
• C++ (cSuneido) scanner.h, scanner.cpp (tests at the end)

Going in Circles

Suneido has a bunch of built-in functions equivalent to its operators. e.g. Add for '+'  These are useful, for example, in functional style programming e.g. list.Reduce(Add)

I noticed that we didn't have a compare function i.e. that would return -1 for less than, 0 for equal, or +1 for greater than. This can be useful in some cases (e.g. in a switch) and avoids needing two comparisons. (Note: I didn't have a need for it, just noticed we didn't have it.)

A few days later I decided to implement it. It was simplest in jSuneido since there was already an internal compare function that I just needed to expose. Whereas cSuneido has the C++ / STL style of using operator < and operator == to implement all the other comparisons. So it needed two comparisons.

Then I started to question why these functions needed to be built-in. Why not just write them in Suneido code in the standard library. The difference in performance would be minor. So I ripped them out of cSuneido and jSuneido and defined them in the standard library. I also updated the documentation.

Then I started to wonder how often these built-in functions are actually used. As you can probably guess, almost never. On top of that I found several places where programmers got confused and thought they needed to use these functions instead of just the operators. (I fixed these.)

So I ended up ripping out the standard library functions I had just written, and the documentation I'd just updated.

In the few places where the functions were used it was easy to replace e.g. Add with function (x, y) { x + y }

I ended up keeping a definition for greater-than since it was used in several places to sort in reverse order. (since list.Sort! takes an optional comparison function defaulting to less-than)

So I started with adding a function, and I ended with deleting a bunch instead.

The moral of the story is to add features when you need them. (YAGNI) That certainly applied to adding the compare function, but it also applies to creating all the other ones in the first place.

And when I ported them all to jSuneido it never occurred to me to question whether they should just be removed.

Another factor is "sunk costs". After spending the time to implement the functions, first in cSuneido and jSuneido and then in the standard library, it felt bad to just delete that work. But you shouldn't let that affect decision making.

Part of my thinking was along the lines of, if we have A, B, and D, then we should have C. But that only makes sense if you had a real need for A, B, and D.

Finally, I think the reason I started this whole process was that it was psychologically attractive to do something that I could get a quick and easy sense of accomplishment from. (As opposed to large projects where it's hard to even see progress.) There's nothing wrong with that, but the choice of what to do should still be rational. i.e. pick an easy task from the ones that make sense to do.

Apart from feeling stupid both for the past and present wasted effort, in the end I'm happy to remove some code and move (if only slightly) towards simpler.

Programming in Typescript

I've been back working on suneido.js recently, which means programming in JavaScript or, my preference, TypeScript. It's been awhile so I had to get back up to speed on what tools to use. In case it's useful to anyone, here's what I've settled on (for now).

• install Node.js (current 6.3 rather than stable since I'm not in production)
• use npm to install TypeScript (currently at 1.8)

I started used Visual Studio Code, which I quite like, but a few things bugged me so I gave Atom a try and found I liked the editing experience better, especially after installing a few extra packages. One of the things I like about Atom is that it compiles Typescript automatically when you save a file, whereas (AFAIK) with Code you need to run tsc -w in a separate terminal window.

Atom packages:

• atom-typescript
• atom-beautify
• highlight-line
• highlight-selected
• simple-drag-drop-text
• Sublime-Style-Column-Select

The last three should be part of Atom core if you ask me, but as long as they're available that's fine.

Atom setting:

• tree-view - hide VCS ignored files
• autosave enabled
• show invisibles
• tab size: 4 (personal preference)

One thing I am still using Code for is debugging. Node's stack traces show the JavaScript line numbers which is a pain to relate back to the TypeScript code. I found some ways around this, but none of them looked simple. Code's debugger works well.

Previously I had also used WebStorm from JetBrains which has support for TypeScript and debugging. I may use it again, although I like the ease of using simple tools like Atom and Code.

Interestingly, Visual Studio Code is built on top of Electron, which was originally developed for Atom. And recently, Monaco, the editor component of Visual Studio Code has been open sourced.

I'm targeting ES6/2015 partly as future-proofing, and partly just because it's a better language. Most of ES6 is pretty well supported in Node and browsers, and there are tools like Babel to run it under ES5. Exploring ES6 is a useful resource. Strangely, none of the browsers support ES6 modules natively. You can still use them in the Typescript code but they need to be compiled to AMD for the browser, and CommonJS for Node (e.g. testing).

Although JavaScript has some quirks that I don't care for, overall I quite like it. And with Typescript for type checking, I don't mind programming in it at all. One thing I like about Typescript, as opposed to things like CoffeeScript is that it doesn't obscure the JavaScript.

Thankfully getting going again has gone smoother this time than my last bout of Yak Shaving

You can follow progress on suneido.js on GitHub. There's still a long way to go!

Yak Shaving

"Yak shaving" is a programmer's slang term for the distance between a task's start and completion and the tangential tasks between you and the solution. If you ever wanted to mail a letter, but couldn't find a stamp, and had to drive your car to get the stamp, but also needed to refill the tank with gas, which then let you get to the post office where you could buy a stamp to mail your letter—then you've done some yak shaving. - Zed Shaw

Yak shaving is one of my least favorite and most frustrating parts of software development. I have vague memories of enjoying the chase when I was younger, but my tolerance for it seems to decrease the older I get. Nowadays it just pisses me off.

I haven't worked on suneido.js for a while but I got a pull request from another programmer that's been helping me with it. I merged the pull request without any problems. Then I went to try out the changes.

I get a cryptic JavaScript error. Eventually I figure out this is a module problem. I see that my tcconfig.js is set to commonjs, which is what is needed to run the tests in node.js, but to run in the browser I need amd. I change it. I love JavaScript modules.

Now I need to recompile my TypeScript. I happen to be using Visual Studio Code, which follows the latest trend in editors to not have any menu options that are discoverable. Instead you need to type commands. I eventually find the keyboard shortcut Ctrl+Shift+B. (After some confusion because I was reading the web site on OS X and it auto-magically only shows you the keyboard shortcuts for the platform you're on.)

Nothing happens. I try tsc (the TypeScript compiler) from the command line but it's not found. Maybe it's just not on my path? I figure out how to see the output window within Code. It's not finding tsc either. (Although it hadn't shown any sign of error until I opened the output window.)

Do I have tsc installed? I'm not sure. Doesn't Code come with TypeScript support? Yes, but does that "support" include the compiler? I find that exact question on StackExchange but it's old and there seems to be some debate over the answer.

I try reinstalling Code. It adds to my path but still doesn't find tsc. (The ever growing path is another pet peeve. And yes, the older I get the longer my list of pet peeves is.)

What the heck was I using last time I worked on this? I'm pretty sure I never deliberately uninstalled TypeScript. I search my computer but all I find are some old versions of TypeScript from Visual Studio (not Code).

I get the bright idea to check my work computer (I'm on my home machine.) I don't find anything there either. Then I remember it's a brand new computer that I set up from scratch. So much for that idea.

I give up and decide to install TypeScript. There are two methods given, via npm (Node.js) or Visual Studio. I'm pretty sure I used Node.js before so that seems to be the way to go.

I try to run npm but it's not found either. Did I lose Node.js as well as TypeScript? No, I find it on disk. I see there's a nodevars.bat file that sets the path. I try that and now I have npm. I bask in the warm glow of accomplishment for a few seconds until I remind myself that running npm was not what I was trying to accomplish.

I run npm install -g typescript. And now I have tsc. But my path didn't change. Maybe I had tsc all along and just needed node on my path? Hard to tell now.

Then I find my tsconfig.js is wrong - it lists .ts files that don't exist because they are just .js files. Again, how did that work last time around? I fix that and it seems to compile.

And finally, miraculously (but several hours later) my suneido.js playground works!

Wait a minute, the pull request was improvements to the CodeMirror Suneido mode, it had nothing to do with the playground. No problem, I'll just start over ...

New PC Software

Once I had my new PC running, the next step was to install software. I prefer to set up new computers from scratch rather than use any kind of migration tool because I don't want to bring over a lot of old junk. It takes a little more time but I think it's worth it. And it's always interesting to review what software I'm actually using day to day, and how that changes over time.

My first install is usually Chrome. Once I sign in to my Google account and sync my browser settings I have all my bookmarks and my Gmail ready to go. I use LastPass for my passwords. I also install Firefox but it's not my day to day browser. I used to also install Safari for Windows but Apple is no longer keeping it updated.

Next is Dropbox which brings down a bunch of my commonly used files. Then Evernote which gives me access to my notes.

After that it's mostly development tools - Visual Studio Community (free) C++, MinGW C++, Java JDK, and Eclipse. The last few Eclipse installs I've been importing my addons from the previous install. But for some reason I couldn't get that to work this time because I couldn't find the right directory to import from. The directories are different since I've been using the Oomph installer. I'm sure I could figure it out, but I only use three addons so it was easier just to install them from the Eclipse Marketplace. (The three are Infinitest, EclEmma coverage, and Bytecode Outline.)

I use GitHub Desktop although both Visual Studio and Eclipse provide their own Git support. (and there's also the command line, of course.)

Although I'm not actively programming in Go these days I like to have it and LiteIDE installed.

For editors I use Scite, because it's the same editing component as we use in Suneido, and Visual Studio Code for JavaScript and Typescript. (Visual Studio Code is not related to Visual Studio. It's a cross platform application written in Typescript and packed with Electron.)

This is all on Windows, but other than the C++ tools I have pretty much exactly the same set of software on my Mac.

New PC

In many ways PC performance has leveled off. There are minor improvements in CPU and memory speed but nothing big. But there has been a significant improvement in performance with the Skylake platform supporting SSD connected via PCI Express.

Based on a little research, here were my goals:

• fast SkyLake CPU
• 32gb DDR4 ram
• 512 gb M2 SSD
• small case (no external cards or drives)
• 4K IPS monitor

And here's what I ended up getting:

• Asus Z170I Pro Gaming mini ITX
• Intel Core I7-6700K 4.00 GHz  
• Corsair Vengeance LPX 32GB (2x16GB) DDR4 DRAM 2666MHz (PC4-21300)
• Samsung 950 PRO - Series 512GB PCIe NVMe - M.2 Internal SSD
• Fractal Design 202 case
• ASUS PB279Q 27" 4K/ UHD 3840x2160 IPS

I don't do any gaming, but that was the only mini ITX motherboard I could find that fit my requirements.

The case was the smallest I could find that would fit this stuff. The frustrating part is that it could be half the size. The empty half of the case is for cards or external drives, but I didn't want either of those. It's a little hard to tell from the picture, but the motherboard is only 7" square, not much bigger than the fan. The power supply is almost as big as the motherboard.

And here's a comparison of my new SSD (top) to the old one (bottom). Higher numbers are better.

The big advantage of SSD over hard disks is the seek time for random IO. But the biggest gains here were for sequential IO. Still, some respectable improvements across the board. Of course, how that translates into actual overall performance is another question.

I try not to buy new machines too often. At least I know my old one, which is still working fine, will be put to good use by someone else in the office.

I'm not a hardware expert, here's where I got some advice:
Building a PC, Part VIII
Our Brave New World of 4K Displays

Mac Remote Desktop Resolution

At home I have a Retina iMac with a 27" 5120 x 2880 display. At work I have a Windows machine with a 27" 2560 x 1440 display set at 125% DPI. I use Microsoft Remote Desktop (available through the Apple app store) to access my work machine from home.

I'm not sure how it was working before, but after I upgraded my work machine to Windows 10, everything got smaller. It comes up looking like 100% (although that's actually 200% on the Mac).

Annoyingly, when you are connected through RDP you aren't allowed to change the DPI.

I looked at the settings on my RDP connection but the highest resolution I could choose (other than "native") was 1920 x 1080 which was close, but a little too big.

Poking around, I found that in the Preferences of Microsoft Remote Desktop you can add your own resolutions (by clicking on the '+' at the bottom left). I added 2048 x 1152 (2560 x 1440 / 1.25)

Then changed the settings on the connection to use that and it's now back to my usual size.

The screen quality with RDP doing the scaling does not seem as good as when Windows is doing the scaling, but at least the size is the same.

I'm guessing from what I saw with searching the web that there might be a way to adjust this on the Terminal Server on my Windows machine, but I didn't find any simple instructions.

If anyone knows a better way to handle this, let me know.

Native UX or Conway's Law?

organizations which design systems ... are constrained to produce designs
which are copies of the communication structures of these organizations
— M. Conway

A pet peeve of mine is applications that are available on different platforms, but are substantially different on each platform. I realize that I'm probably unusual in working roughly equally on Windows and Mac. But even if most people work on a single platform, why make the application unnecessarily different? Isn't that just more work for support and documentation.

I recently listened to a podcast where an employee mentioned that her focus was on making sure their Windows and Mac versions weren't "just" identical, but matched the native user experience. That sounds like an admirable goal. The problem is, their versions have differences that are nothing to do with native UX.

Obviously standards vary between platforms. Do you call it Settings or Preferences? And it makes sense to match the native conventions. But in general those are minor things.

But changing the layout of the screen? Or the way things are displayed? Or the text on buttons? There really aren't any "standards" for those things. And therefore "native" doesn't justify making them different.

I suspect that "matching the native UX" is often a justification for Conway's law. Different teams are developing the different versions and they do things different ways. Coordinating to make them consistent would take more effort so it doesn't happen, or not completely.

My company contracted out the development of our mobile app. The company we contract to believes in "native" (i.e. not portable) apps. They also have a process where different teams develop different versions, so they can be platform experts. The end result - our app is unnecessarily different on different platforms. And any change takes twice as much work. Not to mention having to fix bugs in multiple implementations.

It's sad that after all these years we still don't have a good story for developing portable applications.

Except that's not quite true. We have the web and browsers. And tools like Cordova and Electron that leverage that. And contrary to belief, users don't seem to have a lot of problems with using web apps that don't have native UX.

Suneido RackServer Middleware

A few years ago I wrote an HTTP server for Suneido based on Ruby Rack and Python WSGI designs. We're using it for all our new web servers although we're still using the old HttpServer in some places.

One of the advantages of Rack style servers is that they make it easy to compose your web server using independently written middleware. Unfortunately, when I wrote RackServer I didn't actually write any middleware and my programmers, being unfamiliar with Rack, wrote their RackServers without any middleware. As the saying goes, small pieces, loosely joined. The Rack code we had was in small pieces, but it wasn't loosely joined. The pieces called each other directly, making it difficult, if not impossible, to reuse separately.

Another advantage of the Rack design is that both the apps and the middleware are easy to test since they just take an environment and return a result. They don't directly do any socket connections.

To review, a Rack app is a function (or in Suneido, anything callable e.g. a class with a CallClass or instance with a Call) that takes an environment object and returns an object with three members - the response code, extra response headers, and the body of the response. (If the result is just a string, the response code is assumed to be 200.)

So a simple RackServer would be:

app = function () { return "hello world" }
RackServer(app: app)

Middleware is similar to an app, but it also needs to know the app (or middleware) that it is "wrapping". Rack passes that as an argument to the constructor.

The simplest "do nothing" middleware is:

mw = class
    {
    New(.app)
        { }
    Call(env)
        {
        return (.app)(env: env)
        }
    }

(Where .app is a shortcut for accepting an app argument and then storing it in an instance variable as with .app = app)

Notice we name the env argument (as RackServer does) so simple apps (like the above hello world) are not required to accept it.

and you could use it with the previous app via:

wrapped_app = mw(app)
RackServer(app: wrapped_app)

A middleware component can do things before or after the app that it wraps. It can alter the request environment before passing it to the app, or it can alter the response returned by the app. It also has the option of handling the request itself and not calling the app that it wraps at all.

Uses of middleware include debugging, logging, caching, authentication, setting default content length or content type, handle HEAD using GET, and more. A default Rails app uses about 20 Rack middleware components.

It's common to use a number of middleware components chained together. Wrapping them manually is a little awkward:

wrapped_app = mw1(mw2(mw3(mw4(app))))

so I added a "with" argument to RackServer so you could pass a list of middleware:

RackServer(:app, with: [mw1, mw2, mw3, mw4])

(where :app is shorthand for app: app)

Inside Rackserver it simply does:

app = Compose(@with)(app)

using the compose function I wrote recently. In this case what we are composing are the constructor functions. (In Suneido calling a class defaults to constructing an instance.)

A router is another kind of middleware. Most middleware is "chained" one to the next whereas routing looks at the request and calls one of a number of apps. You could do this by chaining but it's cleaner and more efficient to do it in one step.

So far I've only written a few simple middleware components:

• RackLog - logs the request along with the duration of the processing
• RackDebug - Print's any exceptions along with a stack trace
• RackContentType - Supplies a default content type based on path extensions using MimeTypes
• RackRouter - a simple router that matches the path agains regular expressions to determine which app to call

But with those examples it should be easy for people to see how to compose their web server from middleware.

Suneido Functional Compose

I started on one project, found it required another, which then led to code like:

a(b(c(d(...)))

which seemed ugly. I knew that functional programming has a higher order "compose" to handle this more cleanly. (Higher order because it takes functions as arguments and returns another function.)

Suneido isn't specifically a functional language, but it has a bunch of functional methods and functions. I hadn't written Compose yet but it turned out to be quite easy.

function (@fns)
    {
    return {|@args|
        result = (fns[0])(@args)
        for fn in fns[1..]
            result = fn(result)
        result
        }
    }

If you're not familiar with Suneido code, the '@' is used to accept multiple arguments and to expand them again. The {|...| ... } notation is Suneido's way of writing a "block", i.e. a lambda or closure. (Suneido's use of the term "block" and the syntax come from Smalltalk, one of its roots way back when, pre 2000.)  Because, in Suneido, blocks are used for control flow, "return" returns from the containing function, so to return a result from a block we take advantage of how Suneido by default returns the value of the last statement.

Although in some languages the arguments to compose are listed "outside-in", I chose to have them "inside-out", so Compose(a,b,c) returns a function that does c(b(a(...))) in the same way you would write a *nix pipeline a | b | c i.e. in the order they will be applied.

I documented it, like a good programmer, except when I went to add the "see also" section I found a lot of the other functional stuff had never been documented. So I spent a bunch more time documenting some of the that. In the process I almost got sucked into improving the code, but I resisted pushing yet another task onto the stack!

Taking Notes

When I'm working on something complex, either a bug or new code, I keep notes. Nowadays I could blame that on an aging memory, but I've been doing it too long for that to be the reason. Originally I used paper notebooks, writing in pencil so I could erase and make corrections.

I take notes in a special style that has stayed quite consistent over the years. To make it easier to scan the notes and follow the "logical" structure I prefix sentences with capitalized "keywords". Some of these are familiar ones like "BUG" and "TODO" that are commonly used in source code comments. I also use "Q" to start a question (since the trailing question mark is harder to spot) and "A" for answers. Some less standard prefixes I use are "COULD" and "MAYBE" which I use to prefix proposals or hypothesis. ("MAYBE" being stronger than "COULD") Here's a fabricated example to give you a better idea:

BUG: calculating a checksum occasionally gives an incorrect result

Q why does it work some of the time ?

A probably a concurrency issue

MAYBE add locking

BUT that could lead to deadlocks

SO make sure no other locks are held or taken at the same time

TODO: review code to look for similar problems

These notes are generally just for my own use, but I've still tried to keep the notation fairly self explanatory so if someone else did need to read them it wouldn't be too hard.

You might imagine that the main benefit of keeping notes would be that you can refer back to them. But I find I rarely do that. Occasionally if I run into problems afterwards I'll go back to my notes to refresh my memory, especially if I didn't completely resolve the issue. But that's the exception.

I find the biggest benefit is that it helps me think through a problem, to be a little more logical, to expose and examine my thinking, to see what I've considered and why I think it is or isn't the right approach. It helps prevent me from going in circles and not making any progress. Often it's just as helpful to rule out certain alternatives as it is to find successful ones.

I'm a believer in clear code over a lot of comments. But what clear code doesn't tell you is why you chose that particular approach. It's good to add comments for that type of thing, but comments will seldom tell you the logical path you took to reach that point, the dead ends you explored, the approaches that turned out too complex or had performance issues. That's where the notes can come in.

Unless I know something is going to be complex I don't start out making notes. Instead, I'll work on it for a little bit and if it turns out to be tricky, then I'll make a conscious decision to start taking notes. I might make a few retroactive notes of my thinking so far but usually I'll just start from that point.

For a brief time I used Google Docs for these notes. That worked quite well for bigger projects, but it was too heavyweight for jotting down ideas or brief notes. And in the past they didn't have a very good mobile story.

For the last few years I've been using Evernote. I like how it syncs between all my devices and can be used off-line. The Mac and Windows versions are different enough to be annoying, but I don't imagine there are a lot of people that use both. From a software development perspective it seems crazy that they develop and maintain what appears to be two completely separate programs, but cross platform apps are a struggle, even today.

I briefly tried using Penultimate and Note Taker (from Dan Bricklin of VisiCalc fame) for taking handwritten notes in meetings. Scribbling something was easier and less distracting from the meeting than typing. But then I ended up more or less banning mobile devices from meetings because people find it impossible not to be distracted by all the notifications they get from texts, emails, FaceBook etc. (That's not as much of a problem for me because I have all notifications turned off. And being an antisocial programmer I'm not as addicted to that steady flow of chatter.)

Although I'm not big on diagrams, I did take advantage of pencil and paper to occasionally sketch out things like data data structures. This is a little harder to do when typing into some kind of text editor. I use Google Drawings if I want a more polished result. To quickly sketch diagrams on the iPad one app I like is Jot! Unlike a plain drawing app it lets you move things around and add editable text. However, it doesn't fix up your rectangles the way Paper does. But Paper doesn't let you add typed text to your drawings.

A few days ago someone asked me if I had used any apps that turned handwriting into editable text. I hadn't, unless you count way-back-when with a Palm and its quirky alphabet designed for easy recognition. (Which I quite liked and used a fair bit.)

Coincidentally, I recently got an iPad Pro and its companion Pencil. (Sadly, I'm not immune to the lure of the new new thing, despite feelings of guilt about it.) I hadn't really found a lot of use for the Pencil, although it did work well and had a much better feel than the previous fat soft iPad styluses that I'd tried.

So today I did some searching and found 7notes. Having not tried any handwriting recognition for years I was amazed at how well it worked. I started out printing but soon found it handled even my sloppy cursive. Of course, if I got too sloppy it had trouble, but even I have trouble reading my own handwriting sometimes. It has good support for auto correction and offering alternate possible words. Like the keyboard auto-correct, you do have to keep an eye on how it's interpreting what you write or you can get some funny results.

Although 7notes does have integration with Evernote it requires manually transferring notes back and forth. I started thinking it would be nice if I could use the handwriting recognition in Evernote itself (or other apps). I was happy to find that the same company has an iOS "keyboard" using the same technology - mazec. It doesn't have all the features of 7notes but seems to have the same basic recognition engine.

A Bug Hunter's Life

I recently spent two weeks tracking down a single bug. Needless to say, it was a frustrating process. Like most such things, what made it difficult was not a single problem, it was a whole cascade of errors, including many during the debugging process. I like to think I'm fairly organized and methodical, especially when working on a tough bug. I take notes on what I'm doing, what my assumptions are, what I've tried, what the results were. And yet I still managed to make multiple major mistakes and go a long way down the wrong paths based on false assumptions I should have questioned much sooner.

It started with a customer getting a corrupted database. Most of the time this is from hardware issues or crashing. But this one appeared to be caused by a bug in the code.

It wasn't easy to look into because all I had was the corrupt database, with no clues as to how it might have gotten that way. Thankfully, jSuneido's append-only database means that the database itself forms a kind of log. Based on that I could see that two transactions had committed conflicting updates - something that the whole transaction system is designed to prevent.

It seemed a pretty safe bet that this was a concurrency related issue - my least favorite kind of bug :-( I wasn't surprised that I couldn't make it happen. With roughly 1000 systems running 24x7 this was the only case I was aware of. So my sole sources of information were that database and the code itself.

In order to detect conflicts at commit time (jSuneido uses optimistic concurrency control) you need to know the outstanding overlapping update transactions (i.e. ones that had started after this one and had not completed yet). Perhaps the bug was in determining this set of transactions? I wrote a completely new implementation of this code and used it to cross-check.

We deployed this version to our customers and pretty quickly started to get lots of errors. But only a couple were from the new cross-check. The rest were from code I hadn't touched. And yet they quite clearly started when the new version was deployed. Strange.

The errors indicated that sometimes when it went to complete a transaction it would be missing from the list of outstanding ones. Again, this shouldn't be possible. That would explain why the overlapping set would be wrong, so it made a certain amount of sense. I spent days studying the code and trying (unsuccessfully) to figure out how it was happening.

This turned out to be completely off track - it was a just problem with my new cross checking code.

The stack traces showed a transaction commit that turned into an abort. That happens if an update transaction doesn't end up doing any updates. So I wrote a bunch of tests with those kinds of transactions in the mix, but they all worked fine. If nothing else, the code was getting reviewed and tested pretty heavily.

Finally, after wasting days on this, I realized there was another way for a commit to turn into an abort. The relevant part of the code looked like:

try {
     // commit stuff
} finally {
     abortIfNotComplete();
}

What was happening was that an exception during the commit stuff would get overridden (and lost) if there was an error in abortIfNotComplete(). So I wasn't seeing the original error at all. And the reason that the transaction was missing from the list was that it had got far enough into the commit to remove it.

I'm not sure why I wrote it using finally. In hindsight it was obviously a bad idea. I rewrote it like:

try {
     // commit stuff
} catch (Throwable e) {
     try {
          abortIfNotComplete();
     } catch (Throwable e2) {
          e.addSuppressed(e2);
          throw e;
     }

addSuppressed was added in Java 7. It is used, for example, in try-with-resources for exceptions when closing after another exception.

Part of the reason I didn't realize which path was being taken in the code was that I didn't have line numbers in the stack traces.  That was because in the Ant build task "debuglevel=vars" had been added. The intent had been to increase the amount of debugging information. But because of the way Ant and the Java -g option work, this had actually meant only variable information. I changed it to "debug=true" which becomes "-g" which means all debug information (file, line, and vars). Once this change percolated through and I started to get stack traces with line numbers, then I could see which path was being taken in the code.

One big step forward was when I was finally able to recreate the bug. Running random updates (of certain kinds) in a bunch of threads would encounter the bug within a few seconds. It wasn't the ideal recreation because the bug would occur buried within a lot of other activity, so it still wasn't possible to determine exactly what was happening.

I couldn't be 100% sure that what I was able to recreate was the same bug, but the end result was more or less identical - two transactions deleting the same record. But confusingly, the errors that I saw were about duplicate records being output, leading me to spend a bunch of time looking at the output code.

A huge mistake that I made repeatedly, without realizing it, was that I was building one variation of the jSuneido jar, and then testing with a different out of date one. So I wasn't actually testing with the changes I was making. This was because, for faster turnaround, I wasn't running a full build, I was just building the main jar that we deploy. But I was testing using the jar with Win32 support so I could use the IDE. This explained the totally baffling behavior where the debugging output that I was 100% certain should be triggered didn't appear at all. Which confused me and made me think I was totally wrong in my ideas about what was happening.

This mistake cost me about 4 days. I actually had the bug fixed, but it kept happening because I was testing with the old jar. Of course, since the bug still happened I assumed that I had not located it yet and I went looking for it in other places where, of course, I didn't find it.

Even worse, I thought my method of recreating the bug was quite "fragile". For instance, it would not happen running in the Eclipse debugger. Coming from C and C++, I didn't find that too surprising. And although Java should be more predicable, there would still be timing and threading differences which could affect concurrency. But I was totally wrong - the reason I couldn't recreate it within Eclipse was that I was running the latest code, which had the fix. Arghhh!

The actual bug and fix are obvious in hindsight. Throughout the long process I suspected that would be the likely result. But that didn't make it any easier, in fact it made it even more frustrating.

jSuneido's database uses optimistic concurrency. That means concurrent transactions proceed independently without any locking. Then when it comes time to commit, they verify that nothing has happened that would conflict. For example, they might find that another transaction has output a duplicate key. In which case the transaction will abort due to the conflict. Or it might find that another transaction has deleted a key that we also deleted. And that's where the bug was. I had made the assumption (back when I wrote this code) that it was ok if two concurrent transactions deleted the same key. After all, the end result is the same. And it's still serializable (i.e. the result is as if the transactions had happened one after another instead of concurrently).

The problem is not the deletes by themselves. If that was all the transactions were doing, my logic was correct. But transactions aren't that simple. They perform other actions based on whether the delete succeeded. For example, deleting an old version of a record and outputting a new version.  If you allow two transactions to think they deleted the old version, then they will, incorrectly, both output a new version. (leading to the duplicate outputs I was seeing)

All I had to do to fix the bug was to make duplicate deletes a conflict, like I already did with duplicate outputs. Two weeks for one line of code. How's that for productivity.

One interesting part of this whole adventure was that although it was a concurrency bug, it wasn't the kind of subtle interaction problem that I really fear. It was, in hindsight, a fairly clear, obvious issue.

Problem with Eclipse Mars on Windows after Java Update

I installed the new Java update on my Windows work machine, and removed the old version.

When I tried to run Eclipse (Mars) it said it couldn't find a JRE at the path of the old version of Java.

That made sense because I'd removed it. But why was it specifically looking for that path?

When I'd updated Java in the past Eclipse had found the new version fine.

I found that in the eclipse.ini file it had a -vm option set to the old version of Java. I removed this and now Eclipse starts fine.

I wonder if this was a result of installing the recently released Eclipse Mars with the Oomph installer?

Note: The Oomph installer puts Eclipse in your user folder (e.g. c:/Users/andrew in my case) rather than in Program Files. This may be a workaround to not require admin privileges to install.

Visual Studio Tip

I've been trying out Visual Studio 2015 Community RC and every time I did a build I'd get:

All packages are already installed and there is nothing to restore.
NuGet package restore finished.

It didn't seem to hurt anything, but I prefer to keep the build output clean so eventually I got tired of it and figured out you can get rid of it using Tools > Options

Of course, that's assuming you don't need to check for missing packages during builds.

Transpiling Suneido to JavaScript

Transpiling is source to source compiling, translating from one language to another, where both languages are at a similar level of abstraction.

For the background on why we'd want to do this, see my previous blog post on suneido.js

Transpiling is still compiling, and unless the translation is trivial, you need to parse the source language. I didn't want to write another Suneido parser, especially since I already had one in C++ in cSuneido and another in Java in jSuneido. cSuneido emits byte code as it parses, making it fast, but not very reusable. But jSuneido's parser builds an abstract syntax tree (AST) which is exactly what I needed.

One option would have been to write the transpiler in Java, as part of jSuneido. But I decided to write it in Suneido, for easier development and so it would be more accessible to Suneido programmers.

I added an AstParse function to jSuneido which takes a source string and returns an AST. Instead of converting the entire AST to Suneido objects I wrapped the internal AST and converted it lazily. [aside - I'm also hoping that we can use this in the IDE, e.g. for refactoring tools]

The big issue is deciding how to map from the Suneido language to JavaScript. Some things are easy, for example Suneido strings could be JavaScript strings. But other Suneido data types don't map directly. Even numbers are different since Suneido uses decimal floating point (for accurate business math) whereas JavaScript has binary floating point. Operations also differ so they have to be implemented as calls to a runtime support library. Suneido is also both more strict and more flexible with function arguments so that also requires runtime support.

So far I have statement and expression translation almost complete, and minimal versions of the runtime support routines (see: https://github.com/apmckinlay/suneido.js)

Here's a screenshot from a basic web page (running from a jSuneido server) that demonstrates the translation:

Simplest CodeMirror

Maybe it's just my relative inexperience with JavaScript but I struggled a bit to get a simple example of CodeMirror to work. The examples in the documentation and the download are all partial snippets and it wasn't obvious how to use them. In hopes of saving someone else some time, here's what I came up with. (Of course, it is trivial in retrospect!)

I was originally going to share a JSFiddle, but it does so much of the boilerplate for you, that it defeats the purpose! But you can run it there.

suneido.js

suneido.js = Suneido + JavaScript

After getting back from my last traveling I mentioned to a friend that I was looking forward to getting back to looking into a Go version of Suneido. Not that I’d committed to it, but it seemed like a reasonable path to take since it had the potential to replace both the C++ cSuneido client and the Java jSuneido server.

But my friend pointed out that what we really needed was a web front end. Which he’s told me before, but for some reason this time I paid attention. It’s easy to get into a rut, to make decisions unconsciously based more on what is familiar than on what’s really best. Not that you ever really know what the best path is, but a Go implementation of Suneido wouldn’t really change the game. It’d just be an incremental improvement.

Currently our front end GUI is Windows specific. So to access our application software over the internet you have to use RDP (Remote Desktop Protocol), which requires a Windows server. So even though jSuneido will happily run on Linux, we can’t take advantage of that.

And although RDP works fairly well, our Windows GUI is not the look and feel of the web that people expect these days. We’re starting to get comments that our program looks dated. (Reminds me of when we moved from terminal mode MS-DOS to Windows. And yes, I have been in this business that long!)

If all we wanted was to write web software, there are plenty of languages and tools and frameworks to do that. But we have a million lines of complex Suneido code. Rewriting that in another language for another database is not something I even want to think about!

And therefore the idea of somehow running our existing code, with the least amount of changes, on a web front end.

One way to do that would be to rewrite just the front end in HTML + CSS + JavaScript. But that would mean a certain amount of duplication since some code has to run on the front end and the server. And it would mean getting all our programmers up to speed in HTML, CSS, and JavaScript as well as Suneido.

Suneido’s philosophy (right or wrong) has always been to provide an integrated “everything included” platform that shields programmers from the hassles and complexities of dealing with multiple languages and databases and frameworks. That shielding has also made Suneido a ery stable platform. I can’t think of too many languages or databases or frameworks where code you wrote 15 years ago would work unchanged today. (Java itself has been around that long, but that’s just the language piece of the puzzle.)

So what I really wanted was an approach that would encapsulate and hide the HTML, CSS, and JavaScript from application developers, like Google GWT does. (Of course, you’d need to know that stuff to work on the lower level implementation.) In my mind, that breaks down into two pieces.

The easy part is to be able to compile/translate Suneido code to JavaScript. I’ve been looking into that, and have a lot of it implemented. More on that in another blog post.

The harder part is to figure out how to map our Windows oriented GUI to the browser. At a general level that’s not that hard. Our user interface is “component” based with screens composed from a small number of basic building blocks. It shouldn’t be too hard to obtain or write equivalent “widgets” for the web. On the other hand I’m certain that some of the details will be painful.

One of the issues is that our current user interface is very chatty. It was developed on and for local area networks with low latency. So it uses a lot of lower level requests to the server. If we stayed on local area networks we could stick with the same model, but really we want to be able to run across the internet, where you have much higher latency. Bandwidth is also a factor, but even if you have really high bandwidth the latency will kill you if you’re doing too many little requests.

Talking about this with some of my programmers, one of the questions was whether we would use node.js for the server. I think they were thinking that the whole of Suneido would be re-written in JavaScript, similar to how cSuneido is written in C++ and jSuneido is written in Java. But it wouldn’t make sense to write Suneido’s database in JavaScript. The server can still run the existing jSuneido. It’s perfectly capable of acting as a web server and at the same time providing the database back end. The server back end would still be running Suneido code (compiled to JVM byte code). Only the front end user interface code would be translated to JavaScript to run on the browser.

I wouldn't say I've committed to this direction, but it seems worth looking into it. The big question is how much of our existing code we'd be able to use, and how much revising / rewriting we'd have to do.

Stonebraker on Databases

I recently listened to a podcast on Software Engineering Radio with database expert Michael Stonebraker. (at the recommendation of a friend - thanks Larry) It's a few years old, but still quite relevant.

As usual, I relate much of what I read and hear about to Suneido. In this case it was interesting to see how Suneido's database held up to Stonebraker's criticism of current conventional relational databases.

He talks about profiling a database and finding that 90% of the time was spent on "overhead". The time consisted of four main areas:

buffer pool management
Keeping a cache of database pages, page replacement tracking, and converting from external (disk) record format to internal (in-memory) format. Since most transactional (OLTP) databases now fit in main memory, this work is even more wasteful.

record locking
Most conventional relational databases use pessimistic row level read and write locks to ensure that transactions are atomic, consistent, and isolated (i.e. ACID). Managing locks and waiting to acquire locks can be slow.

thread locking
Most databases are multi-threaded which, in most cases, means they use locking to protect shared data structures. Locking limits parallelism.

crash recovery write-ahead logs
For durability (the last part of ACID) databases commonly use a write-ahead log where changes are written (and in theory flushed to disk) prior to updating the actual database. In case of a crash, the log can be used for recovery. But writing the log is slow.

So how does Suneido do in these areas?

Instead of a buffer pool, the database is memory mapped. This handles databases that fit into memory as well as ones that don't. When they don't the page replacement is handled by the operating system which is in a good position to do this efficiently. Modern operating systems and hardware already have so many layers of caching and buffering that it seems crazy to add yet another layer of your own!

Suneido also does as much work as possible using the external format of records, only converting to internal format when necessary. For example, most database "wheres" are done in external format. The encoding of values into external format maintains ordering, so sorting can also be done while still in external format.

Rather than pessimistic record locking, Suneido uses optimistic multi-version concurrency control in conjunction with an append-only "immutable" database. This means that read transactions do not require any locking and do not interact with update transactions. Write transactions only require locking for the actual commit.

Suneido's database server  is multi-threaded with requests handled by a pool of threads. But most of the internal data is in immutable persistent data structures, which require minimal locking. And the database itself is an immutable persistent data structure requiring minimal locking. (I minimized the locking as much to avoid bugs as for performance.)

Finally, Suneido doesn't use a write-ahead log. Instead, its immutable append-only design makes it a type of log structured database, where the database itself can act as the log.

NOTE: This is based on the newer Java implementation of Suneido which has a different database engine than the older C++ version.

I haven't benchmarked Suneido's database against other systems so I can't make any claims about speed. But in terms of avoiding most of the overheads that Stonebraker identifies, Suneido seems to hold up pretty well.

Beautiful Code for Parsing

I've recently been reading up on JavaScript. One of the books I was re-reading was JavaScript: The Good Parts by Douglas Crockford. In there he mentioned the chapter he wrote in Beautiful Code. (Of which, there are mixed reviews.) It's been a while since I read that book and I didn't remember his chapter. So I pulled out my copy (old enough that it's an actual paper copy). I didn't want to carry the (large) book around so I took advantage of how O'Reilly lets you register your paper books and then buy the ebook for $5. Of course, after I did that I found Crockford's chapter is available on his web site. That's ok, I wouldn't mind re-reading the whole book.

The chapter was on parsing based on Top Down Operator Precedence (TDOP) by Vaughn Pratt. I thought I'd start by reading Pratt's original paper. Crockford's link takes you to the ACM citation which wants to charge you to read the paper (even though it's from 1973), but a quick web search found a public version. I found the paper a little hard to follow.

Crockford's article is an example of a parser for a subset of JavaScript written in that subset. It still took some effort to understand but I found it easier to follow than the original paper.

Suneido's hand written top down recursive descent parser is ok, but it has a lot of methods to parse expressions. It always seemed like there should be a better way. At some point I looked at the Go parser and it manages with a lot less methods by using precedence. I'd be quite interested to see what a Suneido parser would look like using Pratt's approach. Although, because Suneido's syntax grew ad hoc, it has some parts that are a little ugly to parse.

I love coming across elegant new algorithms and ideas. You can tell how much of a geek I am by the fact that I get as much pleasure out of a new algorithm as most people would from a bowl of ice cream :-) Which might help explain why I'm so skinny - not many calories in a delicious algorithm.

When is a PriorityQueue Not Ordered?

aka When does a final field change?

Background

jSuneido limits how long a database update transaction can be active. This is because active update transactions consume resources.

Read-only transactions consume little resources and are not limited. (This is a benefit of the append-only database structure.)

There are two parts to the limiting. If an update transaction commits and it was active more than a certain amount of time (currently 5 seconds) then a warning is logged but otherwise the transaction completes normally.

A separate process runs periodically (currently once per second) and any update transactions that have been active for too long (default 10 seconds) are aborted. A PriorityQueue is used for this so that only the longest running update transaction needs to be looked at.

Problem

Recently I noticed that some of our customers were getting the warning, but the duration given was well over the abort limit. That should have been impossible - the transactions should have been aborted before they reached that point. My first thought was that I had "broken" it with some recent change. But going back through the logs it looked like this had been happening for quite a while.

Searching the logs, I found that there were some transactions getting aborted due to duration, so that part of the code was working at least some of the time.

I added some debugging and found that the problem was that PriorityQueue peek was not returning the oldest transaction. That seemed impossible since by definition PriorityQueue peek returns the smallest element.

I checked my comparator but it was simple and seemed correct. I wrote some equivalent test code and it worked fine.

I started searching on the internet to see if anyone else had run into similar problems. Sure enough they had, but the reason was that they had been modifying the elements after inserting them. (Similar to problems if you modify elements after inserting into hash tables.)

But the field I was ordering by was final so it couldn't be modified.

Or could it? A final field still has to get set at some point. Sure enough, I was inserting the transactions into the queue in the super constructor, which ran before the field was initialized. So the queue insertion was always seeing a value of zero and the order was undefined. Argh!

(Java normally prevents this kind of problem, but I was casting to a derived class in code called by a base class constructor. Moral of the story - avoid tricky code!)

In case you're wondering, I had tested. But it worked when testing since I would only have a single active transaction, and the order was irrelevant.

It was an easy fix to reorganize the code slightly to ensure the queue insertion was done after the field was initialized. (Although that splits what used to be a single synchronized method into two, which makes me nervous about concurrency problems. I think it's ok, hopefully it won't be the subject of a future blog post!)

Gimme Structure

"I believe that it may happen that one will succeed, and one must not begin to despair, even though defeated here and there; and even though one sometimes feels a kind of decay, though things go differently from the expected, it is necessary to take heart again and new courage. For the great things are not done by impulse, but by a series of small things brought together. And great things are not something accidental, but must certainly be willed. What is drawing? How does one learn it? It is working through an invisible iron wall that seems to stand between what one feels and what one can do.”

-- Vincent Van Gogh

I used to think what I was looking for was good design. On more cynical days I'd settle for any design, or not even design, just some kind of structure.

I guess that's a bit like saying you want "quality". Would that be good quality or bad quality? Obviously, good structure is better than bad structure. But even bad structure is better than no structure.

I see, and work with, a lot of bad code, some of it written by my programmers, some of it (sadly) written by myself. The code seems to be split up into methods and classes more or less randomly. Names of variables and methods make no sense or are even outright misleading. It might work (most of the time) but it is difficult to understand, usually has duplication, commonly has logic errors, often old dead code, incorrect comments, etc. It will come as no surprise that it is hard to modify.

Part of the problem is incremental development. Even if there was some structure at some point, unless everyone modifying the code pays attention to maintaining that structure, it will degrade. And if it didn't have much structure to begin with it's even worse.

I don't think you can blame this on "evolution". Bad code is not very "fit". Natural selection would soon kill it off. Evolution is not intelligent design, but it comes up with lean, efficient solutions. It's not sloppy.

Much of the blame goes back to a common weakness in programmers - thinking that you are done when you have something that appears to work. Not going the extra distance to make sure it's readable, understandable, logically complete and correct. Often not even bothering to take care of the low hanging fruit like variable and method names.

And of course, once the code is a tangled mess no one wants to touch it to clean it up. Understandably, since it's a lot of work. And there's no doubt unobvious behavior in that code that you need to figure out and preserve. And there's a high risk of breaking things, and many programmers pay more attention to fear than to any desire for good code.

I have no silver bullets. Just a plea - please try to write code with some sort of comprehensible structure, for your own sake if nothing else.

Effective Modern C++

I just finished reading Effective Modern C++ by Scott Meyers. Like his More Effective C++ and the original Effective C++ it's well written with good explanations and examples. This third book covers the latest C++ features in C++11 and 14.

It's been a long time since I read the first two books. Effective C++ was published in 1991! Back then I was writing fair amounts of C++ code. Nowadays the only C++ programming I do is maintaining the C++ implementation of Suneido.

I expected the new book to be similar to the previous ones - practical advice on how to effectively use modern C++. And there is lots of that. But it was also full of "gotchas" - things that won't compile (and give horrendous error messages), or compile but won't run, or compile and run but do the wrong thing.

C++ has always been a complex language and the new versions have only pushed that even further. If makes me appreciate the simplicity of the Go language which in some ways is a reaction to the complexity of C++.

Don't get me wrong, the new features of C++ are great, they improve the language in many ways. But my head is spinning with things like when perfect forwarding isn't perfect, when universal references aren't, and when uniform initialization isn't uniform.

Safety First

I recently fixed a long standing (many years) bug in the C++ implementation of Suneido. A friend remarked how you'd wish that after this long all the bugs would have been found. Of course, it doesn't take much code to provide room for bugs to lurk.

The problem that was reported was that if you created one thread inside another that cSuneido would crash. It seemed to happen quite consistently and predictably. That was from the IDE. If you ran the same code without the IDE it worked fine. Or if you played around a bit in the IDE first, it would also work fine.

cSuneido "threads" aren't real threads. They are Windows "fibers" - more like coroutines. They don't actually run concurrently, but they allow cooperative multi-tasking. The big advantage is that since you control when the task switching happens and can do it at "safe" points in the code, you don't have to worry about low level concurrency issues. The downside is that you can't take advantage of multiple cpu's. But this was implemented at a time when no one had multiple cpu's and Moore's Law was still happily improving single cpu performance.

Suneido's C++ fiber code had a std::vector of fibers. It also had a main fiber, separate from the vector. The current fiber was a reference (pointer) to either the main fiber or an element of the vector.

Even from that minimal description you could probably guess the problem. Vector implementations normally grow by allocating a new larger array, copying over the data, and throwing out the smaller old array. So adding an element to a vector invalidates any references to its content. So the current fiber reference would be pointing to stale data. (It wouldn't actually be a dangling pointer because cSuneido uses garbage collection.) The reference to stale data could cause an "impossible" situation that would lead to a fatal error. (So the problem was nothing to do with creating one fiber inside another, it was simply that creating two fibers in that sequence happened to be one way to expose the bug.)

The problem was rare because it required a specific sequence of events. First, the vector had to grow. Which is why if you played around first (and expanded the vector) it wouldn't happen. Second, the stale reference had to be used in such a way that it caused a problem. Since the data would normally be identical the stale reference wouldn't matter. And the next fiber switch would update it to a valid value so the stale reference wouldn't hang around.

Actually, I think there was at least one more potential problem scenario. When fibers ended they were removed from the vector. This probably wouldn't cause a reallocation (many implementations never shrink the array) but it would invalidate any references after that item. You'd either end up with a reference to the wrong item or past the end of the array.

I'm a little embarrassed to discover such a long standing blatant mistake, and a newbie mistake at that. All the times I've looked at that code and I never picked up on it. Ouch.

But to me the real moral of the story is "don't use unsafe languages". Interestingly, this bug was not a memory management issue since cSuneido (unlike almost all C++ programs) uses garbage collection. It's just a result of C++ allowing unsafe raw pointers/references.

C++ fans would tell you that modern C++ has plenty of high level features that are "safe". But the point is that it still has lots of unsafe features. (And AFAIK there is no way to enforce use of a "safe" subset. And C++ continues to resist "real" garbage collection.) I would much rather work in a language like Java or Go (or others) that just don't allow unsafe code of this nature, and eliminate a whole class of problems. Figuring out my high level issues is challenging enough without worrying about unsafe low level issues.

Go Editors

Up till recently I've been using Sublime Text with GoSublime to write Go code. It works pretty well. Sublime is a good editor and GoSublime integrates with the Go tools fairly well. But coming back to it after being away I found it quite annoying that compile errors are only shown in the output pane, not marked on the source code. And you can't even click on the error to go to that line. I'm not a big fan of using line numbers but with Sublime I was pretty much forced to display line numbers and use them manually. (There's probably some way to get clicking on errors to go to the line but nothing obvious.)

I'm not sure where Sublime is at. Sublime 3 has been in beta for a long time. GoSublime has some activity but doesn't seem to be doing too much either.

So I've been on the lookout for alternatives. And I needed something that was available on both Mac and Windows.

I came across something about Github's Atom editor and the go-plus extension. I had some difficulties getting it working on Windows, easier on Mac. It has better integration between Go and the editor, showing lines with errors and letting you click on the errors. But it doesn't seem to have much support for things like running tests. I realize that's outside the scope of just an editor, and I can always run the tests outside the editor. But I'd still prefer to have it. (Again, there may be some way to do it, but if so it wasn't obvious.)

Both Eclipse and IntelliJ have facilities for Go but they seem like very heavy weight tools for a "lightweight" language like Go.

The other recommendation I'd seen was LiteIDE. It's somewhere in between a full IDE like Eclipse, and an editor like Atom. It was easier to install than either Sublime or Atom since it's a single package, no add ons to worry about. I haven't used it a lot yet but it seems like it might be a good option. The editor is decent and it doesn't force me to use line numbers. I can run tests. The only weakness I've found so far is that it doesn't support column select or multiple select. I can probably live without that, if need be I can always use another editor for the odd time I need it. And it looks like the Kate editor that LiteIDE uses does support this so I'd guess it might be added at some point.

The project seems quite active. I found a bug where some keyboard shortcuts didn't work when you had multiple windows open. I couldn't find any mention of this problem so I entered a bug for it. Within hours I got a notification of a fix committed. It looked like an easy fix, and I haven't tried to build from source to test it, but it's still impressive that the issue was addressed so quickly.

Just a Minute

I bought a new iMac Retina 5K. (amazing display!) So Shelley gets my previous four year old iMac. (Replacing her even more ancient iMac.) Personally I prefer to set up new machines from scratch rather than migrate potential junk and problems from the old machine. But for Shelley I knew that would be a big hassle so I used Apple's Migration Assistant.

Shelley was out for the afternoon so I started the migration. It's simple to use, you start it up on both machines and indicate which is the source and which is the destination.

The estimated time remaining went up and down, but was around 4 hours. That seemed relatively accurate since after about 4 hours it said it had a minute left. That was good timing since Shelley had just arrived home.

But an hour later it still said it had a minute left. Crap! I started searching on the web and found lots of other people with the same problem. It's been an issue for years, but I didn't find any official response from Apple. For some people it seemed if they left it long enough it would eventually finish. But other people waited e.g. 24 hours and it still didn't finish. I could abort it at any time and leave Shelley with the old computer, but she was ok with waiting overnight.

I could tell it was still doing something because our internal network was slow. In fact, the first clue that it might have finished was that the network suddenly got a lot faster. It ended up taking about another 4 hours for that "last minute". It reminded me of the 90-90 rule in software development that "The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time."

I understand that estimating completion times is difficult, and progress indicators are infamous for stalling at the end. But "a minute" is several orders of magnitude different from 4 hours. Surely Apple could do better, maybe obsess over the migration experience as well as the un-boxing experience.

If they really can't improve the time estimation, then give some visibility to the process. For example, show a list of "things" to be copied and check them off. Sticking at one minute remaining looks like it's hung up and I suspect a lot of people cause additional problems because they kill the process and then tried to recover from a half copied machine.

Other than this hiccup the migration seems to have been successful. But instead of being the hero for giving Shelley a newer, bigger, faster computer, I ended being the indirect cause of "breaking" her Microsoft Office. It needed the product key to reactivate it on the new computer and that seems to be long gone. The key would have been on the physical package which probably got thrown out sometime over the years. And worse, Microsoft now wants you to pay a monthly fee to use Office, rather than just a one time purchase. On top of which, they haven't updated Office for Mac since 2011. Sigh. Home tech support can be a thankless job!

PS. With Migration Assistant you have a choice of copying from the old machine, or copying from a Time Machine backup. I chose to copy from the old machine just in case the backup didn't include everything. Some of what I found on the web seems to indicate that copying from a Time Machine backup doesn't have the same problem.

Java 8 Performance

I was just looking at some stats on the average time our customers' servers take to run our application test suite.

I noticed on a particular day the times dropped from an average of 240 seconds to an average of 200 seconds. (These are averages from about 240 customer sites.) The numbers are generally quite stable so I was curious what changed.

I discovered that was the day we updated everyone to the Java 8 JRE so it looks like that's the reason for the improvement. Assuming that's the correct explanation that's a pretty nice upgrade!

It made sense that it was Java related since customers running the older cSuneido did not show any improvement that day.

Note: jSuneido is still compiled for Java 7, this improvement would just be from the runtime.

Portable Tests

With two implementations of Suneido (the original C++ cSuneido and the newer Java jSuneido) I've ended up with three sets of overlapping tests - in each of the implementations plus in the Suneido standard library. And as I play with implementing Suneido in Go I find myself creating yet another set of tests.

Obviously this is not ideal. Apart from the duplication, each version of the tests has better or worse coverage of different areas depending on where I had issues with the implementation. Ideally, I'd like to run the same complete set of tests everywhere, and if I added a test case it would be included everywhere, not just in one of the versions.

One option would be to use something like Fit or Fitnesse. But that would still require writing code (for fixtures and "slim" interfaces) and it would mean accepting a third party dependency which in turn depends on Java.

I figured the simplest thing would be to have the test cases in text files and to write a test runner for each of the versions.

But what format should I use for the test cases? I realized that I could use a format that was easy to read with the Suneido lexical scanner. Any implementation of Suneido has to have this, and it's generally one of the first things I implement. Using the scanner made it easy to handle quoted strings and to ignore comments and whitespace.

I implemented a test runner in Suneido code first, and designed the format to keep the parsing simple. Here is an example:

@add

1, 1, 2 // i.e. assert 1 + 1 == 2

@regex_match

"abc" "b"
"abc", "x", false // this should not match

"foo
bar", 
"^bar" // ^ should match after a newline

An '@' followed by a name precedes a list of test cases for the named test "fixture". Each version has to implement each of the fixtures, but these are simple and I already have equivalent code in the existing tests.

Normally each line of values is a test case. Commas between values are optional, but newlines are ignored after a comma to allow splitting a test case over several lines.

After the Suneido code version it was straightforward to implement a version in Go. Java and C++ should also be simple.

I still want to run these tests as part of the existing automated testing, but that's easy to do by writing a test (e.g. in JUnit) that calls the test runner for the portable tests.

A remaining question is where to put the test files. Expecting them to be in the current directory or a subdirectory is easiest, but then each version will have its own copy and I'd have to keep them in sync. It makes more sense to have a single copy somewhere, but then I need some way to point each version at that central location. One option would be an environment variable but that can be problematic. Instead I decided I'd put a text file at the root of each project that would contain the directory path to the tests. (And if that doesn't fit, each implementation is free to handle this differently.)

My main concern with this was tests with a lot of different cases, where you'd use data driven tests. (like regular expressions) In other areas what I probably should have is more of a BDD (Behavior-driven development) style of tests that would form a kind of specification for Suneido. To keep this portable it would make sense to use the JBehave style that separates the specification from the implementation.

Go: When nil isn't nil

Go is a fairly simple language. But it's still complex enough to have some oddities. I recently ran into one of them.

Lets say you have a nil pointer and you assign it to an interface:

var p *T = nil

intfc interface{} = p

I expected intfc to now be nil, but it's not.

The explanation (it's even in the FAQ) is in terms of the implementation. An interface contains a pointer and a type. When we assign a nil pointer the interface type is set. And an interface is only nil if both the pointer and the type are "empty".

I understand the explanation and it's not hard to work around once you're aware of it. If you want to return an interface that compares equal to nil you just have to make sure you don't assign a nil pointer to it.

But I don't find the explanation very satisfying. 

First, justifying some external behavior by explaining how you happened to implement it seems wrong. (Although to be fair, maybe the explanation is intended to be in terms of what an interface means, not really how it's implemented.)

Second, the explanation doesn't explain why they chose to make it work this way. Granted, it's simple because the nil check is just for a zeroed value. But it doesn't seem like it would be much harder for the compiler to just check for a zeroed pointer and ignore the type. It seems like this would avoid the unexpected behavior with no real loss of functionality.

I did some searching, but the only justification I could find is that a nil pointer in an interface can still satisfy the interface and you can still call methods on it. Which is fine, but what does that have to do with whether the interface value compares equal to nil? I guess it would introduce a new oddity in that you'd have two kinds of nil interfaces only one of which you could call methods on.

The other issue is that (afaik) you can't determine if an interface holds a nil pointer without using reflection.

It's not a big deal, and I'm not hung up on it, it just seems odd. As far as languages are concerned, I would say Go has relatively few sharp corners to get caught on.

Go Suneido Spike

In software, a spike is some programming to investigate options or answer questions or to test design ideas. The term came from XP (extreme programming) and is used in agile and Scrum.

As a way to learn Go I've been investigating what it would be like to use it to implement Suneido. I've implemented various bits and pieces like lexical scanning, memory mapped file access, decimal floating point numbers, string concatenation, and hash tables.

In the Java implementation of Suneido I was able to leverage the Java virtual machine. In Go (as in the original C++ version of Suneido) I would need to write my own bytecode interpreter. To investigate this, I did an end to end spike from lexing to parsing to code generation to byte code interpreter. I even implemented a basic REPL (Read, Eval, Print Loop). All it currently handles are simple expressions like 100 + 50 - 25 or "hello" $ "world". But it's implemented with the right structure to flesh out into the full language. (The code is on Github if you're interested.)

I've written about 4000 lines of Go code so far. Not enough to be an expert by any means, but enough that I don't feel like a newbie anymore. It's been mostly low level code, I haven't done anything with goroutines and channels yet.

It's been remarkably painless. I think it helps to have a C/C++ background. Of all the languages I've dabbled in in recent years, Go has been the easiest to pick up and be productive in. That's partly due to the small, simple language, and partly due to the good default tools. The Java language wasn't hard to pick up, but the libraries and tools are definitely more complex.

The fast compiles are a key feature. After working in Java I think it would be hard to give this up. Considering the classic compile and link approach, running Go code seems just as fast as running Java code.

I almost find Go's simplicity a little disappointing. I love reading about complex languages like C++ and Scala and all the wild and wonderful things you can do with them. What software geek wouldn't love turing complete templates and type systems! Go doesn't have those kinds of things, and therefore doesn't have intricate books about them.

But as far as something I actually want to use, and not just read about - in that respect Go is great.

A Go Hash Map

I was a little disappointed to discover that the built in Go map doesn't allow interfaces as keys. Considering that interfaces are the way to do anything dynamic in Go, and that dynamic stuff often uses maps, it seems a little odd.

To act as a hash table key, you need equals and hash code methods. But it's easy to define an interface for that and require that keys implement it, similar to how Sort requires a container to implement Len, Less, and Swap.

I looked around to see what's out there and found a few options, but none really excited me. And I'm still learning Go, so it made more sense to implement it myself.

My first thought was to port my hash map code from cSuneido. But I wondered what Go's map was like. The code is straightforward but it uses an interesting approach that I haven't encountered before. It's a variant of separate chaining with each slot in the hash table being a bucket that can hold a small number of entries (e.g. 8). Additional overflow buckets can be chained together. In many hash table designs, collisions are a nuisance to be tolerated, but this design almost embraces them, by making the table 8 times smaller you assume collisions.

Buckets holding a number of entries are also better for cache locality than a linked list.

Another interesting feature is that the buckets have an additional byte per entry that holds the high byte of the hash code of the key. This helps in searching because if this piece of the hash code doesn't match then you can avoid comparing keys (which is cache unfriendly and also slow if keys are large or complex).

This design also works well for small tables since you can use a single bucket, which basically reduces it to a small array with linear searching, which is what you want for a few entries.

So I implemented this design in Go, following the C code fairly closely, except that I didn't implement the incremental resizing. It might be worthwhile in some situations, but it makes the code more complex (especially iteration) and probably makes the resizing slightly slower in total, albeit amortized. The lack of incremental resizing hasn't been a noticeable issue in cSuneido.

Have a look, it's about 200 lines of Go.

The next issue was what to use for hashing strings. Go has standard packages for hashing but they require converting to a byte array which requires allocation and copying. (Go 1.3 has an optimization for this, but only for the built in map.) So again, I wrote my own version, following the approach in hash/fnv.

It seems reasonable. The main drawback comes from Go's lack of generics - it has to work in terms of interface{} (the equivalent of Java Object or C/C++ void*) so you have to cast everything that comes out of it, reminiscent of Java prior to generics. Another minor awkwardness is that you can't use tbl[key] syntax like built in maps.

Another hash table approach which would be a natural fit with Go would be to use a growable slice for each entry in the table (rather than a list of buckets). This would avoid the space overhead from chain links and from partially full buckets, at the cost of the slice itself (a pointer and two int's), plus more individual allocations.

Related interesting reading:

• Fast and Compact Hash Tables for Integer Keys 
• Redesigning the String Hash Table, Burst Trie, and BST to Exploit Cache