Beautiful Code for Parsing

I've recently been reading up on JavaScript. One of the books I was re-reading was JavaScript: The Good Parts by Douglas Crockford. In there he mentioned the chapter he wrote in Beautiful Code. (Of which, there are mixed reviews.) It's been a while since I read that book and I didn't remember his chapter. So I pulled out my copy (old enough that it's an actual paper copy). I didn't want to carry the (large) book around so I took advantage of how O'Reilly lets you register your paper books and then buy the ebook for $5. Of course, after I did that I found Crockford's chapter is available on his web site. That's ok, I wouldn't mind re-reading the whole book.

The chapter was on parsing based on Top Down Operator Precedence (TDOP) by Vaughn Pratt. I thought I'd start by reading Pratt's original paper. Crockford's link takes you to the ACM citation which wants to charge you to read the paper (even though it's from 1973), but a quick web search found a public version. I found the paper a little hard to follow.

Crockford's article is an example of a parser for a subset of JavaScript written in that subset. It still took some effort to understand but I found it easier to follow than the original paper.

Suneido's hand written top down recursive descent parser is ok, but it has a lot of methods to parse expressions. It always seemed like there should be a better way. At some point I looked at the Go parser and it manages with a lot less methods by using precedence. I'd be quite interested to see what a Suneido parser would look like using Pratt's approach. Although, because Suneido's syntax grew ad hoc, it has some parts that are a little ugly to parse.

I love coming across elegant new algorithms and ideas. You can tell how much of a geek I am by the fact that I get as much pleasure out of a new algorithm as most people would from a bowl of ice cream :-) Which might help explain why I'm so skinny - not many calories in a delicious algorithm.

When is a PriorityQueue Not Ordered?

aka When does a final field change?

Background

jSuneido limits how long a database update transaction can be active. This is because active update transactions consume resources.

Read-only transactions consume little resources and are not limited. (This is a benefit of the append-only database structure.)

There are two parts to the limiting. If an update transaction commits and it was active more than a certain amount of time (currently 5 seconds) then a warning is logged but otherwise the transaction completes normally.

A separate process runs periodically (currently once per second) and any update transactions that have been active for too long (default 10 seconds) are aborted. A PriorityQueue is used for this so that only the longest running update transaction needs to be looked at.

Problem

Recently I noticed that some of our customers were getting the warning, but the duration given was well over the abort limit. That should have been impossible - the transactions should have been aborted before they reached that point. My first thought was that I had "broken" it with some recent change. But going back through the logs it looked like this had been happening for quite a while.

Searching the logs, I found that there were some transactions getting aborted due to duration, so that part of the code was working at least some of the time.

I added some debugging and found that the problem was that PriorityQueue peek was not returning the oldest transaction. That seemed impossible since by definition PriorityQueue peek returns the smallest element.

I checked my comparator but it was simple and seemed correct. I wrote some equivalent test code and it worked fine.

I started searching on the internet to see if anyone else had run into similar problems. Sure enough they had, but the reason was that they had been modifying the elements after inserting them. (Similar to problems if you modify elements after inserting into hash tables.)

But the field I was ordering by was final so it couldn't be modified.

Or could it? A final field still has to get set at some point. Sure enough, I was inserting the transactions into the queue in the super constructor, which ran before the field was initialized. So the queue insertion was always seeing a value of zero and the order was undefined. Argh!

(Java normally prevents this kind of problem, but I was casting to a derived class in code called by a base class constructor. Moral of the story - avoid tricky code!)

In case you're wondering, I had tested. But it worked when testing since I would only have a single active transaction, and the order was irrelevant.

It was an easy fix to reorganize the code slightly to ensure the queue insertion was done after the field was initialized. (Although that splits what used to be a single synchronized method into two, which makes me nervous about concurrency problems. I think it's ok, hopefully it won't be the subject of a future blog post!)

Gimme Structure

"I believe that it may happen that one will succeed, and one must not begin to despair, even though defeated here and there; and even though one sometimes feels a kind of decay, though things go differently from the expected, it is necessary to take heart again and new courage. For the great things are not done by impulse, but by a series of small things brought together. And great things are not something accidental, but must certainly be willed. What is drawing? How does one learn it? It is working through an invisible iron wall that seems to stand between what one feels and what one can do.”

-- Vincent Van Gogh

I used to think what I was looking for was good design. On more cynical days I'd settle for any design, or not even design, just some kind of structure.

I guess that's a bit like saying you want "quality". Would that be good quality or bad quality? Obviously, good structure is better than bad structure. But even bad structure is better than no structure.

I see, and work with, a lot of bad code, some of it written by my programmers, some of it (sadly) written by myself. The code seems to be split up into methods and classes more or less randomly. Names of variables and methods make no sense or are even outright misleading. It might work (most of the time) but it is difficult to understand, usually has duplication, commonly has logic errors, often old dead code, incorrect comments, etc. It will come as no surprise that it is hard to modify.

Part of the problem is incremental development. Even if there was some structure at some point, unless everyone modifying the code pays attention to maintaining that structure, it will degrade. And if it didn't have much structure to begin with it's even worse.

I don't think you can blame this on "evolution". Bad code is not very "fit". Natural selection would soon kill it off. Evolution is not intelligent design, but it comes up with lean, efficient solutions. It's not sloppy.

Much of the blame goes back to a common weakness in programmers - thinking that you are done when you have something that appears to work. Not going the extra distance to make sure it's readable, understandable, logically complete and correct. Often not even bothering to take care of the low hanging fruit like variable and method names.

And of course, once the code is a tangled mess no one wants to touch it to clean it up. Understandably, since it's a lot of work. And there's no doubt unobvious behavior in that code that you need to figure out and preserve. And there's a high risk of breaking things, and many programmers pay more attention to fear than to any desire for good code.

I have no silver bullets. Just a plea - please try to write code with some sort of comprehensible structure, for your own sake if nothing else.

Effective Modern C++

I just finished reading Effective Modern C++ by Scott Meyers. Like his More Effective C++ and the original Effective C++ it's well written with good explanations and examples. This third book covers the latest C++ features in C++11 and 14.

It's been a long time since I read the first two books. Effective C++ was published in 1991! Back then I was writing fair amounts of C++ code. Nowadays the only C++ programming I do is maintaining the C++ implementation of Suneido.

I expected the new book to be similar to the previous ones - practical advice on how to effectively use modern C++. And there is lots of that. But it was also full of "gotchas" - things that won't compile (and give horrendous error messages), or compile but won't run, or compile and run but do the wrong thing.

C++ has always been a complex language and the new versions have only pushed that even further. If makes me appreciate the simplicity of the Go language which in some ways is a reaction to the complexity of C++.

Don't get me wrong, the new features of C++ are great, they improve the language in many ways. But my head is spinning with things like when perfect forwarding isn't perfect, when universal references aren't, and when uniform initialization isn't uniform.

Safety First

I recently fixed a long standing (many years) bug in the C++ implementation of Suneido. A friend remarked how you'd wish that after this long all the bugs would have been found. Of course, it doesn't take much code to provide room for bugs to lurk.

The problem that was reported was that if you created one thread inside another that cSuneido would crash. It seemed to happen quite consistently and predictably. That was from the IDE. If you ran the same code without the IDE it worked fine. Or if you played around a bit in the IDE first, it would also work fine.

cSuneido "threads" aren't real threads. They are Windows "fibers" - more like coroutines. They don't actually run concurrently, but they allow cooperative multi-tasking. The big advantage is that since you control when the task switching happens and can do it at "safe" points in the code, you don't have to worry about low level concurrency issues. The downside is that you can't take advantage of multiple cpu's. But this was implemented at a time when no one had multiple cpu's and Moore's Law was still happily improving single cpu performance.

Suneido's C++ fiber code had a std::vector of fibers. It also had a main fiber, separate from the vector. The current fiber was a reference (pointer) to either the main fiber or an element of the vector.

Even from that minimal description you could probably guess the problem. Vector implementations normally grow by allocating a new larger array, copying over the data, and throwing out the smaller old array. So adding an element to a vector invalidates any references to its content. So the current fiber reference would be pointing to stale data. (It wouldn't actually be a dangling pointer because cSuneido uses garbage collection.) The reference to stale data could cause an "impossible" situation that would lead to a fatal error. (So the problem was nothing to do with creating one fiber inside another, it was simply that creating two fibers in that sequence happened to be one way to expose the bug.)

The problem was rare because it required a specific sequence of events. First, the vector had to grow. Which is why if you played around first (and expanded the vector) it wouldn't happen. Second, the stale reference had to be used in such a way that it caused a problem. Since the data would normally be identical the stale reference wouldn't matter. And the next fiber switch would update it to a valid value so the stale reference wouldn't hang around.

Actually, I think there was at least one more potential problem scenario. When fibers ended they were removed from the vector. This probably wouldn't cause a reallocation (many implementations never shrink the array) but it would invalidate any references after that item. You'd either end up with a reference to the wrong item or past the end of the array.

I'm a little embarrassed to discover such a long standing blatant mistake, and a newbie mistake at that. All the times I've looked at that code and I never picked up on it. Ouch.

But to me the real moral of the story is "don't use unsafe languages". Interestingly, this bug was not a memory management issue since cSuneido (unlike almost all C++ programs) uses garbage collection. It's just a result of C++ allowing unsafe raw pointers/references.

C++ fans would tell you that modern C++ has plenty of high level features that are "safe". But the point is that it still has lots of unsafe features. (And AFAIK there is no way to enforce use of a "safe" subset. And C++ continues to resist "real" garbage collection.) I would much rather work in a language like Java or Go (or others) that just don't allow unsafe code of this nature, and eliminate a whole class of problems. Figuring out my high level issues is challenging enough without worrying about unsafe low level issues.

Go Editors

Up till recently I've been using Sublime Text with GoSublime to write Go code. It works pretty well. Sublime is a good editor and GoSublime integrates with the Go tools fairly well. But coming back to it after being away I found it quite annoying that compile errors are only shown in the output pane, not marked on the source code. And you can't even click on the error to go to that line. I'm not a big fan of using line numbers but with Sublime I was pretty much forced to display line numbers and use them manually. (There's probably some way to get clicking on errors to go to the line but nothing obvious.)

I'm not sure where Sublime is at. Sublime 3 has been in beta for a long time. GoSublime has some activity but doesn't seem to be doing too much either.

So I've been on the lookout for alternatives. And I needed something that was available on both Mac and Windows.

I came across something about Github's Atom editor and the go-plus extension. I had some difficulties getting it working on Windows, easier on Mac. It has better integration between Go and the editor, showing lines with errors and letting you click on the errors. But it doesn't seem to have much support for things like running tests. I realize that's outside the scope of just an editor, and I can always run the tests outside the editor. But I'd still prefer to have it. (Again, there may be some way to do it, but if so it wasn't obvious.)

Both Eclipse and IntelliJ have facilities for Go but they seem like very heavy weight tools for a "lightweight" language like Go.

The other recommendation I'd seen was LiteIDE. It's somewhere in between a full IDE like Eclipse, and an editor like Atom. It was easier to install than either Sublime or Atom since it's a single package, no add ons to worry about. I haven't used it a lot yet but it seems like it might be a good option. The editor is decent and it doesn't force me to use line numbers. I can run tests. The only weakness I've found so far is that it doesn't support column select or multiple select. I can probably live without that, if need be I can always use another editor for the odd time I need it. And it looks like the Kate editor that LiteIDE uses does support this so I'd guess it might be added at some point.

The project seems quite active. I found a bug where some keyboard shortcuts didn't work when you had multiple windows open. I couldn't find any mention of this problem so I entered a bug for it. Within hours I got a notification of a fix committed. It looked like an easy fix, and I haven't tried to build from source to test it, but it's still impressive that the issue was addressed so quickly.

Just a Minute

I bought a new iMac Retina 5K. (amazing display!) So Shelley gets my previous four year old iMac. (Replacing her even more ancient iMac.) Personally I prefer to set up new machines from scratch rather than migrate potential junk and problems from the old machine. But for Shelley I knew that would be a big hassle so I used Apple's Migration Assistant.

Shelley was out for the afternoon so I started the migration. It's simple to use, you start it up on both machines and indicate which is the source and which is the destination.

The estimated time remaining went up and down, but was around 4 hours. That seemed relatively accurate since after about 4 hours it said it had a minute left. That was good timing since Shelley had just arrived home.

But an hour later it still said it had a minute left. Crap! I started searching on the web and found lots of other people with the same problem. It's been an issue for years, but I didn't find any official response from Apple. For some people it seemed if they left it long enough it would eventually finish. But other people waited e.g. 24 hours and it still didn't finish. I could abort it at any time and leave Shelley with the old computer, but she was ok with waiting overnight.

I could tell it was still doing something because our internal network was slow. In fact, the first clue that it might have finished was that the network suddenly got a lot faster. It ended up taking about another 4 hours for that "last minute". It reminded me of the 90-90 rule in software development that "The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time."

I understand that estimating completion times is difficult, and progress indicators are infamous for stalling at the end. But "a minute" is several orders of magnitude different from 4 hours. Surely Apple could do better, maybe obsess over the migration experience as well as the un-boxing experience.

If they really can't improve the time estimation, then give some visibility to the process. For example, show a list of "things" to be copied and check them off. Sticking at one minute remaining looks like it's hung up and I suspect a lot of people cause additional problems because they kill the process and then tried to recover from a half copied machine.

Other than this hiccup the migration seems to have been successful. But instead of being the hero for giving Shelley a newer, bigger, faster computer, I ended being the indirect cause of "breaking" her Microsoft Office. It needed the product key to reactivate it on the new computer and that seems to be long gone. The key would have been on the physical package which probably got thrown out sometime over the years. And worse, Microsoft now wants you to pay a monthly fee to use Office, rather than just a one time purchase. On top of which, they haven't updated Office for Mac since 2011. Sigh. Home tech support can be a thankless job!

PS. With Migration Assistant you have a choice of copying from the old machine, or copying from a Time Machine backup. I chose to copy from the old machine just in case the backup didn't include everything. Some of what I found on the web seems to indicate that copying from a Time Machine backup doesn't have the same problem.

Java 8 Performance

I was just looking at some stats on the average time our customers' servers take to run our application test suite.

I noticed on a particular day the times dropped from an average of 240 seconds to an average of 200 seconds. (These are averages from about 240 customer sites.) The numbers are generally quite stable so I was curious what changed.

I discovered that was the day we updated everyone to the Java 8 JRE so it looks like that's the reason for the improvement. Assuming that's the correct explanation that's a pretty nice upgrade!

It made sense that it was Java related since customers running the older cSuneido did not show any improvement that day.

Note: jSuneido is still compiled for Java 7, this improvement would just be from the runtime.

Portable Tests

With two implementations of Suneido (the original C++ cSuneido and the newer Java jSuneido) I've ended up with three sets of overlapping tests - in each of the implementations plus in the Suneido standard library. And as I play with implementing Suneido in Go I find myself creating yet another set of tests.

Obviously this is not ideal. Apart from the duplication, each version of the tests has better or worse coverage of different areas depending on where I had issues with the implementation. Ideally, I'd like to run the same complete set of tests everywhere, and if I added a test case it would be included everywhere, not just in one of the versions.

One option would be to use something like Fit or Fitnesse. But that would still require writing code (for fixtures and "slim" interfaces) and it would mean accepting a third party dependency which in turn depends on Java.

I figured the simplest thing would be to have the test cases in text files and to write a test runner for each of the versions.

But what format should I use for the test cases? I realized that I could use a format that was easy to read with the Suneido lexical scanner. Any implementation of Suneido has to have this, and it's generally one of the first things I implement. Using the scanner made it easy to handle quoted strings and to ignore comments and whitespace.

I implemented a test runner in Suneido code first, and designed the format to keep the parsing simple. Here is an example:

@add

1, 1, 2 // i.e. assert 1 + 1 == 2

@regex_match

"abc" "b"
"abc", "x", false // this should not match

"foo
bar", 
"^bar" // ^ should match after a newline

An '@' followed by a name precedes a list of test cases for the named test "fixture". Each version has to implement each of the fixtures, but these are simple and I already have equivalent code in the existing tests.

Normally each line of values is a test case. Commas between values are optional, but newlines are ignored after a comma to allow splitting a test case over several lines.

After the Suneido code version it was straightforward to implement a version in Go. Java and C++ should also be simple.

I still want to run these tests as part of the existing automated testing, but that's easy to do by writing a test (e.g. in JUnit) that calls the test runner for the portable tests.

A remaining question is where to put the test files. Expecting them to be in the current directory or a subdirectory is easiest, but then each version will have its own copy and I'd have to keep them in sync. It makes more sense to have a single copy somewhere, but then I need some way to point each version at that central location. One option would be an environment variable but that can be problematic. Instead I decided I'd put a text file at the root of each project that would contain the directory path to the tests. (And if that doesn't fit, each implementation is free to handle this differently.)

My main concern with this was tests with a lot of different cases, where you'd use data driven tests. (like regular expressions) In other areas what I probably should have is more of a BDD (Behavior-driven development) style of tests that would form a kind of specification for Suneido. To keep this portable it would make sense to use the JBehave style that separates the specification from the implementation.

Go: When nil isn't nil

Go is a fairly simple language. But it's still complex enough to have some oddities. I recently ran into one of them.

Lets say you have a nil pointer and you assign it to an interface:

var p *T = nil

intfc interface{} = p

I expected intfc to now be nil, but it's not.

The explanation (it's even in the FAQ) is in terms of the implementation. An interface contains a pointer and a type. When we assign a nil pointer the interface type is set. And an interface is only nil if both the pointer and the type are "empty".

I understand the explanation and it's not hard to work around once you're aware of it. If you want to return an interface that compares equal to nil you just have to make sure you don't assign a nil pointer to it.

But I don't find the explanation very satisfying. 

First, justifying some external behavior by explaining how you happened to implement it seems wrong. (Although to be fair, maybe the explanation is intended to be in terms of what an interface means, not really how it's implemented.)

Second, the explanation doesn't explain why they chose to make it work this way. Granted, it's simple because the nil check is just for a zeroed value. But it doesn't seem like it would be much harder for the compiler to just check for a zeroed pointer and ignore the type. It seems like this would avoid the unexpected behavior with no real loss of functionality.

I did some searching, but the only justification I could find is that a nil pointer in an interface can still satisfy the interface and you can still call methods on it. Which is fine, but what does that have to do with whether the interface value compares equal to nil? I guess it would introduce a new oddity in that you'd have two kinds of nil interfaces only one of which you could call methods on.

The other issue is that (afaik) you can't determine if an interface holds a nil pointer without using reflection.

It's not a big deal, and I'm not hung up on it, it just seems odd. As far as languages are concerned, I would say Go has relatively few sharp corners to get caught on.

Go Suneido Spike

In software, a spike is some programming to investigate options or answer questions or to test design ideas. The term came from XP (extreme programming) and is used in agile and Scrum.

As a way to learn Go I've been investigating what it would be like to use it to implement Suneido. I've implemented various bits and pieces like lexical scanning, memory mapped file access, decimal floating point numbers, string concatenation, and hash tables.

In the Java implementation of Suneido I was able to leverage the Java virtual machine. In Go (as in the original C++ version of Suneido) I would need to write my own bytecode interpreter. To investigate this, I did an end to end spike from lexing to parsing to code generation to byte code interpreter. I even implemented a basic REPL (Read, Eval, Print Loop). All it currently handles are simple expressions like 100 + 50 - 25 or "hello" $ "world". But it's implemented with the right structure to flesh out into the full language. (The code is on Github if you're interested.)

I've written about 4000 lines of Go code so far. Not enough to be an expert by any means, but enough that I don't feel like a newbie anymore. It's been mostly low level code, I haven't done anything with goroutines and channels yet.

It's been remarkably painless. I think it helps to have a C/C++ background. Of all the languages I've dabbled in in recent years, Go has been the easiest to pick up and be productive in. That's partly due to the small, simple language, and partly due to the good default tools. The Java language wasn't hard to pick up, but the libraries and tools are definitely more complex.

The fast compiles are a key feature. After working in Java I think it would be hard to give this up. Considering the classic compile and link approach, running Go code seems just as fast as running Java code.

I almost find Go's simplicity a little disappointing. I love reading about complex languages like C++ and Scala and all the wild and wonderful things you can do with them. What software geek wouldn't love turing complete templates and type systems! Go doesn't have those kinds of things, and therefore doesn't have intricate books about them.

But as far as something I actually want to use, and not just read about - in that respect Go is great.

A Go Hash Map

I was a little disappointed to discover that the built in Go map doesn't allow interfaces as keys. Considering that interfaces are the way to do anything dynamic in Go, and that dynamic stuff often uses maps, it seems a little odd.

To act as a hash table key, you need equals and hash code methods. But it's easy to define an interface for that and require that keys implement it, similar to how Sort requires a container to implement Len, Less, and Swap.

I looked around to see what's out there and found a few options, but none really excited me. And I'm still learning Go, so it made more sense to implement it myself.

My first thought was to port my hash map code from cSuneido. But I wondered what Go's map was like. The code is straightforward but it uses an interesting approach that I haven't encountered before. It's a variant of separate chaining with each slot in the hash table being a bucket that can hold a small number of entries (e.g. 8). Additional overflow buckets can be chained together. In many hash table designs, collisions are a nuisance to be tolerated, but this design almost embraces them, by making the table 8 times smaller you assume collisions.

Buckets holding a number of entries are also better for cache locality than a linked list.

Another interesting feature is that the buckets have an additional byte per entry that holds the high byte of the hash code of the key. This helps in searching because if this piece of the hash code doesn't match then you can avoid comparing keys (which is cache unfriendly and also slow if keys are large or complex).

This design also works well for small tables since you can use a single bucket, which basically reduces it to a small array with linear searching, which is what you want for a few entries.

So I implemented this design in Go, following the C code fairly closely, except that I didn't implement the incremental resizing. It might be worthwhile in some situations, but it makes the code more complex (especially iteration) and probably makes the resizing slightly slower in total, albeit amortized. The lack of incremental resizing hasn't been a noticeable issue in cSuneido.

Have a look, it's about 200 lines of Go.

The next issue was what to use for hashing strings. Go has standard packages for hashing but they require converting to a byte array which requires allocation and copying. (Go 1.3 has an optimization for this, but only for the built in map.) So again, I wrote my own version, following the approach in hash/fnv.

It seems reasonable. The main drawback comes from Go's lack of generics - it has to work in terms of interface{} (the equivalent of Java Object or C/C++ void*) so you have to cast everything that comes out of it, reminiscent of Java prior to generics. Another minor awkwardness is that you can't use tbl[key] syntax like built in maps.

Another hash table approach which would be a natural fit with Go would be to use a growable slice for each entry in the table (rather than a list of buckets). This would avoid the space overhead from chain links and from partially full buckets, at the cost of the slice itself (a pointer and two int's), plus more individual allocations.

Related interesting reading:

• Fast and Compact Hash Tables for Integer Keys 
• Redesigning the String Hash Table, Burst Trie, and BST to Exploit Cache

CharMatcher in Go

The Guava library for Java has a CharMatcher that provides a way of composing character matching predicates plus functions that use those predicates.

For example, AnyOf("\r\n").Negate() creates a CharMatcher that matches any character except return or newline. You can then do things like cm.IndexIn(str) or cm.CountIn(str)

Some of this you can do directly with Go libraries. The unicode package provides some standard "matchers" like IsDigit and IsLetter. And the strings package has functions like IndexFunc and TrimFunc that take predicates.

But they don't do everything that CharMatcher does, so as an exercise I thought I'd try implementing something like CharMatcher in Go.

My first approach was basically an object-oriented style like I'd use in Java with CharMatch as an interface.

But when I started adding more matchers it seemed excessive to have to define three pieces for each - a struct, a match method for the struct, and a function to construct the struct.

My next thought was to get rid of the interface and have a generic struct containing a matching function as a member. This uses closures to store the matcher parameters rather than structs.

I was stuck on the idea of a struct so that I could define methods like Negate and IndexIn on it. Then I realized that in Go I could make CharMatch just a function, and still define methods on it. That led to this version:

I used InRange for DIGIT and AnyOf for SPACE as examples, these could also use the unicode package equivalents.

IndexIn is an example of a method that just wraps a strings package function, whereas CountIn has no strings equivalent.

The tests give some examples of how it's used.

One potential drawback of this approach is that the matcher parameters are "buried" in closures. This makes it impossible to do any processing or optimization (like the Guava CharMatcher precomputed method). For example, Is('a').Or(Is('b')) could be folded into AnyOf('ab'). If you wanted to do this, I think you'd have to go back to using structs (like my first approach).

More Concatenation

I did some quick benchmarks in Go. (I really like how that ability is part of the standard tools.) Here are some results. As with any benchmark, don't take them as exact. Changing the parameters of the benchmarks gives varying numbers but with the same overall result.

Buffer	37322 ns/op	51104 B/op	10 allocs/op
Array	49456 ns/op	53632 B/op	17 allocs/op
Linked	122558 ns/op	54047 B/op	1010 allocs/op
Merge	311005 ns/op	323552 B/op	1998 allocs/op
Naive	2225408 ns/op	5371680 B/op	999 allocs/op

An "op" in this case was appending 10 characters, 1000 times.

Some observations:

• Naive concatenation is indeed bad, both in speed and memory
• Almost anything is much better than the naive approach
• As expected, a buffer is the best in both speed and memory
• An array of substrings (without any merging) does surprisingly well
• For an immutable option, a linked list isn't too bad
• A merge tree was not such a good idea

Overlooking the Simple Solution

For no particular reason I've been thinking about concatenating strings. (I know, get a life, but this is at least part of my life.) It was partly prompted by thinking about Go and how it might work to implement different facets of Suneido. 

Most languages warn you about the poor performance of building a large string by repeated concatenation. They recommend using a StringBuilder (Java) or its equivalent. 

But Suneido explicitly optimizes repeated concatenation internally so the programmer doesn't have to worry about when to concatenate and when to switch to "building". 

In cSuneido (the original C++ implementation) concatenation just creates a linked list of substrings, deferring the actual allocation and copying. 

Originally, I ported that approach to jSuneido (the Java version). But I cut a few corners that I thought were safe to cut. That came back to haunt me. Rather than fix the problems I looked for a better solution. (There are some issues when the linked list gets too big.) I considered some kind of merge tree but decided that was more complex than necessary.

Instead of a linked list I used an array of pieces which was expanded as needed. If the array got big it would merge small pieces. That has been working fine. 

Analyzing the problem from a more theoretical basis I figured the worst approach is repeated naive concatenation and the best (?) is something like StringBuilder that uses a buffer that expands e.g. by doubling in size. My array approach is somewhere in between, as is a merge tree. 

At that point it struck me that I could just use a StringBuilder rather than my array approach. Duh!

That eliminated about a hundred lines of code and ran about 20% faster. 

I feel stupid for not thinking of this sooner. It seems so obvious (now!) But I was stuck on the idea of deferring concatenating. 

And now I feel even more stupid because I just searched my own blog and found that I did consider a buffer approach but decided it required too much copying. (String Building Internals) Using a StringBuilder does mean copying into it and then eventually copying the result back out. But considering that Java compiles string concatenation into using StringBuilder, the overhead can't be too big. (Go, on the other hand, compiles string concatenation into calls to a runtime function that allocates a new string and copies directly into it, without any intermediate buffer.)
One advantage of my original linked list approach is that everything is immutable and therefore threadsafe without locking. That's attractive. Both the array and the StringBuilder are mutable and require locking. That's not a big deal on Java since the locks will almost always be uncontested and therefore very fast. And the locking is at the leaves of the call tree so they should be safe from issues like deadlock. 

But in Go, locking is less acceptable and an immutable solution would be nice. I have an idea for an immutable merge tree approach - stay tuned :-)

Java 8, Eclipse Kepler, and Infinitest

When I updated my Mac to Java 8, Infinitest quit working. I've been hoping it would update and start working but it didn't happen.

I went looking and I found that the Eclipse Marketplace version of Infinitest is several years old. Had the project been abandoned?

I found an Infinitest web site which linked to the Github page. The readme there gave an update site of http://update.improvingworks.com/ and when I installed/updated from that site Infinitest started working again.

The web page gives an update site of http://infinitest.github.io which appears to point to the same version (5.1.110). I'm not sure which is the "correct" choice.

Now I'm just waiting for Proguard to be updated for Java 8.

Lexing in Go

One of my standard exercises when I'm looking at a new language is to implement the lexical scanner for Suneido. I've done this in quite a variety of languages - C++, Java, C#, D, and now Go. The scanner is simple, and generally the implementations are similar.

The Go code is longer (in lines) than most of the other implementations for a couple of reasons. One is that Go doesn't have ?: and you have to use if-else. Another is that gofmt puts enum type constants one per line.

Go only supports simple numeric constants to implement enums. That works ok, but it's awkward for debugging because if you print them, you just get a number.

One interesting thing about the Go implementation is that it handles unicode without really worrying about it too much.

I debated over whether to return the results as a struct or just as multiple return values. But that really depends on which is easier for the calling code.

There is a good talk by Rob Pike about Lexical Scanning in Go. If you're not interested enough to watch the video, you can skim the slides. I didn't need to use his fancier concurrent state machine design but it's an interesting example of using Go. You can see a full implementation in the Go template package.

Here's the code: (or view it on GitHub)

Hamcrest Style Matchers in Go

I've grown quite accustomed to using Hamcrest style matchers in Java. So I looked for something similar in Go. I found github.com/rdrdr/hamcrest, but the last activity was three years ago and when I tried to use it I got errors. (Go has changed) I also found Gomega but for some reason it didn't attract me.

I started to write my own, then stopped myself from getting side tracked from what I was doing. But I kept thinking about it, and ended up writing something very simple.

What I came up with allows you to write assertions like:

Assert(t).That(..., Equals(expected))

and to add more information to the error messages with:

Assert(t).That(..., Equals(expected).Comment("..."))

Where t is the *testing.T that Go's testing framework supplies.

Equals returns a tester function (a closure capturing the expected value). A tester returns "" on success, or else an error message.

That is a method that takes a value of any type (i.e. interface{}) and a tester function, and calls the tester with the value. If the tester returns an error message it calls t.Error

Comment is a method on a tester (taking advantage of Go's ability to define methods on any type, not just on classes). It returns a new tester (a closure capturing the message) that passes success ("") through unchanged, but appends it's message to any error messages.

Taking advantage of Go interfaces, I didn't make the code depend on Go's testing.T type. Instead I defined my own interface with a single Error method (matching the one in testing.T) and made Assert wrap that. So it will work with anything that has a suitable Error method. I didn't have any particular usage in mind for that, but it's a nice way to avoid dependencies.

Initially I wrote Equals using "==". That worked for simple values but not for things like slices. I ended up using reflect.DeepEqual which seems to work. I'm not sure if this is the best approach. Obviously it won't work for things like less than or greater than.

One of the problems I had was that errors would be reported as always occurring on the same line of my hamcrest.go file where I called Error rather than the relevant line in my test. This is a more general problem whenever tests use any kind of helper function that ends up calling Error. Maybe that's not the normal style, but I tend to do it a lot. I found the code where it does this in the decorate method in testing.go but there doesn't appear to be any way to override it. It would be easy enough to modify testing.go, but I'm not sure how I'd get "go test" to use it, short of building a custom version of Go which doesn't seem like a good solution. Ideally Go test would report the portion of the call stack within my code.

I ended up just adding my own reporting. Rather than hard coding how far back in the call stack to go (as testing.go does), I looked for the first call after the testing framework, i.e. the actual top level test function. So errors have the correct location on the end:

hamcrest.go:38: expected 5790 but got 579 {dbldisp_test.go:16}

Obviously, this is not a complete implementation of Hamcrest style matchers. It was a good exercise to explore some of Go's features like interfaces, function value, and closures. I've been using it to write tests but I'm not sure if I'll do more on it and use it longer term or find something else to use.

UPDATE: Something I forgot to mention is that when I'm using this I'm doing:

import . "hamcrest"

The dot allows you to use the exported names from hamcrest (e.g. Assert and Equals) without requiring the package name prefix. This is discouraged, but in this case it seems preferable to writing:

hamcrest.Assert(t).That(..., hamcrest.Equals(expected))

Here's the code on GitHub:

Decimal Floating Point Arithmetic

If you're doing things like financial calculations, you have to be careful about using conventional binary floating point because it can't represent decimal fractions exactly.

One approach is to use "scaled" numbers, e.g. represent your dollar amount in cents or hundredths of cents so you are always working in integers. And it requires big integers, 32 bits is only about 9 decimal digits and the 52 bits of double floats is about 15. You really need 64 bit integers which are about 19 digits. (10 bits ~ 3 decimal digits) But that still doesn't give you the ability to deal with general purpose floating point.

So Suneido has always had a decimal floating point numeric type. (Internally, for performance, it also uses plain integers when possible.) Another advantage of a decimal type is that it is simple and quick to convert to and from string form.

Back when I first wrote Suneido (~ 15 years ago) there were no 64 bit integers in C++ compilers ("long" was 32 bits) and 32 bits wasn't sufficient precision. So I had to use multiple values to hold the coefficient. Since I had to use multiple integers anyway, to simplify overflow (by using 32 bit ints for intermediate results) I used four 16 bit ints, each one holding four decimal digits for an overall precision of 16 decimal digits. (To simplify "shifting" the exponent is in terms of the 16 bit ints, i.e. it jumps 4 decimals at a time. This "granularity" causes problems with precision. Depending on the exponent, in the worst case you get as few as 10 decimal digits of precision.)

Of course, having to use multiple integers and trying to get decent performance complicated the code, especially division. I won't claim it's the greatest code, but nevertheless it's worked reasonably well for a long time.

When I implemented jSuneido, I used Java's BigDecimal. Because of the different implementation there were a few minor differences, but they were mostly edge cases that didn't matter in practical usage. (Unfortunately I had made the external dump format for numbers mirror cSuneido's internal representation so it's a little awkward converting to and from BigDecimals.)

Recently, we've started to run into issues with using API's that deal with 64 bit integers, because we don't have enough precision to store them. In jSuneido it would be easy to bump up the BigDecimal precision to 20 digits. In theory I could do the same with cSuneido, but unfortunately, the code is fairly specific to the current precision. e.g. loops are unrolled. The thought of making this change is not pleasant :-(

The other problem is that some of the code assumes that you can convert to and from 64 bit integers losslessly. But 20 decimal digits won't always fit in a 64 bit integer.

Now that we have 64 bit integer types, the obvious answer seems to be to use a 64 bit integer for the coefficient. This will be faster and simpler than using multiple small integers, and probably faster than BigDecimal since it handles arbitrary precision. And if I used the same approach in both cSuneido and jSuneido this would ensure consistent results.

Since I'm in the middle of playing with Go, I figured I'd try writing a Go version first. It should be relatively easy to port to C++ and Java if I decide to.

It took me a couple of days to write it. One of the challenges is detecting overflow when calculating with 64 bit integers, since you don't have a larger type to use for intermediate calculations. Hacker's Delight provided a few tips for this. Another useful reference was General Decimal Arithmetic.

It's about 500 lines for add, subtract, multiply, divide, and conversion to and from strings. (That's about half the size of the cSuneido C++ code.) Since I'm new to Go, it may not be the most idiomatic code. And I have only done basic testing and refactoring. "float10" isn't the greatest name. Maybe "decimal" or even "dec"? (in keeping with Go's predilection for short names) I'm open to suggestions...

I chose to pass and return by value rather than by pointer. I'm not sure if this is the best choice for a 10 byte struct. Would it be faster to pass by pointer? Returning by pointer forces heap allocation for intermediate results which isn't ideal. Pass and return by value is a good fit for immutable values which are my preference.

Go makes it really easy to benchmark so I checked the speed of division (the slowest operation). Micro-benchmarks are always dubious, but it gave me a rough idea. It showed about 400 ns per divide (on my iMac). I don't have comparable benchmarks for cSuneido or jSuneido, but that seems pretty good. I'm pretty sure it's better than cSuneido. (Of course, it's nowhere near as fast as native binary floating point done in hardware. The same benchmark with float64 gives about 7 ns per divide, although this is so small that it's even less likely to be accurate.)

As far as evaluating Go, so far I like it. Of course, it's well suited to low level code like this. Sublime Text + GoSublime works well. (The only issues have been with learning Sublime since I haven't used it much.) I might have broken out the debugger a couple of times if I'd been working in Java or C++, but I get the impression the debugger story for Go isn't that great. I managed easily enough with old school prints :-) I plan to give Eclipse + GoClipse a try at some point since I'm already familiar with Eclipse.

The code is embedded below but it's probably easier to read (or download) on GitHub.

TortoiseSVN + TortoiseHg Problem

I use Subversion (SVN) for cSuneido (for historical reasons) and Mercurial (Hg) for jSuneido, both on SourceForge (again for historical reasons).

On Windows I use TortoiseSVN (1.8.5) and TortoiseHg (2.11.2) with Pageant (part of PuTTY, but supplied with TortoiseHg) so I don't have to type a password all the time. This combination has worked well for a long time.

I came into work this morning and TortoiseSVN kept popping up a Plink dialog asking for my password. That's what Pageant is supposed to avoid, especially since SourceForge needs an SSH key, not a password.

TortoiseHg was working fine, which meant Pageant was ok.

I used TortoiseSVN a few days ago. As far as I can recall I didn't change anything since then. But possibly I updated it. There are so many updates going by these days that it's hard to remember.

I searched the web but didn't find anything that seemed to be related.

I tried rebooting. I tried changing my path to put TortoiseHg and TortoiseSVN in the opposite order. Didn't help.

After some digging I found TortoiseHg was using older versions of TortoisePlink and pageant (both from 2012) whereas TortoiseSVN had a new TortoisePlink (from 2014). I wasn't sure it was a good idea, but I tried replacing the new TortoisePlink with the old one, thinking that maybe it needed to match the version of pageant.

That worked! Or at least appears to work. (I even rebooted to make sure the problem wouldn't come back.) It's probably going to break next time I update TortoiseSVN, and I'll probably forget the fix, but at least I'll have this blog post to jog my memory :-) And hopefully in the long run this will get sorted out. I can't be the only person running both. I'm not sure why TortoiseHg has such old versions. There seem to have been similar version issues a few years ago.