Streaming Functional

Suneido has functional aspects like map and filter. They're not a major part of the language but they're definitely useful. However, it has always bothered me that they aren't "streaming". An operation like Map or Filter always generated a new collection. For small collections and many usages that works ok. Obviously it doesn't allow you to work with infinite streams but that's not a common requirement for our applications. But it's annoyingly inefficient if the desired result isn't a collection. e.g. If you end up joining the results into a string or writing them to a file.

I've thought about how I could change Suneido's functional methods to be streaming but it alway seemed like it would be a big change. When Java 8 introduced its streaming methods (for similar reasons) it was another reminder. However I didn't really like their approach - having to start by calling .stream and end by with some kind of collector. It makes sense and works well but it's verbose. One of the things I like about functional style is that it tends to be quite succinct.

The other day I was thinking about it again and I realized I might be closer than I thought, that Suneido might already have most of what I needed.

Early on in Suneido's development I'd face a similar problem with iterating over the members of an object. The Members method returned a new collection but that was wasteful if you just wanted to iterate. So I'd introduced "sequences" which could be iterated over without instantiating a new collection. Basically they just wrapped an iterator. However, they could also be treated as collections. If you called a collection method on them, they would instantiate the collection automatically. This meant a single method like Members could be used both to iterate lazily, and to generate a new collection, automatically and invisibly.

Later I added a Sequence function so that you could write these kinds of lazy sequences in user code. For example, we used it for a string lines method. But it never got used extensively.

My insight (as usual, obvious in hindsight) was that Sequence provided almost everything I needed to make the functional methods streaming. I simply rewrote Map and Filter as iterators and wrapped them in Sequence.

I wrote them as standalone functions, but also made them methods on collections. I prefer to write list.Filter(...).Map(...) than Map(Filter(list))

But I immediately ran into a problem. list.Filter(...).Map(...) worked, but it wasn't streaming. i.e. The result of Filter was still getting instantiated. The reason was that the sequence/iterator didn't have a Map method so it would instantiate the collection which did.

That was easy to fix, although it required a change to the runtime implementation of Sequence. There was no reason that you couldn't call user defined collection methods on sequences. It was only the built-in collection methods that required instantiation.

The next problem I ran into was that people were misusing Map purely for side effects. They should have been using Each instead. That was easy to fix, but somewhat tedious to examine every use of Map.

Then I found that Join would trigger an instantiation because it was a built-in collection method. So I added a built-in Join method on sequences.

I also found that laziness (as with these streams) and mutability don't play well together. I ended up needing to explicitly instantiate sequences in a few places because otherwise things would change out from under them. This is ugly and I'm not really happy with it, although thankfully it seems quite rare. But I could see it stumping someone who wasn't aware of how sequences work.

Being explicit about when to transition between collections and streams (like Java 8) avoids many of these issues, but doing it automatically fits better with Suneido's philosophy.

This project was a bit of a roller coaster (aren't they all?) alternating between thinking it was easy and I was done, with finding "fundamental" flaws without obvious solutions.  Unusually, even quite flawed implementations would handle most (simple) scenarios. It was the corner cases that gave me grief.

After hanging Suneido a few times I ended up adding a way to mark infinite sequences and refusing to instantiate them or display their contents. For fun I wrote a Primes infinite sequence.

Overall I'm pretty happy with how it turned out. I haven't done many benchmarks. I'm sure I can come up with scenarios where it will be a big win, but I doubt it'll make much difference in the larger picture.

One weakness is that Suneido doesn't have any easy way to write sequence operations. Currently they have to be written as iterators, which is ok for simple stuff but not very nice for things like tree traversals. Ideally you'd have some kind of "generators" with "yield". A project for another time!

Concurrency is Hard episode n+1

For almost a year a bug has been haunting me. Once every few thousand server startups, the server would hang. It wasn't totally frozen - the server monitor web page was still running and the server would still accept connections (but not service them). Simply restarting would fix it. There was no discernible pattern, customers got it at random and there didn't seem to be anything in common between them.

At first we wrote it off to bad hardware or server glitches. But it continued to happen a little too consistently for that to be the cause.

It was hard to pinpoint exactly when it started since we didn't pick up on it right away. But we knew the rough time frame. So I could look at version control and see what changes I'd made. But nothing seemed like it could cause it. I could have rolled back changes, but most of them were bug fixes so I wasn't keen to do that. And for all we knew it could have been a Windows update or a Java update or changes to the application code that triggered it.

Once we recognized we had a problem, my first thought was that it was some kind of deadlock during the startup sequence. No problem, Java has ThreadMXBean findDeadlockedThreads. It worked great when I forced deadlocks. I scheduled it to run five minutes after startup. It didn't find anything. I scheduled it to run every five minutes. Still didn't find anything. So either it wasn't a deadlock, or the deadlock detection wasn't working. In other words, no progress.

It seems terrible that we had this bug for almost a year. I did work on it, and many hours in total, but it was never really urgent. An individual customer might get it once every six months and they simply had to restart. So there wasn't a lot of external pressure to fix it. It certainly "bugged" me but with no clues it was hard to even know what to try.

I studied the code. I rewrote large sections. I ran the ThreadSafe static analysis tool on it. We added logging in numerous places.

We did stress testing to try and make it happen, although it's hard to "stress test" server startup. All we could do is repeatedly start and stop the server. Of course, it never happened when we did this.

What I really needed was to examine a hung system with a debugger (e.g. jvisualvm). But we only install a JRE on our customers servers, not the full JDK. If it happened consistently on one customer we could have installed the JDK on their server but it didn't. Towards the end I was starting to think about installing the JDK on all our customers. I would have been happy even to make the bug worse so it happened more frequently. At least then I'd have a chance of getting it under the microscope.

Looking at the Java documentation I found you could debug core dumps. So next time we got a chance to access a hung server, we forced a core dump. But we're using a launcher to run as a Windows service, and the launcher meant the debugging tools didn't recognize the core dump. So much for that idea.

One of the problems with Java's "synchronized" keyword is that you can't put any timeouts on it. So towards the end I was considering replacing all my uses of synchronized with explicit locks with timeouts. At least then deadlocks would time out and give me a clue where they were happening. In retrospect, this would have worked to locate where the problem was.

One of the problems was the slow turnaround. After making a change to jSuneido we would run it internally for at least a week. Then we'd roll it out to beta customers, and then finally to the rest of the customers. Then we had to wait for several days to see if the problem reoccured. So after each change I'd have to wait several weeks to see if it made a difference. Of course, there was nothing stopping me from continuing to work on the problem during this time, but the natural inclination was to want to wait and see if the change had worked.

One pattern we noticed was that it seemed to be related to requests to the HTTP server during startup. We weren't aware of that last time we stress tested so I thought it might be something to try. So I set up a server to restart every minute and arranged for a constant stream of HTTP requests. It ran all day without a hiccup. I normally shut down my computer at night. But that was only about 500 restarts, not really enough to have a good chance of detecting something that only happens every few thousand times. What the heck, I left it running overnight.

When I went to check it in the morning I had no expectation of finding anything other than it happily running. So it took me a minute of staring at the screen to realize it had hung! Eureka! Funny to be so excited about managing to hang your software!

I was almost afraid to touch it, for fear of disturbing the hung process. I fired up jvisualvm and did a thread dump. As I'd expected from the beginning, it was a deadlock. Two threads each owning a lock that the other wants.

And still the deadlock detection didn't show anything. I realized the reason was that one of the threads was not waiting on a lock, it was "parked" in Guava cache code. The code it was in was documented as "thread safe". As Inigo says in The Princess Bride, "You keep using that word. I do not think it means what you think it means."

I briefly entertained the (attractive) thought that the deadlock was in the Guava code and not my fault. That would have been nice, although harder to get fixed. But unfortunately it was quite obvious from the thread dump that it was my code that was at fault, despite the one thread being parked in Guava code.

The problem, and one of the lessons from this bug, is that code that calls a user supplied function is never truly thread safe because you can't control what that function does. That's also one of the problems with "functional" code, it is not amenable to static analysis.

In hindsight I realize that ThreadMXBean also has dumpAllThreads. I might have been able to write code to detect when the server hung and dumped the threads automatically. That would have given me the information I needed to solve this. Of course, I had no way to know that before now.

When I started rewriting Suneido from C++ to Java one of the big reasons was that we needed a multi-threaded server, and I thought Java had good concurrency support. But what it has is locks and shared state, which we've come to realize is not a great way to handle concurrency. Even back then I knew that, and I tried hard to minimize locking (which also helps performance) by keeping state thread contained and using immutable data. But I still used a certain amount of locking and concurrent data structures (which commonly use locking). And it's mostly worked out pretty well - jSuneido hasn't had a lot of concurrency problems. However, it's a bit like the lack of safety and garbage collection in C++, if you're careful you can get away with it, but sooner or later it's going to catch you.

Bug Tale

Yesterday, towards the end of the day, one of my staff came to me with a bug in cSuneido. Actually it was the second time he'd brought it to me. The first time I sent him away with a request for more details. This time he brought me a printed screen shot of the error. Perhaps that seemed more tangible.

Sometimes I get the impression that my staff gets a certain amount of secret enjoyment out of pointing out my bugs. Some of that's probably just my imagination and frustration. But even if it's true, it's understandable. They have to put up with me critiquing their code all the time.

I wasn't happy about it. It had been a busy day with the usual parade of interruptions. I didn't feel like I'd accomplished anything, and now this steaming turd had been deposited on my desk. I started to work on the bug, although I was pretty sure there wouldn't be enough time left in the day to get anywhere on it. The only advantage of starting late was that the interruptions had died out now that most people had left for the day.

On the positive side, the bug was repeatable. On the negative side it wasn't simple to reproduce - you had to run client-server, open our application, and then run a bunch of queries for over a minute. After thinking I'd fixed it several times I realized that it also only happened the first time you did this after starting the client. Subsequent runs worked fine.

The error itself was an access violation. The more I work in "safe" languages like Java or JavaScript or Go, the more I hate unsafe languages like C++. An access violation could be anything - a dangling pointer, a garbage collection issue, an uninitialized variable, an invalidated reference ...

On top of this, the error occurred inside a background fiber. Even better, it didn't occur when I ran the debug version of cSuneido.

As I expected, I didn't get anywhere before I headed home for the day, pissed off.

After supper I debated whether to work on it some more. If I could make a little progress, even just get a clue or narrow it down, then I'd end the day feeling a lot better. On the other hand, if I just ended up banging my head on the wall, I'd probably feel even worse.

I took the gamble, of course. Programmers are nothing if not eternal optimists. They have to be. But I hedged my bet by not "getting serious" and firing up the big iMac. I just sat in the living room with my laptop. That way if I failed I could tell myself it was because I'd just been poking around.

I didn't find the bug, but I did narrow it down enough that I felt I was hot on the trail. I could recreate the problem with a much simpler test case, and I'd found that I could recreate it in the debugger as long as I used the release build. It's harder to debug in the optimized version but being able to use the debugger at all was a big help.

It turned out the only significance of the queries running for over a minute was that during that minute several timer tasks got queued and ran concurrently when the queries ended. I could get the same result by just starting two fibers "at the same time".

Thankfully, the next day I was working at home and could focus on the problem. It was quite puzzling at first. I could see (in the debugger) exactly where the problem was, but the code looked correct, and almost all the time it worked correctly. I even resorted to looking at the assembly language and register contents, something I haven't done for a long time.

Stepping through the code I found there was a fiber context switch in the middle of the problem lines. And from looking at the assembler it was pretty obvious the compiler was caching the results of some common subexpressions, which it probably wasn't doing in the debug version. But I couldn't see how that caused a problem.

With fibers being cooperative and not pre-emptive, you don't really need to worry about concurrency issues. But this turned out to be a concurrency issue after all.

The problem lines were:

tls().thedbms = dbms_remote_async(server_ip);
tls().thedbms->auth(tok); 

tls() was getting evaluated and cached. But if dbms_remote_async "blocked" waiting to connect, then another fiber would run, and if that other fiber created a new fiber, and this caused the fibers vector to grow (reallocate), then the cached value of tls() would be invalid, causing the access violation.

Sure enough, if I called reserve on the vector to pre-grow it, then the problem went away.

It only happened the first time because after that the vector wouldn't need to grow and the tls() reference would stay valid.

I was grateful that the problem was so repeatable. If this had been real threads it would have been much more erratic. One of the advantages of fibers is that they are deterministic.

One local fix was to rewrite it as:

auto dbms = dbms_remote_async(server_ip);
tls().thedbms = dbms;
dbms->auth(tok); 

But where else in the code did I have this potential problem? And what would stop me from reintroducing the problem in future code.

My next thought was that I needed to tell the compiler that tls() was "volatile", i.e. it could change concurrently. But that wasn't really the problem. Even in single threaded code inserting into a vector invalidates any references, that's part of its contract.

One option was to use Windows fiber local storage. This didn't exist back when I rolled my own.

Another option was to dynamically allocate the tls structure so it didn't reside inside the vector.

However, there could potentially be other reference into the vector. I'd had problems with this in the past and "solved" them by using indexes into the vector rather than pointers. But it was still something I had to continually be on the lookout for.

Instead, I decided to switch from a vector to a simple fixed size static array. cSuneido isn't designed for huge numbers of fibers anyway. References into a fixed size static array were safe, nothing can invalidate them.

Problem solved (fingers crossed) and my mood has distinctly improved :-)

If you're interested in the nitty gritty, the change is on Github

Windows Overlapped Socket IO with Completion Routines

Up till now, cSuneido has used WSAAsyncSelect to do non-blocking socket IO. But WSAAsyncSelect is deprecated and it's not the nicest approach anyway. cSuneido needs non-blocking socket IO for background fibers, the main fiber uses synchronous blocking IO. (Although that means the main fiber will block background fibers.) Note: Windows fibers are single threaded, cooperative multi-tasking, coroutines. The advantage of fibers is that because they are single threaded and you control the context switches, you don't have the concurrency issues you would with "real" preemptive threads.

I thought that the WSAAsyncSelect code was the source of some failures we were seeing so I decided to rewrite it. My first rewrite used a polling approach. I know that's not scalable, but cSuneido doesn't do a lot of background processing so I figured it would be ok. Basically, I put the sockets in non-blocking mode, and whenever an operation returned WSAWOULDBLOCK the fiber would give up the rest of its time slice (e.g. 50ms) This was quite simple to implement and seemed to work fine.

However, I soon found it was too slow for more than a few requests. For example, one background task was doing roughly 400 requests. 400 * 50 ms is 20 seconds - ouch!

Back to the drawing board. One option was to use WSAEventSelect, but it looked complicated and I wasn't quite sure how to fit it in with the GUI event message loop.

Then I saw that WSARecv and WSASend allowed completion routines, a bit like XMLHttpRequest or Node's non-blocking IO. This seemed like a simpler approach. The fiber could block (yielding to other fibers) and the completion routine could unblock it.

At first I thought I had to use WSASocket and specify overlapped, but it turned out that the regular socket function sets overlapped mode by default. That's ok because it has no effect unless you use WSARecv or WSASend in overlapped mode.

Sending was the easy part since there was no need to block the sending fiber. It could just "fire and forget". One question was whether it would always do the full transfer or whether it might just do a partial transfer and require calling WSASend again (from the completion routine) to finish the transfer. I couldn't find a definitive answer for this. I found several people saying that in practice, unless there is a major issue (like running out of memory), it will always do the full transfer. Currently I just have an assert to confirm this.

Receiving is trickier. You may need to block until the data is available. And the completion routine may get called for partial data in which case you need to call WSARecv again for the remainder. (Although this complicates the code, it's actually a good thing since it allows you to request larger amounts of data and receive it as it arrives.)

WSASend and WSARecv can succeed immediately. However, the completion routine will still be called later. And for WSARecv at least, "success" may only be a partial transfer, in which case you still need to block waiting for the rest.

One complication to this style of overlapped IO is that completion routines are only called when you're in an "alertable" state. There are only a handful of functions that are alertable. I used MsgWaitForMultipleObjectsEx in the message loop, and SleepEx with a zero delay in a few other places. Note: although the MSDN documentation is unclear, you must specify MWMO_AWAITABLE for MsgWaitForMultipleObjectsEx to be alertable. (And it has to be the Ex version.)

Each overlapped WSASend or WSARecv is given an WSAOVERLAPPED structure and this structure must stay valid until the completion routine is called. I ran into problems because in some cases the completion routine wasn't getting called until after the socket had been closed, at which point I'd free'd the WSAOVERLAPPED structure. I got around this be calling SleepEx with a zero delay so the completion routines would run.

When I looked at some debugging tracing I noticed that it seldom blocked for very long. So I added a 1ms SleepEx before blocking to see if the data would arrive, in which case it wouldn't need to block and incur a context switch. This eliminated some blocking, but sometimes it didn't seem to work. I realized it was probably because the sleep was getting ended by an unrelated completion routine (e.g. from the preceding write). So I added a loop to ensure it was waiting at least a millisecond and that fixed it. Of course, I'm testing with the client and server on the same machine so the latency is very low. Across the network it will be slower and will still need to block sometimes.

Although the code wasn't that complicated, it took me a while to get it working properly (i.e. fast). As always, the devil is in the details. But the end result looks good. Background socket IO now runs about 30% faster than the old WSAAsyncSelect version, and almost as fast as the foreground synchronous blocking IO.

Java ByteBuffer Gotcha

I had a weird bug in some Java code in jSuneido that took me a while to figure out. I briefly thought I'd found a bug in the ByteBuffer implementation, although I realized that was a low probability. (Especially given the confusing nature of ByteBuffer.) In the end it makes sense, although it perhaps could be documented better.

Here's the scenario - you slice or duplicate a ByteBuffer. This makes a new ByteBuffer instance that shares its data with the original. Then you compact the original buffer. Note - this does not modify the data that you are interested in. However, it does move it.

ByteBuffer buf = ByteBuffer.allocate(8192);
for (int i = 0; i <= 300; ++i)
    buf.put((byte) i);
buf.flip();
for (int i = 0; i < 12; ++i)
    buf.get();
ByteBuffer slice = buf.duplicate();
System.out.println(slice.get(0));
buf.compact();
System.out.println(slice.get(0));

This will print 0 and then 12, i.e. the slice has changed, even though you didn't explicitly modify the buffer.

In retrospect it's obvious - compact does alter the buffer, which is shared with the slice. So the contents of the slice "changes".

I'd be tempted to add a warning to the documentation for compact that it will "invalidate" any existing slices or duplicates, the same way that C++ documentation points out which operations invalidate iterators. But maybe it should be obvious.

I'm mostly posting this because my searches for the problem didn't find anything. Perhaps this will help someone in the future find the problem quicker.

Mobile Git

I've used the Textastic source code editor iOS app on my phone and tablet for quite a while. I don't do a lot of heavy editing with it, but it's handy when I'm thinking about code and want to look at how something is implemented. (Textastic is also available on the Mac, but I've never tried that version. I have plenty of editors there, and I prefer ones that are also available on Windows.)

One of the limitations was that it wouldn't pull directly from Git. It does pull from Dropbox though, so I ended up keeping a copy of my source code there. But that had a few drawbacks. First, I had to remember to update the Dropbox copy of the code. Second to update the Textastic version I had to download the entire source, which could be slow depending on the connection.

Recently I discovered the Working Copy Git iOS app which lets you clone Git repos to your phone or tablet. You can even commit changes, although I generally wouldn't want to do that when I couldn't test the changes. It would be ok for documentation. It has its own editor, but even better it uses new iOS features to let you access the files from other editors, like Textastic.

Now, not only do I have easily updated offline access to my source code, I have offline access to my Git history.

One minor disappointment was that neither of these apps has syntax highlighting for TypeScript. I was a bit surprised because Typescript seems as popular as some of the other languages they include. Textastic does support Textmate bundles so in theory you could add it that way, but it didn't sound easy so I haven't bothered yet. It would be easier (for me!) if they just included it out of the box.

Unfortunately for Android users, both of these apps appear to be iOS only. If you know of good Android alternatives, let us know in the comments.

Progress on suneido.js

A project like suneido.js goes through distinct stages.

The first stage is just thinking and researching the platform. (In this case JavaScript, ES6/2015, TypeScript, node.js etc.) I've been doing that in the background for quite a while.

But eventually you have to start writing some actual code, and that's what I've been doing lately. This can be a frustrating stage because it's three steps forward, two steps back (and that’s on a good day). You pick an area to implement, decide on an approach, and write some code. That often goes smoothly. But then you start on the next area and you realize that the approach you picked for the first area won't work for the next. So you come up with an approach that works for both areas and go back and rewrite the existing code. Rinse and repeat.

Progress seems painfully slow during this stage. Important advances are being made, it’s just that it’s in terms of design decisions, not lines of code. (On the positive side, the Suneido language is, in many ways, quite similar to JavaScript, so the impedance mismatch is much lower than with C++ or Java. Meaning I can map many features quite directly from one to the other.)

There's also learning involved since I'm on a new platform, with a new language and tools. I went through the same thing with jSuneido since I'd never worked in Java before I started that project. It's one thing to read books about languages and platforms. It's a whole 'nother story using them to implement a large project. (I have to say, ES6/2015 features have been a big plus on this project, as has TypeScript.)

Eventually the approach firms up and all of a sudden you can make rapid progress in the code. It almost becomes mechanical, simply translating from jSuneido or cSuneido. Of course, issues still arise as you encounter different corners and edge cases. But mostly they are easily addressed.

It reminds me a bit of search strategies like simulated annealing where you start by making large changes all over, and as you get closer to a solution, you “cool down” and the changes are smaller and smaller as you approach a solution. Of course, it doesn’t mean you’ve found the “best” solution. But hopefully the initial larger explorative jumps covered enough of the search space that you’ve found something reasonably close to optimal.

I’m always surprised by just how many dead ends there are. Of course, when you’re programming the possibilities are infinite, so a lot of them have to be wrong. On the other hand, there is supposed to be intelligent design at work here, which you'd think would avoid so many dead ends. But many things are hard to evaluate before you actually try them. (Which brings to mind Wolfram’s idea of computational irreducibility.)

The reward for this plodding is that seemingly all of a sudden, I have a system that can actually handle real problems, not just limited toy examples. There is still lots to do, but suneido.js can now successfully run some of Suneido's standard library tests. It takes a surprising amount of infrastructure to reach this stage. Even “simple” tests tend to use a lot of language features.This seems like a big leap forward, but I know from implementing jSuneido that it’s really just another small step in a long process.

It's a bit hard to follow the project I'm afraid. The TypeScript code is on GitHub, but the actual transpiler is written in Suneido code and lives on our private version control system. And running the system requires jSuneido (which is used to parse Suneido code to an AST) and cSuneido (for the IDE), plus node.js and Typescript and other bits and pieces. I really should try to simplify or at least package it so it's easier for other people to access.

I'm glad to have reached this (positive) point, since I’m heading off travelling again, and probably won’t get too much done on this for a while.

PS. Hearing about some paragliding deaths recently I thought (not for the first time), "what if I get killed paragliding?". And my first thought was, "but who would finish suneido.js?". Ah, the mind of a true geek :-)

Old School

As I progress on suneido.js I'm accumulating a number of miscellaneous commands I need to run regularly. So far I've avoided any kind of build tool. I started writing shell scripts (OS X) and batch files (Windows) which worked but is ugly. And led to git problems with executable permissions.

So I started looking at build tools. The big one these days seems to be Gulp but so far I don't need its fancy streaming and plugins. So I searched for something simpler, and came across a comment that said something to the effect "if you're from those days, you could use make". They likely meant it in a derogatory way, but it made sense to me. I already have make on Windows and OS X and know how to use it. And it's dead simple to use it to simply run a few commands.

So now I can run "make bundle" or "make test". It's perfect for what I need. The hip kids can shake their heads all they want.

Modules Again

Things had been running smoothly in suneido.js with the UMD approach. Until today when I went to run my code in the browser (I hadn’t done this for a little while because I’d been working on transpiling) and I got errors saying one of my modules couldn’t be found. I realized that I had started using a Node.js module in my code, which of course couldn’t be found in the browser. Crap!

I searched the web and most recommendations were to use browserify. It sounded straightforward, and it made sense to combine all my JavaScript into one file. So I installed browserify and switched my tsconfig.json back to commonjs.

But it wouldn’t work. It kept telling me it couldn’t find one of my modules. But as far as I could tell there was nothing different about that particular module or where it was imported.

I finally remembered that the flashy dependency graph feature of Alm had shown I had a circular dependency. Was that a problem with browserify? Searching the web didn’t find anything definitive, but there were some references to it being problematic.

It wasn’t hard to fix the dependency (and I’d been meaning to do it eventually anyway).

But it still didn’t work! Then I realized that the code shown in the Chrome developer tools debugger wasn’t my fixed version. Caching? A little more digging and I found you can right click on the refresh button and pick “Hard Refresh and Reset Caches”. Finally it worked! (Although I wondered if any of my other approaches would have worked if I’d done this???)

It seems like a reasonable solution, other than taking a dependency on yet another tool and adding another build step. (I do have to admit that npm makes it easy to install these tools.)

Alm TypeScript Editor

When I was reading TypeScript Deep Dive (recommended) I noticed a mention of an "alm" TypeScript editor. I'd never heard of it so I figured I'd better check it out.

The developer turned out to be the same as the author of the book - Basarat Ali Syed aka "bas". Who also turned out to be the the original developer of the atom-typescript plugin that I've been using.

Alm is a new project but it's moving quickly and is already full featured. And there's actually documentation :-) It has features like go to definition, find references, and even rename refactor. It also has some flashier features like dependency graphs and AST views. It has an outline side bar (alm calls it a Semantic View) which is a feature I really like in Eclipse and miss in Visual Studio. (We also have it in Suneido.)

The current "packaging" is a little quirky - you start it from a command prompt and the actually UI runs in Chrome. It would be nice to see it packaged with Electron, like Atom, but that's not critical.

You can use Chrome's Save To Desktop to open it as a "bare" window without the browser chrome, but you still have to start Alm first. No doubt there's a way to automate that. Or you can use "alm -o" which will open it in a tab in Chrome, and then use something like the Chrome extension Open as Popup.

I was interested to see that it was using the CodeMirror JavaScript code editor component which is what we have been using in the suneido.js project. But recently it changed to use the Monaco editor which was written for Visual Studio Code and recently released as a separate open source component. That makes sense because Monaco is written in TypeScript, and TypeScript was the original target language for Visual Studio Code.

Alm leverages the actual TypeScript compiler for source code analysis, which seems like the ideal approach.

I've only used it for a few days and I'm still learning my way around, but it looks like it will be my preferred editor for TypeScript.

TypeScript Modules

TypeScript will translate ES6 (ES2015) modules into a variety of formats e.g. CommonJS for Node.js or AMD for require.js in browsers.

I found it a pain to have to switch my tsconfig.json back and forth to run my tests in Node.js or to run the actual code in the browser. Inevitably I'd forget to switch and/or recompile and it wouldn't work. I had to check one or the other version of tsconfig.json into Git, but that would confuse anyone who downloaded and tried to run the code.

I could probably set up a build process to generate both versions but I never got around to figuring that out.

I knew some code was written in a way that would work with both CommonJS and AMD and wondered why TypeScript didn't generate that format. Guess what, it does :-) TypeScript calls this output "UMD" or "isomorphic". (See TypeScript Modules under Code Generation)

    "compilerOptions": {
        "module": "umd",

The generated code is slightly larger since it basically contains both versions, but that's not a big deal.

Eventually (I hope) software will support ES6/2015 modules natively and you won't have to translate at all.

For backgrounds see What is AMD, CommonJS, and UMD?

TypeScript const enums

In my previous post I mentioned that if you make an enum const then you don't get the run time bidirectional mapping between numbers and names. It turns out that's not quite true.

If you turn on the preserveConstEnums compiler option (set it to true) then TypeScript still emits the mapping object, even though it still in-lines the enum values.

That seems like the best of both worlds. But I couldn't get it to work. If you try to use the mapping, you get:

import { Token } from "./tokens"
...
Tokens[token]

=> A const enum member can only be accessed using a string literal.

I searched online and found some suggestions to type assert to "any", but it didn't work.

import { Token } from "./tokens"
...
(Tokens as any)[token]

=> 'const' enums can only be used in property or index access expressions or the right hand side of an import declaration or export assignment.

So I gave up and didn't use const.

Then I came across a slightly different type assertion that worked!

import * as tokens from "./tokens"
...
(tokens as any).Tokens[token]

As the saying goes: "All problems in computer science can be solved by another level of indirection"

Note: As is recommended, I'm using the newer "as" form of type assertions instead of the older <type> form.

Lexer in TypeScript

One of the standard things I do when I'm investigating a new language is to implement Suneido's lexical scanner. Not because this is necessarily the best test of a language. It's more that I'm usually evaluating the language in terms of how it would work to implement Suneido. I realize that's a very biased view :-)

I wasn't planning to do this with TypeScript / JavaScript because suneido.js isn't intended to be an "implementation" of Suneido. It's a transpiler that runs on jSuneido (using the jSuneido lexer and parser) that translates Suneido code to JavaScript. The only manually written TypeScript / JavaScript code is the runtime support library. (And that's partly a bootstrapping process. Once the transpiler is functional much of the support library could be written in Suneido code.)

But as I implement the runtime support library, obviously I need tests. And I'm getting tired of writing the same tests for each version of Suneido. (Or worse, different tests for each version.) That's why I started developing "portable tests" some time ago. So I decided I should implement the portable test framework in TypeScript. At which point I remembered that this required a Suneido lexical scanner.

So I decided to write yet another version. I took about a day, which is about what I expected. And some of that time is still learning TypeScript and JavaScript. I based it mostly on the C# version. The Go version is more recent but Go has more language differences.

Overall, I was pretty happy with how it went. I find the repetition of "this." a little ugly. In Suneido you can abbreviate this.foo as just .foo, which is a lot more compact and less "noise" in the code, while still making it clear it's an instance variable.

I find JavaScript's equality testing somewhat baroque, the recommendation is to use === most of the time, but that won't always work for strings (because of primitive strings and object strings). And then there's the quirks of -0 versus +0, and NaN.

ES6 arrow functions and the functional methods like map are nice. And spread / rest (...) and for-of are good. I only specified types on function parameters and returns, but this worked well and gave me very little grief.

I was pleasantly surprised to find that TypeScript enum's automatically create a reverse mapping from name to number. Although the gotcha is that if you make them const you don't get this because they are inlined. Using a simple numeric enum doesn't let me attach extra information for parsing, but I'm not planning to write a parser so that's not a problem. The other issue is debugging, where all you see is a number.

Removing const from my token enum exposed another issue. I was relying on Atom's auto-compile but it only compiles the file you are editing, not the other files that may depend on it. So I went back to running tsc -w in a separate terminal window. (Open question - does anyone know why tsc lists some of my files as relative paths and some as absolute? It's consistent about which files, except that gradually more are switching to absolute. It's not causing any problems, I just can't figure out why.)

Although it can mean shorter code, I'm not a fan of "truthy" and "falsey" values. It was a source of problems 30 years ago in C. Now I got caught by it with a function that returned a number or undefined. I expected undefined to be "falsey" but I forgot that zero is also "falsey". With Suneido I took a stricter approach that things like if and while only accept true or false and throw an exception for any other type of value.

Now that I have a lexical scanner I can implement the portable test framework, which also shouldn't take me too long.

Versions, in order of age. cSuneido and jSuneido are production code and more full featured than the other more experimental versions.

• TypeScript tokens.ts, lexer.ts, lexer_test.ts
• Go tokens.go, lexer.go, lexer_test.go
• C# Tokens.cs, Lexer.cs (tests at the end)
• D token.d, lexer.d (tests at the end)
• Java (jSuneido) Token.java, Lexer.java, LexerTest.java
• C++ (cSuneido) scanner.h, scanner.cpp (tests at the end)

Going in Circles

Suneido has a bunch of built-in functions equivalent to its operators. e.g. Add for '+'  These are useful, for example, in functional style programming e.g. list.Reduce(Add)

I noticed that we didn't have a compare function i.e. that would return -1 for less than, 0 for equal, or +1 for greater than. This can be useful in some cases (e.g. in a switch) and avoids needing two comparisons. (Note: I didn't have a need for it, just noticed we didn't have it.)

A few days later I decided to implement it. It was simplest in jSuneido since there was already an internal compare function that I just needed to expose. Whereas cSuneido has the C++ / STL style of using operator < and operator == to implement all the other comparisons. So it needed two comparisons.

Then I started to question why these functions needed to be built-in. Why not just write them in Suneido code in the standard library. The difference in performance would be minor. So I ripped them out of cSuneido and jSuneido and defined them in the standard library. I also updated the documentation.

Then I started to wonder how often these built-in functions are actually used. As you can probably guess, almost never. On top of that I found several places where programmers got confused and thought they needed to use these functions instead of just the operators. (I fixed these.)

So I ended up ripping out the standard library functions I had just written, and the documentation I'd just updated.

In the few places where the functions were used it was easy to replace e.g. Add with function (x, y) { x + y }

I ended up keeping a definition for greater-than since it was used in several places to sort in reverse order. (since list.Sort! takes an optional comparison function defaulting to less-than)

So I started with adding a function, and I ended with deleting a bunch instead.

The moral of the story is to add features when you need them. (YAGNI) That certainly applied to adding the compare function, but it also applies to creating all the other ones in the first place.

And when I ported them all to jSuneido it never occurred to me to question whether they should just be removed.

Another factor is "sunk costs". After spending the time to implement the functions, first in cSuneido and jSuneido and then in the standard library, it felt bad to just delete that work. But you shouldn't let that affect decision making.

Part of my thinking was along the lines of, if we have A, B, and D, then we should have C. But that only makes sense if you had a real need for A, B, and D.

Finally, I think the reason I started this whole process was that it was psychologically attractive to do something that I could get a quick and easy sense of accomplishment from. (As opposed to large projects where it's hard to even see progress.) There's nothing wrong with that, but the choice of what to do should still be rational. i.e. pick an easy task from the ones that make sense to do.

Apart from feeling stupid both for the past and present wasted effort, in the end I'm happy to remove some code and move (if only slightly) towards simpler.

Programming in Typescript

I've been back working on suneido.js recently, which means programming in JavaScript or, my preference, TypeScript. It's been awhile so I had to get back up to speed on what tools to use. In case it's useful to anyone, here's what I've settled on (for now).

• install Node.js (current 6.3 rather than stable since I'm not in production)
• use npm to install TypeScript (currently at 1.8)

I started used Visual Studio Code, which I quite like, but a few things bugged me so I gave Atom a try and found I liked the editing experience better, especially after installing a few extra packages. One of the things I like about Atom is that it compiles Typescript automatically when you save a file, whereas (AFAIK) with Code you need to run tsc -w in a separate terminal window.

Atom packages:

• atom-typescript
• atom-beautify
• highlight-line
• highlight-selected
• simple-drag-drop-text
• Sublime-Style-Column-Select

The last three should be part of Atom core if you ask me, but as long as they're available that's fine.

Atom setting:

• tree-view - hide VCS ignored files
• autosave enabled
• show invisibles
• tab size: 4 (personal preference)

One thing I am still using Code for is debugging. Node's stack traces show the JavaScript line numbers which is a pain to relate back to the TypeScript code. I found some ways around this, but none of them looked simple. Code's debugger works well.

Previously I had also used WebStorm from JetBrains which has support for TypeScript and debugging. I may use it again, although I like the ease of using simple tools like Atom and Code.

Interestingly, Visual Studio Code is built on top of Electron, which was originally developed for Atom. And recently, Monaco, the editor component of Visual Studio Code has been open sourced.

I'm targeting ES6/2015 partly as future-proofing, and partly just because it's a better language. Most of ES6 is pretty well supported in Node and browsers, and there are tools like Babel to run it under ES5. Exploring ES6 is a useful resource. Strangely, none of the browsers support ES6 modules natively. You can still use them in the Typescript code but they need to be compiled to AMD for the browser, and CommonJS for Node (e.g. testing).

Although JavaScript has some quirks that I don't care for, overall I quite like it. And with Typescript for type checking, I don't mind programming in it at all. One thing I like about Typescript, as opposed to things like CoffeeScript is that it doesn't obscure the JavaScript.

Thankfully getting going again has gone smoother this time than my last bout of Yak Shaving

You can follow progress on suneido.js on GitHub. There's still a long way to go!

Yak Shaving

"Yak shaving" is a programmer's slang term for the distance between a task's start and completion and the tangential tasks between you and the solution. If you ever wanted to mail a letter, but couldn't find a stamp, and had to drive your car to get the stamp, but also needed to refill the tank with gas, which then let you get to the post office where you could buy a stamp to mail your letter—then you've done some yak shaving. - Zed Shaw

Yak shaving is one of my least favorite and most frustrating parts of software development. I have vague memories of enjoying the chase when I was younger, but my tolerance for it seems to decrease the older I get. Nowadays it just pisses me off.

I haven't worked on suneido.js for a while but I got a pull request from another programmer that's been helping me with it. I merged the pull request without any problems. Then I went to try out the changes.

I get a cryptic JavaScript error. Eventually I figure out this is a module problem. I see that my tcconfig.js is set to commonjs, which is what is needed to run the tests in node.js, but to run in the browser I need amd. I change it. I love JavaScript modules.

Now I need to recompile my TypeScript. I happen to be using Visual Studio Code, which follows the latest trend in editors to not have any menu options that are discoverable. Instead you need to type commands. I eventually find the keyboard shortcut Ctrl+Shift+B. (After some confusion because I was reading the web site on OS X and it auto-magically only shows you the keyboard shortcuts for the platform you're on.)

Nothing happens. I try tsc (the TypeScript compiler) from the command line but it's not found. Maybe it's just not on my path? I figure out how to see the output window within Code. It's not finding tsc either. (Although it hadn't shown any sign of error until I opened the output window.)

Do I have tsc installed? I'm not sure. Doesn't Code come with TypeScript support? Yes, but does that "support" include the compiler? I find that exact question on StackExchange but it's old and there seems to be some debate over the answer.

I try reinstalling Code. It adds to my path but still doesn't find tsc. (The ever growing path is another pet peeve. And yes, the older I get the longer my list of pet peeves is.)

What the heck was I using last time I worked on this? I'm pretty sure I never deliberately uninstalled TypeScript. I search my computer but all I find are some old versions of TypeScript from Visual Studio (not Code).

I get the bright idea to check my work computer (I'm on my home machine.) I don't find anything there either. Then I remember it's a brand new computer that I set up from scratch. So much for that idea.

I give up and decide to install TypeScript. There are two methods given, via npm (Node.js) or Visual Studio. I'm pretty sure I used Node.js before so that seems to be the way to go.

I try to run npm but it's not found either. Did I lose Node.js as well as TypeScript? No, I find it on disk. I see there's a nodevars.bat file that sets the path. I try that and now I have npm. I bask in the warm glow of accomplishment for a few seconds until I remind myself that running npm was not what I was trying to accomplish.

I run npm install -g typescript. And now I have tsc. But my path didn't change. Maybe I had tsc all along and just needed node on my path? Hard to tell now.

Then I find my tsconfig.js is wrong - it lists .ts files that don't exist because they are just .js files. Again, how did that work last time around? I fix that and it seems to compile.

And finally, miraculously (but several hours later) my suneido.js playground works!

Wait a minute, the pull request was improvements to the CodeMirror Suneido mode, it had nothing to do with the playground. No problem, I'll just start over ...

New PC Software

Once I had my new PC running, the next step was to install software. I prefer to set up new computers from scratch rather than use any kind of migration tool because I don't want to bring over a lot of old junk. It takes a little more time but I think it's worth it. And it's always interesting to review what software I'm actually using day to day, and how that changes over time.

My first install is usually Chrome. Once I sign in to my Google account and sync my browser settings I have all my bookmarks and my Gmail ready to go. I use LastPass for my passwords. I also install Firefox but it's not my day to day browser. I used to also install Safari for Windows but Apple is no longer keeping it updated.

Next is Dropbox which brings down a bunch of my commonly used files. Then Evernote which gives me access to my notes.

After that it's mostly development tools - Visual Studio Community (free) C++, MinGW C++, Java JDK, and Eclipse. The last few Eclipse installs I've been importing my addons from the previous install. But for some reason I couldn't get that to work this time because I couldn't find the right directory to import from. The directories are different since I've been using the Oomph installer. I'm sure I could figure it out, but I only use three addons so it was easier just to install them from the Eclipse Marketplace. (The three are Infinitest, EclEmma coverage, and Bytecode Outline.)

I use GitHub Desktop although both Visual Studio and Eclipse provide their own Git support. (and there's also the command line, of course.)

Although I'm not actively programming in Go these days I like to have it and LiteIDE installed.

For editors I use Scite, because it's the same editing component as we use in Suneido, and Visual Studio Code for JavaScript and Typescript. (Visual Studio Code is not related to Visual Studio. It's a cross platform application written in Typescript and packed with Electron.)

This is all on Windows, but other than the C++ tools I have pretty much exactly the same set of software on my Mac.

New PC

In many ways PC performance has leveled off. There are minor improvements in CPU and memory speed but nothing big. But there has been a significant improvement in performance with the Skylake platform supporting SSD connected via PCI Express.

Based on a little research, here were my goals:

• fast SkyLake CPU
• 32gb DDR4 ram
• 512 gb M2 SSD
• small case (no external cards or drives)
• 4K IPS monitor

And here's what I ended up getting:

• Asus Z170I Pro Gaming mini ITX
• Intel Core I7-6700K 4.00 GHz  
• Corsair Vengeance LPX 32GB (2x16GB) DDR4 DRAM 2666MHz (PC4-21300)
• Samsung 950 PRO - Series 512GB PCIe NVMe - M.2 Internal SSD
• Fractal Design 202 case
• ASUS PB279Q 27" 4K/ UHD 3840x2160 IPS

I don't do any gaming, but that was the only mini ITX motherboard I could find that fit my requirements.

The case was the smallest I could find that would fit this stuff. The frustrating part is that it could be half the size. The empty half of the case is for cards or external drives, but I didn't want either of those. It's a little hard to tell from the picture, but the motherboard is only 7" square, not much bigger than the fan. The power supply is almost as big as the motherboard.

And here's a comparison of my new SSD (top) to the old one (bottom). Higher numbers are better.

The big advantage of SSD over hard disks is the seek time for random IO. But the biggest gains here were for sequential IO. Still, some respectable improvements across the board. Of course, how that translates into actual overall performance is another question.

I try not to buy new machines too often. At least I know my old one, which is still working fine, will be put to good use by someone else in the office.

I'm not a hardware expert, here's where I got some advice:
Building a PC, Part VIII
Our Brave New World of 4K Displays

Mac Remote Desktop Resolution

At home I have a Retina iMac with a 27" 5120 x 2880 display. At work I have a Windows machine with a 27" 2560 x 1440 display set at 125% DPI. I use Microsoft Remote Desktop (available through the Apple app store) to access my work machine from home.

I'm not sure how it was working before, but after I upgraded my work machine to Windows 10, everything got smaller. It comes up looking like 100% (although that's actually 200% on the Mac).

Annoyingly, when you are connected through RDP you aren't allowed to change the DPI.

I looked at the settings on my RDP connection but the highest resolution I could choose (other than "native") was 1920 x 1080 which was close, but a little too big.

Poking around, I found that in the Preferences of Microsoft Remote Desktop you can add your own resolutions (by clicking on the '+' at the bottom left). I added 2048 x 1152 (2560 x 1440 / 1.25)

Then changed the settings on the connection to use that and it's now back to my usual size.

The screen quality with RDP doing the scaling does not seem as good as when Windows is doing the scaling, but at least the size is the same.

I'm guessing from what I saw with searching the web that there might be a way to adjust this on the Terminal Server on my Windows machine, but I didn't find any simple instructions.

If anyone knows a better way to handle this, let me know.

Native UX or Conway's Law?

organizations which design systems ... are constrained to produce designs
which are copies of the communication structures of these organizations
— M. Conway

A pet peeve of mine is applications that are available on different platforms, but are substantially different on each platform. I realize that I'm probably unusual in working roughly equally on Windows and Mac. But even if most people work on a single platform, why make the application unnecessarily different? Isn't that just more work for support and documentation.

I recently listened to a podcast where an employee mentioned that her focus was on making sure their Windows and Mac versions weren't "just" identical, but matched the native user experience. That sounds like an admirable goal. The problem is, their versions have differences that are nothing to do with native UX.

Obviously standards vary between platforms. Do you call it Settings or Preferences? And it makes sense to match the native conventions. But in general those are minor things.

But changing the layout of the screen? Or the way things are displayed? Or the text on buttons? There really aren't any "standards" for those things. And therefore "native" doesn't justify making them different.

I suspect that "matching the native UX" is often a justification for Conway's law. Different teams are developing the different versions and they do things different ways. Coordinating to make them consistent would take more effort so it doesn't happen, or not completely.

My company contracted out the development of our mobile app. The company we contract to believes in "native" (i.e. not portable) apps. They also have a process where different teams develop different versions, so they can be platform experts. The end result - our app is unnecessarily different on different platforms. And any change takes twice as much work. Not to mention having to fix bugs in multiple implementations.

It's sad that after all these years we still don't have a good story for developing portable applications.

Except that's not quite true. We have the web and browsers. And tools like Cordova and Electron that leverage that. And contrary to belief, users don't seem to have a lot of problems with using web apps that don't have native UX.

Suneido RackServer Middleware

A few years ago I wrote an HTTP server for Suneido based on Ruby Rack and Python WSGI designs. We're using it for all our new web servers although we're still using the old HttpServer in some places.

One of the advantages of Rack style servers is that they make it easy to compose your web server using independently written middleware. Unfortunately, when I wrote RackServer I didn't actually write any middleware and my programmers, being unfamiliar with Rack, wrote their RackServers without any middleware. As the saying goes, small pieces, loosely joined. The Rack code we had was in small pieces, but it wasn't loosely joined. The pieces called each other directly, making it difficult, if not impossible, to reuse separately.

Another advantage of the Rack design is that both the apps and the middleware are easy to test since they just take an environment and return a result. They don't directly do any socket connections.

To review, a Rack app is a function (or in Suneido, anything callable e.g. a class with a CallClass or instance with a Call) that takes an environment object and returns an object with three members - the response code, extra response headers, and the body of the response. (If the result is just a string, the response code is assumed to be 200.)

So a simple RackServer would be:

app = function () { return "hello world" }
RackServer(app: app)

Middleware is similar to an app, but it also needs to know the app (or middleware) that it is "wrapping". Rack passes that as an argument to the constructor.

The simplest "do nothing" middleware is:

mw = class
    {
    New(.app)
        { }
    Call(env)
        {
        return (.app)(env: env)
        }
    }

(Where .app is a shortcut for accepting an app argument and then storing it in an instance variable as with .app = app)

Notice we name the env argument (as RackServer does) so simple apps (like the above hello world) are not required to accept it.

and you could use it with the previous app via:

wrapped_app = mw(app)
RackServer(app: wrapped_app)

A middleware component can do things before or after the app that it wraps. It can alter the request environment before passing it to the app, or it can alter the response returned by the app. It also has the option of handling the request itself and not calling the app that it wraps at all.

Uses of middleware include debugging, logging, caching, authentication, setting default content length or content type, handle HEAD using GET, and more. A default Rails app uses about 20 Rack middleware components.

It's common to use a number of middleware components chained together. Wrapping them manually is a little awkward:

wrapped_app = mw1(mw2(mw3(mw4(app))))

so I added a "with" argument to RackServer so you could pass a list of middleware:

RackServer(:app, with: [mw1, mw2, mw3, mw4])

(where :app is shorthand for app: app)

Inside Rackserver it simply does:

app = Compose(@with)(app)

using the compose function I wrote recently. In this case what we are composing are the constructor functions. (In Suneido calling a class defaults to constructing an instance.)

A router is another kind of middleware. Most middleware is "chained" one to the next whereas routing looks at the request and calls one of a number of apps. You could do this by chaining but it's cleaner and more efficient to do it in one step.

So far I've only written a few simple middleware components:

• RackLog - logs the request along with the duration of the processing
• RackDebug - Print's any exceptions along with a stack trace
• RackContentType - Supplies a default content type based on path extensions using MimeTypes
• RackRouter - a simple router that matches the path agains regular expressions to determine which app to call

But with those examples it should be easy for people to see how to compose their web server from middleware.

Suneido Functional Compose

I started on one project, found it required another, which then led to code like:

a(b(c(d(...)))

which seemed ugly. I knew that functional programming has a higher order "compose" to handle this more cleanly. (Higher order because it takes functions as arguments and returns another function.)

Suneido isn't specifically a functional language, but it has a bunch of functional methods and functions. I hadn't written Compose yet but it turned out to be quite easy.

function (@fns)
    {
    return {|@args|
        result = (fns[0])(@args)
        for fn in fns[1..]
            result = fn(result)
        result
        }
    }

If you're not familiar with Suneido code, the '@' is used to accept multiple arguments and to expand them again. The {|...| ... } notation is Suneido's way of writing a "block", i.e. a lambda or closure. (Suneido's use of the term "block" and the syntax come from Smalltalk, one of its roots way back when, pre 2000.)  Because, in Suneido, blocks are used for control flow, "return" returns from the containing function, so to return a result from a block we take advantage of how Suneido by default returns the value of the last statement.

Although in some languages the arguments to compose are listed "outside-in", I chose to have them "inside-out", so Compose(a,b,c) returns a function that does c(b(a(...))) in the same way you would write a *nix pipeline a | b | c i.e. in the order they will be applied.

I documented it, like a good programmer, except when I went to add the "see also" section I found a lot of the other functional stuff had never been documented. So I spent a bunch more time documenting some of the that. In the process I almost got sucked into improving the code, but I resisted pushing yet another task onto the stack!

Taking Notes

When I'm working on something complex, either a bug or new code, I keep notes. Nowadays I could blame that on an aging memory, but I've been doing it too long for that to be the reason. Originally I used paper notebooks, writing in pencil so I could erase and make corrections.

I take notes in a special style that has stayed quite consistent over the years. To make it easier to scan the notes and follow the "logical" structure I prefix sentences with capitalized "keywords". Some of these are familiar ones like "BUG" and "TODO" that are commonly used in source code comments. I also use "Q" to start a question (since the trailing question mark is harder to spot) and "A" for answers. Some less standard prefixes I use are "COULD" and "MAYBE" which I use to prefix proposals or hypothesis. ("MAYBE" being stronger than "COULD") Here's a fabricated example to give you a better idea:

BUG: calculating a checksum occasionally gives an incorrect result

Q why does it work some of the time ?

A probably a concurrency issue

MAYBE add locking

BUT that could lead to deadlocks

SO make sure no other locks are held or taken at the same time

TODO: review code to look for similar problems

These notes are generally just for my own use, but I've still tried to keep the notation fairly self explanatory so if someone else did need to read them it wouldn't be too hard.

You might imagine that the main benefit of keeping notes would be that you can refer back to them. But I find I rarely do that. Occasionally if I run into problems afterwards I'll go back to my notes to refresh my memory, especially if I didn't completely resolve the issue. But that's the exception.

I find the biggest benefit is that it helps me think through a problem, to be a little more logical, to expose and examine my thinking, to see what I've considered and why I think it is or isn't the right approach. It helps prevent me from going in circles and not making any progress. Often it's just as helpful to rule out certain alternatives as it is to find successful ones.

I'm a believer in clear code over a lot of comments. But what clear code doesn't tell you is why you chose that particular approach. It's good to add comments for that type of thing, but comments will seldom tell you the logical path you took to reach that point, the dead ends you explored, the approaches that turned out too complex or had performance issues. That's where the notes can come in.

Unless I know something is going to be complex I don't start out making notes. Instead, I'll work on it for a little bit and if it turns out to be tricky, then I'll make a conscious decision to start taking notes. I might make a few retroactive notes of my thinking so far but usually I'll just start from that point.

For a brief time I used Google Docs for these notes. That worked quite well for bigger projects, but it was too heavyweight for jotting down ideas or brief notes. And in the past they didn't have a very good mobile story.

For the last few years I've been using Evernote. I like how it syncs between all my devices and can be used off-line. The Mac and Windows versions are different enough to be annoying, but I don't imagine there are a lot of people that use both. From a software development perspective it seems crazy that they develop and maintain what appears to be two completely separate programs, but cross platform apps are a struggle, even today.

I briefly tried using Penultimate and Note Taker (from Dan Bricklin of VisiCalc fame) for taking handwritten notes in meetings. Scribbling something was easier and less distracting from the meeting than typing. But then I ended up more or less banning mobile devices from meetings because people find it impossible not to be distracted by all the notifications they get from texts, emails, FaceBook etc. (That's not as much of a problem for me because I have all notifications turned off. And being an antisocial programmer I'm not as addicted to that steady flow of chatter.)

Although I'm not big on diagrams, I did take advantage of pencil and paper to occasionally sketch out things like data data structures. This is a little harder to do when typing into some kind of text editor. I use Google Drawings if I want a more polished result. To quickly sketch diagrams on the iPad one app I like is Jot! Unlike a plain drawing app it lets you move things around and add editable text. However, it doesn't fix up your rectangles the way Paper does. But Paper doesn't let you add typed text to your drawings.

A few days ago someone asked me if I had used any apps that turned handwriting into editable text. I hadn't, unless you count way-back-when with a Palm and its quirky alphabet designed for easy recognition. (Which I quite liked and used a fair bit.)

Coincidentally, I recently got an iPad Pro and its companion Pencil. (Sadly, I'm not immune to the lure of the new new thing, despite feelings of guilt about it.) I hadn't really found a lot of use for the Pencil, although it did work well and had a much better feel than the previous fat soft iPad styluses that I'd tried.

So today I did some searching and found 7notes. Having not tried any handwriting recognition for years I was amazed at how well it worked. I started out printing but soon found it handled even my sloppy cursive. Of course, if I got too sloppy it had trouble, but even I have trouble reading my own handwriting sometimes. It has good support for auto correction and offering alternate possible words. Like the keyboard auto-correct, you do have to keep an eye on how it's interpreting what you write or you can get some funny results.

Although 7notes does have integration with Evernote it requires manually transferring notes back and forth. I started thinking it would be nice if I could use the handwriting recognition in Evernote itself (or other apps). I was happy to find that the same company has an iOS "keyboard" using the same technology - mazec. It doesn't have all the features of 7notes but seems to have the same basic recognition engine.

A Bug Hunter's Life

I recently spent two weeks tracking down a single bug. Needless to say, it was a frustrating process. Like most such things, what made it difficult was not a single problem, it was a whole cascade of errors, including many during the debugging process. I like to think I'm fairly organized and methodical, especially when working on a tough bug. I take notes on what I'm doing, what my assumptions are, what I've tried, what the results were. And yet I still managed to make multiple major mistakes and go a long way down the wrong paths based on false assumptions I should have questioned much sooner.

It started with a customer getting a corrupted database. Most of the time this is from hardware issues or crashing. But this one appeared to be caused by a bug in the code.

It wasn't easy to look into because all I had was the corrupt database, with no clues as to how it might have gotten that way. Thankfully, jSuneido's append-only database means that the database itself forms a kind of log. Based on that I could see that two transactions had committed conflicting updates - something that the whole transaction system is designed to prevent.

It seemed a pretty safe bet that this was a concurrency related issue - my least favorite kind of bug :-( I wasn't surprised that I couldn't make it happen. With roughly 1000 systems running 24x7 this was the only case I was aware of. So my sole sources of information were that database and the code itself.

In order to detect conflicts at commit time (jSuneido uses optimistic concurrency control) you need to know the outstanding overlapping update transactions (i.e. ones that had started after this one and had not completed yet). Perhaps the bug was in determining this set of transactions? I wrote a completely new implementation of this code and used it to cross-check.

We deployed this version to our customers and pretty quickly started to get lots of errors. But only a couple were from the new cross-check. The rest were from code I hadn't touched. And yet they quite clearly started when the new version was deployed. Strange.

The errors indicated that sometimes when it went to complete a transaction it would be missing from the list of outstanding ones. Again, this shouldn't be possible. That would explain why the overlapping set would be wrong, so it made a certain amount of sense. I spent days studying the code and trying (unsuccessfully) to figure out how it was happening.

This turned out to be completely off track - it was a just problem with my new cross checking code.

The stack traces showed a transaction commit that turned into an abort. That happens if an update transaction doesn't end up doing any updates. So I wrote a bunch of tests with those kinds of transactions in the mix, but they all worked fine. If nothing else, the code was getting reviewed and tested pretty heavily.

Finally, after wasting days on this, I realized there was another way for a commit to turn into an abort. The relevant part of the code looked like:

try {
     // commit stuff
} finally {
     abortIfNotComplete();
}

What was happening was that an exception during the commit stuff would get overridden (and lost) if there was an error in abortIfNotComplete(). So I wasn't seeing the original error at all. And the reason that the transaction was missing from the list was that it had got far enough into the commit to remove it.

I'm not sure why I wrote it using finally. In hindsight it was obviously a bad idea. I rewrote it like:

try {
     // commit stuff
} catch (Throwable e) {
     try {
          abortIfNotComplete();
     } catch (Throwable e2) {
          e.addSuppressed(e2);
          throw e;
     }

addSuppressed was added in Java 7. It is used, for example, in try-with-resources for exceptions when closing after another exception.

Part of the reason I didn't realize which path was being taken in the code was that I didn't have line numbers in the stack traces.  That was because in the Ant build task "debuglevel=vars" had been added. The intent had been to increase the amount of debugging information. But because of the way Ant and the Java -g option work, this had actually meant only variable information. I changed it to "debug=true" which becomes "-g" which means all debug information (file, line, and vars). Once this change percolated through and I started to get stack traces with line numbers, then I could see which path was being taken in the code.

One big step forward was when I was finally able to recreate the bug. Running random updates (of certain kinds) in a bunch of threads would encounter the bug within a few seconds. It wasn't the ideal recreation because the bug would occur buried within a lot of other activity, so it still wasn't possible to determine exactly what was happening.

I couldn't be 100% sure that what I was able to recreate was the same bug, but the end result was more or less identical - two transactions deleting the same record. But confusingly, the errors that I saw were about duplicate records being output, leading me to spend a bunch of time looking at the output code.

A huge mistake that I made repeatedly, without realizing it, was that I was building one variation of the jSuneido jar, and then testing with a different out of date one. So I wasn't actually testing with the changes I was making. This was because, for faster turnaround, I wasn't running a full build, I was just building the main jar that we deploy. But I was testing using the jar with Win32 support so I could use the IDE. This explained the totally baffling behavior where the debugging output that I was 100% certain should be triggered didn't appear at all. Which confused me and made me think I was totally wrong in my ideas about what was happening.

This mistake cost me about 4 days. I actually had the bug fixed, but it kept happening because I was testing with the old jar. Of course, since the bug still happened I assumed that I had not located it yet and I went looking for it in other places where, of course, I didn't find it.

Even worse, I thought my method of recreating the bug was quite "fragile". For instance, it would not happen running in the Eclipse debugger. Coming from C and C++, I didn't find that too surprising. And although Java should be more predicable, there would still be timing and threading differences which could affect concurrency. But I was totally wrong - the reason I couldn't recreate it within Eclipse was that I was running the latest code, which had the fix. Arghhh!

The actual bug and fix are obvious in hindsight. Throughout the long process I suspected that would be the likely result. But that didn't make it any easier, in fact it made it even more frustrating.

jSuneido's database uses optimistic concurrency. That means concurrent transactions proceed independently without any locking. Then when it comes time to commit, they verify that nothing has happened that would conflict. For example, they might find that another transaction has output a duplicate key. In which case the transaction will abort due to the conflict. Or it might find that another transaction has deleted a key that we also deleted. And that's where the bug was. I had made the assumption (back when I wrote this code) that it was ok if two concurrent transactions deleted the same key. After all, the end result is the same. And it's still serializable (i.e. the result is as if the transactions had happened one after another instead of concurrently).

The problem is not the deletes by themselves. If that was all the transactions were doing, my logic was correct. But transactions aren't that simple. They perform other actions based on whether the delete succeeded. For example, deleting an old version of a record and outputting a new version.  If you allow two transactions to think they deleted the old version, then they will, incorrectly, both output a new version. (leading to the duplicate outputs I was seeing)

All I had to do to fix the bug was to make duplicate deletes a conflict, like I already did with duplicate outputs. Two weeks for one line of code. How's that for productivity.

One interesting part of this whole adventure was that although it was a concurrency bug, it wasn't the kind of subtle interaction problem that I really fear. It was, in hindsight, a fairly clear, obvious issue.

Problem with Eclipse Mars on Windows after Java Update

I installed the new Java update on my Windows work machine, and removed the old version.

When I tried to run Eclipse (Mars) it said it couldn't find a JRE at the path of the old version of Java.

That made sense because I'd removed it. But why was it specifically looking for that path?

When I'd updated Java in the past Eclipse had found the new version fine.

I found that in the eclipse.ini file it had a -vm option set to the old version of Java. I removed this and now Eclipse starts fine.

I wonder if this was a result of installing the recently released Eclipse Mars with the Oomph installer?

Note: The Oomph installer puts Eclipse in your user folder (e.g. c:/Users/andrew in my case) rather than in Program Files. This may be a workaround to not require admin privileges to install.

Visual Studio Tip

I've been trying out Visual Studio 2015 Community RC and every time I did a build I'd get:

All packages are already installed and there is nothing to restore.
NuGet package restore finished.

It didn't seem to hurt anything, but I prefer to keep the build output clean so eventually I got tired of it and figured out you can get rid of it using Tools > Options

Of course, that's assuming you don't need to check for missing packages during builds.