Lexer in TypeScript

One of the standard things I do when I'm investigating a new language is to implement Suneido's lexical scanner. Not because this is necessarily the best test of a language. It's more that I'm usually evaluating the language in terms of how it would work to implement Suneido. I realize that's a very biased view :-)

I wasn't planning to do this with TypeScript / JavaScript because suneido.js isn't intended to be an "implementation" of Suneido. It's a transpiler that runs on jSuneido (using the jSuneido lexer and parser) that translates Suneido code to JavaScript. The only manually written TypeScript / JavaScript code is the runtime support library. (And that's partly a bootstrapping process. Once the transpiler is functional much of the support library could be written in Suneido code.)

But as I implement the runtime support library, obviously I need tests. And I'm getting tired of writing the same tests for each version of Suneido. (Or worse, different tests for each version.) That's why I started developing "portable tests" some time ago. So I decided I should implement the portable test framework in TypeScript. At which point I remembered that this required a Suneido lexical scanner.

So I decided to write yet another version. I took about a day, which is about what I expected. And some of that time is still learning TypeScript and JavaScript. I based it mostly on the C# version. The Go version is more recent but Go has more language differences.

Overall, I was pretty happy with how it went. I find the repetition of "this." a little ugly. In Suneido you can abbreviate this.foo as just .foo, which is a lot more compact and less "noise" in the code, while still making it clear it's an instance variable.

I find JavaScript's equality testing somewhat baroque, the recommendation is to use === most of the time, but that won't always work for strings (because of primitive strings and object strings). And then there's the quirks of -0 versus +0, and NaN.

ES6 arrow functions and the functional methods like map are nice. And spread / rest (...) and for-of are good. I only specified types on function parameters and returns, but this worked well and gave me very little grief.

I was pleasantly surprised to find that TypeScript enum's automatically create a reverse mapping from name to number. Although the gotcha is that if you make them const you don't get this because they are inlined. Using a simple numeric enum doesn't let me attach extra information for parsing, but I'm not planning to write a parser so that's not a problem. The other issue is debugging, where all you see is a number.

Removing const from my token enum exposed another issue. I was relying on Atom's auto-compile but it only compiles the file you are editing, not the other files that may depend on it. So I went back to running tsc -w in a separate terminal window. (Open question - does anyone know why tsc lists some of my files as relative paths and some as absolute? It's consistent about which files, except that gradually more are switching to absolute. It's not causing any problems, I just can't figure out why.)

Although it can mean shorter code, I'm not a fan of "truthy" and "falsey" values. It was a source of problems 30 years ago in C. Now I got caught by it with a function that returned a number or undefined. I expected undefined to be "falsey" but I forgot that zero is also "falsey". With Suneido I took a stricter approach that things like if and while only accept true or false and throw an exception for any other type of value.

Now that I have a lexical scanner I can implement the portable test framework, which also shouldn't take me too long.

Versions, in order of age. cSuneido and jSuneido are production code and more full featured than the other more experimental versions.

• TypeScript tokens.ts, lexer.ts, lexer_test.ts
• Go tokens.go, lexer.go, lexer_test.go
• C# Tokens.cs, Lexer.cs (tests at the end)
• D token.d, lexer.d (tests at the end)
• Java (jSuneido) Token.java, Lexer.java, LexerTest.java
• C++ (cSuneido) scanner.h, scanner.cpp (tests at the end)

Going in Circles

Suneido has a bunch of built-in functions equivalent to its operators. e.g. Add for '+'  These are useful, for example, in functional style programming e.g. list.Reduce(Add)

I noticed that we didn't have a compare function i.e. that would return -1 for less than, 0 for equal, or +1 for greater than. This can be useful in some cases (e.g. in a switch) and avoids needing two comparisons. (Note: I didn't have a need for it, just noticed we didn't have it.)

A few days later I decided to implement it. It was simplest in jSuneido since there was already an internal compare function that I just needed to expose. Whereas cSuneido has the C++ / STL style of using operator < and operator == to implement all the other comparisons. So it needed two comparisons.

Then I started to question why these functions needed to be built-in. Why not just write them in Suneido code in the standard library. The difference in performance would be minor. So I ripped them out of cSuneido and jSuneido and defined them in the standard library. I also updated the documentation.

Then I started to wonder how often these built-in functions are actually used. As you can probably guess, almost never. On top of that I found several places where programmers got confused and thought they needed to use these functions instead of just the operators. (I fixed these.)

So I ended up ripping out the standard library functions I had just written, and the documentation I'd just updated.

In the few places where the functions were used it was easy to replace e.g. Add with function (x, y) { x + y }

I ended up keeping a definition for greater-than since it was used in several places to sort in reverse order. (since list.Sort! takes an optional comparison function defaulting to less-than)

So I started with adding a function, and I ended with deleting a bunch instead.

The moral of the story is to add features when you need them. (YAGNI) That certainly applied to adding the compare function, but it also applies to creating all the other ones in the first place.

And when I ported them all to jSuneido it never occurred to me to question whether they should just be removed.

Another factor is "sunk costs". After spending the time to implement the functions, first in cSuneido and jSuneido and then in the standard library, it felt bad to just delete that work. But you shouldn't let that affect decision making.

Part of my thinking was along the lines of, if we have A, B, and D, then we should have C. But that only makes sense if you had a real need for A, B, and D.

Finally, I think the reason I started this whole process was that it was psychologically attractive to do something that I could get a quick and easy sense of accomplishment from. (As opposed to large projects where it's hard to even see progress.) There's nothing wrong with that, but the choice of what to do should still be rational. i.e. pick an easy task from the ones that make sense to do.

Apart from feeling stupid both for the past and present wasted effort, in the end I'm happy to remove some code and move (if only slightly) towards simpler.

Programming in Typescript

I've been back working on suneido.js recently, which means programming in JavaScript or, my preference, TypeScript. It's been awhile so I had to get back up to speed on what tools to use. In case it's useful to anyone, here's what I've settled on (for now).

• install Node.js (current 6.3 rather than stable since I'm not in production)
• use npm to install TypeScript (currently at 1.8)

I started used Visual Studio Code, which I quite like, but a few things bugged me so I gave Atom a try and found I liked the editing experience better, especially after installing a few extra packages. One of the things I like about Atom is that it compiles Typescript automatically when you save a file, whereas (AFAIK) with Code you need to run tsc -w in a separate terminal window.

Atom packages:

• atom-typescript
• atom-beautify
• highlight-line
• highlight-selected
• simple-drag-drop-text
• Sublime-Style-Column-Select

The last three should be part of Atom core if you ask me, but as long as they're available that's fine.

Atom setting:

• tree-view - hide VCS ignored files
• autosave enabled
• show invisibles
• tab size: 4 (personal preference)

One thing I am still using Code for is debugging. Node's stack traces show the JavaScript line numbers which is a pain to relate back to the TypeScript code. I found some ways around this, but none of them looked simple. Code's debugger works well.

Previously I had also used WebStorm from JetBrains which has support for TypeScript and debugging. I may use it again, although I like the ease of using simple tools like Atom and Code.

Interestingly, Visual Studio Code is built on top of Electron, which was originally developed for Atom. And recently, Monaco, the editor component of Visual Studio Code has been open sourced.

I'm targeting ES6/2015 partly as future-proofing, and partly just because it's a better language. Most of ES6 is pretty well supported in Node and browsers, and there are tools like Babel to run it under ES5. Exploring ES6 is a useful resource. Strangely, none of the browsers support ES6 modules natively. You can still use them in the Typescript code but they need to be compiled to AMD for the browser, and CommonJS for Node (e.g. testing).

Although JavaScript has some quirks that I don't care for, overall I quite like it. And with Typescript for type checking, I don't mind programming in it at all. One thing I like about Typescript, as opposed to things like CoffeeScript is that it doesn't obscure the JavaScript.

Thankfully getting going again has gone smoother this time than my last bout of Yak Shaving

You can follow progress on suneido.js on GitHub. There's still a long way to go!