Java 20

I don't program much in Java these days, just minimal maintenance on jSuneido. But I still have an interest in what's happening with the language.

Once upon a time, Java had a problem - no new releases for a long time. They solved that by switching to a fixed release every six months. For a while we got all kinds of new features. It was great. But lately it's been a new problem, the releases have been on time, but they haven't "released" much. Here's the complete list of features for the latest release:

Features

429:	Scoped Values (Incubator)
432:	Record Patterns (Second Preview)
433:	Pattern Matching for switch (Fourth Preview)
434:	Foreign Function & Memory API (Second Preview)
436:	Virtual Threads (Second Preview)
437:	Structured Concurrency (Second Incubator)
438:	Vector API (Fifth Incubator)

Every feature is either incubator or preview. I understand the need for incubator and preview. But "Fourth Preview" and "Fifth Incubator"? Yes, you want to get it "right", but if you wait for perfect in the eyes of committees, you could be waiting a long time.

I wonder if this is partly a matter of Java's "enterprise" association. Let's face it, the hip kids aren't using Java. They don't care if e.g. Java Virtual Threads are ever released. On the other hand the enterprise customers don't really want new features. They're still running Java 8. But they do want to be reassured that Java is alive and well. Previews and incubators serve that purpose just fine.

Maybe I'm just impatient. I'm excited about some of the new features (and others, like Valhalla that have been "coming soon" for many years but haven't even made it to incubation). But I don't have that many decades left.

Blogging, Suneido, and Programming Languages

Another blogger I follow has announced they are stopping blogging. It's sad (to me) how the world has shifted away from blogging. Of course, there are still active blogs, but far fewer than there were. I thought maybe the pandemic would give people time to do more blogging but it didn't seem to work that way. Of course, it always surprised me that smart productive busy people would take the time to write good blog posts.

I'm no exception. I post less and less often, other than photographs on my non-work blog. But even that is a shift from my past longer text posts. Readers have dwindled too. (Not that I ever had a big audience.)

But looking back through some of my blog posts, I'm glad I wrote them even if hardly anyone read them. It's a bit like a sporadic diary, but more thoughtfully written than a personal diary would be. I started blogging in 2005. It would have been interesting to go back farther but even in 2005 the tech world was a different place than today.

Thinking about this led me back here. I was writing a document for my staff, and some of it seemed of slightly more general interest so I thought I'd write about it.

Here's roughly when the versions of Suneido went into "production" (used by customers)

• 2000 cSuneido (C++)
• 2010 jSuneido (Java)
• 2020 gSuneido client (Go)
• 2022 gSuneido server
• 2022 suneido.js (browser client)

Two data points doesn't mean much, but it's interesting that it was roughly 10 years between the major versions. The concurrent release of suneido.js is because another programmer did the development. (I started it, but developing both gSuneido and suneido.js at the same time was too much.)

All three implementation languages (C++, Java, Go) were in their early stages when I started using them. I was lucky that all went on to become mainstream. Some of the other languages I considered, such as C#/.Net and Kotlin have also been successful. Scala and D not as much, although they're still around.

C++ is a fascinating language. But it's too complex and it's unsafe. Life is too short. (gSuneido does have some unsafe code to interface with Windows and it has been the source of the hardest bugs.)

Java has a world class runtime with a great garbage collector and JIT compiler. But the language itself is not really a good fit for low level systems programming. gSuneido, with a simple byte code interpreter and no JIT still manages to outperform jSuneido. If Project Valhalla ever gets released that will help. But it's been "coming soon" since 2014.

Go felt like a bit of a gamble at the time. The garbage collector wasn't very good and neither was performance. But the language was a good fit for implementing Suneido. And it seemed like Google was committed to it. Although I wish it was a little less of a one-company language.

One of the things that bugged me about Go was its lack of generics. But now that it has them, I don't really use them that much. The people that resisted generics would say "I told you so". But it did simplify certain kinds of code and I'm still glad to have it when I need it. 

I wish Go had better support for immutability. C++ and Java aren't much better but at least they have a little bit. Immutability (and pure functions) are one of the best tools to handle concurrency, and concurrency is supposed to be Go's strong point. I think there's potential for a conventional mainstream language with strong immutability. I realize there are functional languages with strong immutability, but so far they're not really mainstream.

The big thing these days is Rust, but Suneido is garbage collected, and unless you want to write your own garbage collector (no thanks, been there, done that) the implementation language also needs to be garbage collected.

As far as the Suneido language itself, I think it has stood the test of time reasonably well. It was never intended to be especially novel or original, although at the time object-oriented programming and dynamic languages were leading edge. There have been a few minor adjustments to the language, a bit of syntactic sugar here and there, but for the most part code written 20 years ago would still run today. There are a few things I'd change but overall, when I program in Suneido, there's not a lot I wish was different.

Scala Native & More

I recently discovered Scala Native, a version of Scala that compiles to a native executable rather than running on the JVM like the original Scala. (via Modern Systems Programming with Scala Native)

Scala is an attractive language. Scala is to Java as C++ is to C. Meaning it's much more complex. (Scala promotes its simple syntax, but that doesn't mean it's a simple language.)

In some ways I like Kotlin better than Scala. It seems to give a lot of the advantages of Scala, without Scala's complexity. And there are non-JVM versions of Kotlin. But the problem is that Kotlin relies on the Java libraries, which is a portability problem for the non-JVM versions. Whereas, one of the strengths of Scala is its libraries (e.g. collections).

Libraries bring up an interesting issue. To write good libraries seems to require more features and complexity than application code. A lot of the complexity of C++ is to enable writing good libraries, and I suspect the same is true of Scala. The other alternative is the Go approach. Go has a built in "map" collection. You can't write map in the Go language. If you want a slightly different style of map, what you can write in Go is nothing like the built in one. This is at the heart of the Go generics struggle. On one hand, you want to be able to write your own map in Go. On the other hand, you want to keep Go's simplicity and not end up with another C++.

Another factor is that people seem to like complexity, regardless of whether it's counter-productive, myself included, sadly. Languages like C++ are fascinating. There's always something more you can learn. Whereas simpler languages like Go are vaguely disappointing in that respect. I love how the C++ folks gleefully pile complexity on top of glittering complexity. But when it comes time to actually get some work done ... then maybe you want something simpler. (Especially when the wondrous pyramid is built on unsafe ground.)

Complexity also tends to affect build times. C++ and Scala are both known for slow compiles, whereas Go is extremely fast.

Unfortunately, from what I can see of Scala Native it's not very mature. It's currently using the a conservative Boehm garbage collector. That's what I use in cSuneido and from experience I know the weaknesses of conservative garbage collection. It's definitely not as strong as the JVM garbage collector. But they are working on a new garbage collector. Go (and Mono) also started out weak in this area but improved greatly. I can't tell whether Scala Native has the same libraries (e.g. collections) as regular Scala. The libraries are (mostly?) written in Scala so I would hope they would be portable.

Scala Native also appears to be weak in the concurrency area, which seems like a fatal flaw in today's multi-core world.

One feature mentioned on the web site is that Scala Native provides access to malloc and realloc. While this might appear attractive to C programmers, it seems a little odd in a garbage collected language. It also allows you to explicitly allocate data on the stack (like alloca). I'd much rather have escape analysis (like Go) that does that automatically. (I did see a mention that they are working on escape analysis.)

I'm not sure if it's indicative of the project as a whole, but the PragProg book seems focused on writing unsafe C code in Scala syntax. Personally, I have no desire to go back to the days of buffer overflows and mysterious crashes from dangling pointers and memory access violations. It might give higher performance, but I suspect you'd get similar performance from writing it in safe Go.

It's an interesting project that I plan to keep an eye on, but I don't think it's ready for my use yet.

Go Interfaces and Immediate Integers

One of the unique things about the Go language is how it handles "dynamic" references compared to a language like Java.

In Java we have:

Whereas in Go we have:

In Go an interface is a "fat pointer". Your first thought might be, six of one, half a dozen of the other. What difference does it make where you put the type. However, there are significant tradeoffs.

Next, you might think, I could have multiple references to the same value, in which case Go will use more space because it will duplicate the type. But if we have a large array of values (in which case we're unlikely to have references to all of them), then Go would save space because it wouldn't need the type on each value.

The other big saving is if you have statically (compile time) typed references in Go they will look like:

And again, you save space because there's no need to store the type at runtime.

Given that background, the problem I was thinking about was how to handle integers in a Go version of Suneido. In Java you have to used boxed (on the heap) integers for generic references.

Older versions of Go were perfect for this purpose since they stored integers right in the interface in the same place as the pointer.

But sadly (for me) when they improved the garbage collector they removed immediate values:

The implementation of interface values has been modified. In earlier releases, the interface contained a word that was either a pointer or a one-word scalar value, depending on the type of the concrete object stored. This implementation was problematical for the garbage collector, so as of 1.4 interface values always hold a pointer. In running programs, most interface values were pointers anyway, so the effect is minimal, but programs that store integers (for example) in interfaces will see more allocations.

So now if you store an integer in an interface it gets allocated on the heap, effectively "boxing" it like Java does, although with less overhead.

For something like implementing Suneido, this really sucks. It adds a huge amount of overhead (relatively) on simple integer operations.

You could make an "extra fat" value that had an interface and an integer, but that's ugly too.

In C++ you can use a union to handle immediate integers and that's what cSuneido does. But Go doesn't have unions in the C/C++ sense.

But the union in cSuneido basically reserves a portion of the address space for use by immediate integers. i.e. a pointer falling in that range of addresses is assumed to be and is treated as an integer.

So why not do something similar in Go. Here's my proof of concept:

var space [1024 * 1024 * 1024]int8
var base = uintptr(unsafe.Pointer(&space[0]))

func ExampleSmi() {
   n := 123456789   

   x := &space[n]
   i := uintptr(unsafe.Pointer(x)) - base
   fmt.Println(i)
   // Output:   

   // 123456789

}

It does use the unsafe package, but only in a read-only, relatively safe way. (IMO) Pointers will always point into valid memory so the garbage collector will be happy. And it should be fast since the unsafe functions are primarily a compile time feature, they don't do much at runtime.

This example handles 30 bit integers (1 gb). (Unsigned here, but easily modified to handle signed.) I tried to allocate a 4 gb array to handle 32 bit integers but the compiler appears to have a limit of 2 gb. (Note: 4gb is an insignificant portion of a 64 bit address space.) That's too bad because I could probably ignore overflow from math on 32 bit integers, but not 30 bit integers.

We want to declare the "space" statically so that "base" is a constant. One question I had was whether this would add 1gb to the executable. With Go 1.9 on OS X it doesn't. (It appears to go in SNOPTRBSS which I translate as "no pointer uninitialized/zero memory") My other question is whether it's smart enough to allocate zeroed virtual memory and avoid actually zeroing the memory. I'm not sure how to check this, but there doesn't seem to be any lag in startup. Also, looking at the process in the OS X Activity Monitor is shows real memory usage under 2mb.  Presumably if the virtual memory is never modified it will never use any real memory. Also, importantly, the array is of type int8 so it won't need to be scanned by the garbage collector.

Even better, we can make our own type based on int8 (e.g, type Smi int8) and then define methods on this type. As long as we don't make the Smi type public and are careful to only make instances within the array then it will work like any other type. We basically have the old behavior of Go, albeit with a smaller range of integers.

It's a bit of kludge but it should work quite well and it's all standard (albeit "unsafe") Go. And it's no worse than tagged pointers  or nan boxing.

Suneido Functional Compose

I started on one project, found it required another, which then led to code like:

a(b(c(d(...)))

which seemed ugly. I knew that functional programming has a higher order "compose" to handle this more cleanly. (Higher order because it takes functions as arguments and returns another function.)

Suneido isn't specifically a functional language, but it has a bunch of functional methods and functions. I hadn't written Compose yet but it turned out to be quite easy.

function (@fns)
    {
    return {|@args|
        result = (fns[0])(@args)
        for fn in fns[1..]
            result = fn(result)
        result
        }
    }

If you're not familiar with Suneido code, the '@' is used to accept multiple arguments and to expand them again. The {|...| ... } notation is Suneido's way of writing a "block", i.e. a lambda or closure. (Suneido's use of the term "block" and the syntax come from Smalltalk, one of its roots way back when, pre 2000.)  Because, in Suneido, blocks are used for control flow, "return" returns from the containing function, so to return a result from a block we take advantage of how Suneido by default returns the value of the last statement.

Although in some languages the arguments to compose are listed "outside-in", I chose to have them "inside-out", so Compose(a,b,c) returns a function that does c(b(a(...))) in the same way you would write a *nix pipeline a | b | c i.e. in the order they will be applied.

I documented it, like a good programmer, except when I went to add the "see also" section I found a lot of the other functional stuff had never been documented. So I spent a bunch more time documenting some of the that. In the process I almost got sucked into improving the code, but I resisted pushing yet another task onto the stack!

Effective Modern C++

I just finished reading Effective Modern C++ by Scott Meyers. Like his More Effective C++ and the original Effective C++ it's well written with good explanations and examples. This third book covers the latest C++ features in C++11 and 14.

It's been a long time since I read the first two books. Effective C++ was published in 1991! Back then I was writing fair amounts of C++ code. Nowadays the only C++ programming I do is maintaining the C++ implementation of Suneido.

I expected the new book to be similar to the previous ones - practical advice on how to effectively use modern C++. And there is lots of that. But it was also full of "gotchas" - things that won't compile (and give horrendous error messages), or compile but won't run, or compile and run but do the wrong thing.

C++ has always been a complex language and the new versions have only pushed that even further. If makes me appreciate the simplicity of the Go language which in some ways is a reaction to the complexity of C++.

Don't get me wrong, the new features of C++ are great, they improve the language in many ways. But my head is spinning with things like when perfect forwarding isn't perfect, when universal references aren't, and when uniform initialization isn't uniform.

Safety First

I recently fixed a long standing (many years) bug in the C++ implementation of Suneido. A friend remarked how you'd wish that after this long all the bugs would have been found. Of course, it doesn't take much code to provide room for bugs to lurk.

The problem that was reported was that if you created one thread inside another that cSuneido would crash. It seemed to happen quite consistently and predictably. That was from the IDE. If you ran the same code without the IDE it worked fine. Or if you played around a bit in the IDE first, it would also work fine.

cSuneido "threads" aren't real threads. They are Windows "fibers" - more like coroutines. They don't actually run concurrently, but they allow cooperative multi-tasking. The big advantage is that since you control when the task switching happens and can do it at "safe" points in the code, you don't have to worry about low level concurrency issues. The downside is that you can't take advantage of multiple cpu's. But this was implemented at a time when no one had multiple cpu's and Moore's Law was still happily improving single cpu performance.

Suneido's C++ fiber code had a std::vector of fibers. It also had a main fiber, separate from the vector. The current fiber was a reference (pointer) to either the main fiber or an element of the vector.

Even from that minimal description you could probably guess the problem. Vector implementations normally grow by allocating a new larger array, copying over the data, and throwing out the smaller old array. So adding an element to a vector invalidates any references to its content. So the current fiber reference would be pointing to stale data. (It wouldn't actually be a dangling pointer because cSuneido uses garbage collection.) The reference to stale data could cause an "impossible" situation that would lead to a fatal error. (So the problem was nothing to do with creating one fiber inside another, it was simply that creating two fibers in that sequence happened to be one way to expose the bug.)

The problem was rare because it required a specific sequence of events. First, the vector had to grow. Which is why if you played around first (and expanded the vector) it wouldn't happen. Second, the stale reference had to be used in such a way that it caused a problem. Since the data would normally be identical the stale reference wouldn't matter. And the next fiber switch would update it to a valid value so the stale reference wouldn't hang around.

Actually, I think there was at least one more potential problem scenario. When fibers ended they were removed from the vector. This probably wouldn't cause a reallocation (many implementations never shrink the array) but it would invalidate any references after that item. You'd either end up with a reference to the wrong item or past the end of the array.

I'm a little embarrassed to discover such a long standing blatant mistake, and a newbie mistake at that. All the times I've looked at that code and I never picked up on it. Ouch.

But to me the real moral of the story is "don't use unsafe languages". Interestingly, this bug was not a memory management issue since cSuneido (unlike almost all C++ programs) uses garbage collection. It's just a result of C++ allowing unsafe raw pointers/references.

C++ fans would tell you that modern C++ has plenty of high level features that are "safe". But the point is that it still has lots of unsafe features. (And AFAIK there is no way to enforce use of a "safe" subset. And C++ continues to resist "real" garbage collection.) I would much rather work in a language like Java or Go (or others) that just don't allow unsafe code of this nature, and eliminate a whole class of problems. Figuring out my high level issues is challenging enough without worrying about unsafe low level issues.

Go: When nil isn't nil

Go is a fairly simple language. But it's still complex enough to have some oddities. I recently ran into one of them.

Lets say you have a nil pointer and you assign it to an interface:

var p *T = nil

intfc interface{} = p

I expected intfc to now be nil, but it's not.

The explanation (it's even in the FAQ) is in terms of the implementation. An interface contains a pointer and a type. When we assign a nil pointer the interface type is set. And an interface is only nil if both the pointer and the type are "empty".

I understand the explanation and it's not hard to work around once you're aware of it. If you want to return an interface that compares equal to nil you just have to make sure you don't assign a nil pointer to it.

But I don't find the explanation very satisfying. 

First, justifying some external behavior by explaining how you happened to implement it seems wrong. (Although to be fair, maybe the explanation is intended to be in terms of what an interface means, not really how it's implemented.)

Second, the explanation doesn't explain why they chose to make it work this way. Granted, it's simple because the nil check is just for a zeroed value. But it doesn't seem like it would be much harder for the compiler to just check for a zeroed pointer and ignore the type. It seems like this would avoid the unexpected behavior with no real loss of functionality.

I did some searching, but the only justification I could find is that a nil pointer in an interface can still satisfy the interface and you can still call methods on it. Which is fine, but what does that have to do with whether the interface value compares equal to nil? I guess it would introduce a new oddity in that you'd have two kinds of nil interfaces only one of which you could call methods on.

The other issue is that (afaik) you can't determine if an interface holds a nil pointer without using reflection.

It's not a big deal, and I'm not hung up on it, it just seems odd. As far as languages are concerned, I would say Go has relatively few sharp corners to get caught on.

The Go Programming Language

Go has been around for a while. I looked at it when it first came out but didn't get too excited, partly because there weren't any good books about it, and that's how I like to investigate a language.

Recently I've been looking at it again. I've read two books - Programming in Go by Mark Summerfield and The Go Programming Language Phrasebook by David Chisnall. An Introduction to Programming in Go by Caleb Doxsey is available for free. These are decent, but so far I haven't found a Go book that I'd call great. There's a lot of material on the web which is useful, but I still prefer a good book. Also, most of the material is introductory - I haven't found much expert level material.

Here are my thoughts on Go (1.2), in no particular order. These are personal opinions and biased by thinking in terms of implementing Suneido, because that's the best basis for comparison that I have.

+ safe
Pointers, but no pointer arithmetic. Array bounds are checked. In theory, crash proof. (Like Java, unlike C/C++) But if you need it, there's the unsafe package.

+ garbage collection
Not as mature or high performance as Java, but steadily improving. And because not everything has to be on the heap (as in Java) there is less pressure on GC.

+ available on main platforms (Linux, Mac, Windows)

+ optional semicolons
Minor, but it's one less thing to type and to clutter up the code, and definitely my preference. I was always disappointed that D chose to keep semicolons.

+ capitalization for public / private
I'm biased since I use a similar approach in Suneido. The only (minor) thing I don't like about it is that you can't use capitalization to differentiate type names as is normal in C++ or Java.

+ goroutines and channels
An attractive alternative to threads and locks.

– no generics
Not so critical for application code, but this makes it really hard for library writers to provide things like alternate data structures. There's a built-in map that is generic, but if you want an ordered map, or a concurrent map or an immutable map you can't make it generic. Obviously, this is not an easy thing to add and it adds a lot of complexity to the language, but to me it's a drawback.

? built-in maps (hash tables)
Maps are obviously a good thing to have, but making them built-in to the language seems like an admission that the language itself is not sufficient to write such things (due to lack of generics and operator overloading)

+ fast compiles
This was one of the explicit goals of Go.

+ simple, standard build system
No complicated make or ant files.

+ native executables (no runtime required like Java or .NET)
I don't mind having a VM like Java or .NET, but it is an extra hassle. Even with .NET, which is part of Windows, you can run into versioning issues. And having to distribute a 90mb JRE is a pain.

+ variable types can be inferred from initialization
e.g. var x = 123 or just x := 123
This avoids having to duplicate long types e.g. m := map[long_type]another_long_type

+ multiple return values, multiple assignment
e.g. a,b = fn() or x,y = y,x
This means you don't have to create special data structures (and in Java, allocate them on the heap) just to return multiple values.

– no ? : ternary operator
In conjunction with mandatory braces, this turns thing like "return x < y ? x : y" into 5 lines of code. Supposedly ?: is hard to understand, personally I've never found it a problem. If you don't want the extra operators you can make if-then-else an expression like Scala and write "return if (x < y) x else y". However you write it, it's still a branch you have to understand.

+ standalone functions
Not just methods as in Java. (Java 8 now has lambdas, but still doesn't allow top-level functions.)

+ functions are first class values
You can pass around references to functions, put them in variables or data structures, etc.

+ closures
Function literals are automatically closures that can reference variables in their environment, even if the function outlasts the environment.

+ slices
A slice is a safe reference to a portion of an array. Unlike C/C++ pointers, a slice includes length and capacity. This is a big improvement over bare pointers. (D has similar slices.) See Arrays, slices (and strings)

– no immutable or read-only or const (other than const scalars)
I'm not surprised at this, but it's too bad. Immutability is very useful, especially with concurrency. Interestingly, Go strings are immutable (so at some level people see the benefit), but nothing else. One unfortunate result of this is that conversions between strings and arrays of bytes require copying, even though the underlying data is identical. This is another issue, like generics, that can add considerable complexity to the language and type system. And even where it is attempted, like C++ const, it often isn't ideal. The D language does have immutable and pure. But I would love to see immutable data and pure functions. (You can write your own immutable data structures, but again the lack of generics and operator overloading makes them clumsy.)

– no concurrent containers
I realize the Go way is to use channels and goroutines for concurrency, but I think there are still going to be times when a concurrent container would be useful and give better performance. And without generics, it's hard for libraries to provide this.

+ not everything on the heap (as opposed to Java)
You can embed one struct inside another without a pointer and a separate heap allocation. And you can pass or return a struct or put it in a variable by value, again meaning it doesn't have to be on the heap.

? no classes or inheritance
This is one of the more unconventional aspects of Go. Not having written much Go code it's hard to judge this. My feeling is that Go provides good alternatives. The only drawback may be porting existing code that uses classes and inheritance.

+ no separate primitives and boxed versions
This is a pain point in Java and a definite performance issue when implementing something like Suneido.

+ type declarations
Similar to a C/C++ typedef except that a Go type declaration introduces a new type. This is useful to create short name for a complex type, or to prevent mixing incompatible units e.g. celsius and fahrenheit.

+ can define methods on scalars (e.g. int or string), not just on classes
For example, you could declare Celsius as int, and then define a ToFahrenheit method on it. (This is a bit like the Common Lisp Object System)

+ "duck" typed interfaces
Types satisfy interfaces implicitly (by having the required methods), they do not have to explicitly declare what interfaces they satisfy. Very nice.

? no function overloading
Minor. It means more name variations, but keeps the language simpler.

? no operator overloading
Mostly not an issue. Makes it more awkward to use library provided data types.

+ flexible switch statement
Not limited to integer constants like C.

? no assert
This is explained in the FAQ but I'm not sure I agree. But it's minor because it's easy enough to write your own.

? error return values instead of exceptions, no try-catch-finally
This is different than what I'm used to but I haven't written enough Go code to really evaluate it. Go's panic and recover are sufficient to implement Suneido's exceptions. See the FAQ, Error handling and Go, and Defer, Panic, and Recover

+ can integrate with C

+ strings are arrays of bytes (generally but not necessarily UTF8), not wide characters
The D language also takes this approach. It makes sense to me, and also fits with how Suneido works. One of the advantages is that it reduces conversions when reading UTF8 files. It also reduces the size in memory when dealing with mostly ASCII. See Strings, bytes, runes and characters in Go

+ decent standard libraries and an increasing number of third party ones

+ standard testing framework
Basic but reasonable.

+ good standard tools
Nice to have standard tools for building, formatting, embedded documentation, profiling, race detection, etc.

– limited IDE and refactoring support
Although there are Eclipse and IntelliJ plugins available, they are fairly basic and don't include much refactoring support. I've been using the Sublime Text plugin. It sounds like the primary Go developers aren't IDE users so this area has lagged a bit. There is some limited refactoring ability in the go fmt tool. This area will likely improve over time.

+ standard formatting
Minor, but it's nice to sidestep any formatting debates. I like that they chose tabs for indenting, since that's always been my preference. But I'm sure it bugs people who prefer spaces.

Fixing a Suneido Design Problem

In Suneido object.Delete(key) returns false if the member isn't found, otherwise it returns the object.

Several times we've had bugs resulting from doing things like:

object.Delete(key).Add(...)

which works as expected, except when key isn't found and Delete returns false.

It's likely the worst of possible designs. It would have been better if it returned nothing, or true/false, or my favorite - always the object.

I've been aware of this problem for quite a while but I was hesitant to change it because I was afraid I'd break existing code. I finally decided to go through our code and see if it would actually break much.

There were about 800 uses of Delete.

By far the majority ignored the return value.

I didn't find a single place where we made use of the false return value.

I did find a number of uses which assumed that it always returned the object - i.e. potential bugs.

Other than being tedious, the worst part was seeing all the ugly code. I wonder if it's possible to write code that doesn't make you cringe when you come back to it later.

I found quite a few places where we were doing multiple deletes so while I was at it, I changed Delete to handle multiple arguments.

This is a small change, but I think it's just as important to get the details right as it is to work on the big picture.

Kotlin - a JVM language

I've been aware of the Kotlin JVM based language for a while, but hadn't taken a close look at it. I have to admit I thought it was odd for a new language to be coming from an IDE developer (Jetbrains). But who am I to talk about who should or shouldn't develop their own language :-)

I like Scala for its power, but the complexity (especially the type system) scares me. I like Xtend for its simplicity, but it's somewhat limited. Kotlin seems to be aiming for a sweet spot somewhere in between the two - more powerful than Xtend, but not as scary as Scala.

Here are a few of the things I like (in no particular order)

• optional semicolons
• var and val 
• modest type inference
• extension methods (like D and C# but more powerful)
  (I like this better than Scala's implicit conversions)
• traits (interfaces with method implementations)
• function literals aka lambdas or closures
• IDE support from the start
• improved handling of null references
• no checked exceptions
• standalone functions (everything doesn't have to be in a class)
• operator overloading
• infix function calls
• default and named arguments
• no "new" keyword
• when expressions
• static imports
• a goal of fast compilation
• good Java integration

There are a few more attractive features like inlining and runtime generics that are coming.

Of course, because it runs on the JVM and has good Java integration, I could mix Kotlin with Java in jSuneido. And I'd still have access to libraries like java.util.concurrent and Guava and ASM.

Another intriguing feature is that Kotlin can compile to JavaScript. I'm not sure why they're working on this, since their stated motivation for Kotlin is to have a better alternative to Java to write their IDE and other tools. Regardless, it's a neat idea. I've even had the crazy idea of implementing Suneido in JavaScript, but I'm not sure what the advantage would be, other than running in the browser. And I prefer statically typed languages for systems programming. I guess if I converted jSuneido to Kotlin, then in theory it could run in the browser (except for all the Java libraries I use).

I installed Jetbrains IntelliJ IDEA Community Edition (free) and added the Kotlin plugin. I've never really used IntilliJ but it wasn't hard to get started. Hello world was easy and so was implementing one of the classes I've been using for language evaluation. The Kotlin version of this class was the most clear and concise of the languages I've played with lately. (Without being so concise that it becomes cryptic.) I didn't have any installation problems or weird errors. Although this is beta software, it seems quite good.

I have mixed feelings about Kotlin arising from a commercial company. On the one hand it can mean there are some real resources behind it. And if Jetbrains really does use it for their own applications, then they will have a strong motivation to improve and polish it. (And hopefully less tendency to go down the rabbit hole of complexity.) On the other hand, a company can decide to drop projects for various business reasons. Thankfully, much of Kotlin is open source, so it has a life outside of just Jetbrains.

In a way I'm glad I didn't look into Kotlin when it was first announced (July 2011). Even now, it's still in beta and somewhat in flux. But it seems far enough along to play with and consider using at some point.

Hi, my name is Andrew, I'm a programming language addict ...

nSuneido ?

It's been a long time since I've looked very closely at C# and .Net so when I saw a special on the C# 5.0 Pocket Reference I figured I should catch up. I read it cover to cover, and surprisingly (to me) I was quite impressed. C# has come a long way since version 1. It's certainly moved much faster than Java. In my recent System Programming Languages post I discounted C#, but I'm starting to wonder if that was a mistake.

* "nSuneido" comes from the convention of using 'n' for .Net projects e.g. NUnit, NHibernate, etc.

I also picked up C# in Depth by Jon Skeet (of StackOverflow fame). It was perfect for me because it covered the evolution of the language, not just how to use it.

One concern with C# and .Net is portability. But Mono seems to be in good shape and in active development. So far all my playing with C# has been on the Mac with Mono and  MonoDevelop. Of course, Mono lags somewhat behind - it's on .Net 4.0, whereas the latest is .Net 4.5. And it's performance isn't quite as good, but it's not bad.

There are a lot of things I like about C# and .Net (in no particular order)

• It meets my mandatory entry level requirements of safety and garbage collection
• Unlike Java, it has value types (in addition to reference types)
• Like Java, .Net has a VM and IL (byte code) that Suneido can target
  (whereas with native languages like C++ Suneido has to implement its own VM)
• Stable, robust, performant platform
• Rich ecosystem of books, tools, libraries, developers, etc.
• Easier to find programmers who know C#
• Extension methods 
• Class members and fields default to private 
• Decimal floating type (one less thing to implement)
• Lambdas :-)
• Delegates (method handles) 
• Named and optional arguments
• Type inference (var) although only for local variables
• Dynamic language runtime (DLR) 
• Good access to Windows API's (PInvoke)
• Annotations
• Anonymous types and tuples
• Operating overloading
• Nullable (a bit like Scala Option)
• Unsafe code if needed
• Good IDE support
• Generics on unboxed primitives (unlike Java)(no need for Trove collections)
• Reified generics (unlike Java)
• LINQ

There are, of course, some things I don't like. I'd really prefer optional semicolons (like Scala and Go), but I can live with them. It's taking a little getting used to capitalizing public methods and fields even though this is similar to Suneido. I miss Java's static imports - there's no way (that I've found) to have something you can call as simply Foo(). It always has to be Something.Foo()

With Java I could get almost all the tools and libraries I needed open source. With C# it seems a lot more commercial. Visual Studio and popular add-ons like ReSharper and Reflector are all commercial products.

I'm maybe just not up to speed on the C#/.Net ecosystem, but it doesn't seem like there aren't as many good libraries available. I would miss Guava. I need to learn more about what is and isn't available in the standard .Net libaries.

Once more, I started by implementing the Suneido lexer. This is turning into a coding kata for me, but I still find ways to improve or simplify each time I re-implement it. I took advantage of a number of C# features like extension methods and even lambdas. The code is on GitHub along with the D version. By the time I finished this exercise in D I had lost much of my enthusiasm and was somewhat disillusioned. In contrast, after the C# implementation I was still quite keen. In the long run, that probably doesn't mean much, but it's a good sign. As much as we might like these decisions to be objective it often comes down to subjective factors.

Ideally I would have just a single implementation of Suneido. (There are sometimes advantages to multiple implementations, but it's a lot more work.) Although the Java version is great for larger servers, it's not well suited to the current Windows GUI. And it's not great for smaller servers (e.g. smaller EC2 instances). An intriguing possibility is using Scala on both the JVM and .Net. I really like Scala, and the idea of writing one implementation that could be compiled to run on either the JVM or .Net is very attractive. The .Net version of Scala isn't production ready yet, but it's something to keep an eye on.

D-etractions

I love the potential of the D language, but I have to admit I'm becoming a little disillusioned with the current reality. This is of course, extremely subjective. I don't want to get in a fight with D supporters. I agree it's a great language. To me, most of the issues revolve around a lack of maturity and a small user base. If I was just playing, that wouldn't be a big deal. But for an implementation language for Suneido, I'm looking for something rock solid. Java may be old and boring and "weak", but it's solid.

One of the big issues for me is the lack of tools, especially good IDE support including refactoring. You know this is unlikely to be a priority when the main developers are old school text editor users. (In my mind I can hear them saying "we don't need no stinking IDE") Scala had the same problem until recently, also due to the main developers not using IDE's. But recently they've made IDE support more of a priority. Maybe not so coincidentally, this coincided with the formation of a commercial entity (TypeSafe) promoting Scala. I've done my share of old school editor + command line programming, but to me, a large part of the benefit of statically typed languages is that they allow powerful IDE manipulations.

A similar issue is the meager ecosystem e.g. books, libraries, tools, etc. I look at the libraries and tools I use with Java and few, if any, are available for D.

One thing that makes me a little nervous is D's fairly simple (from what I've seen) garbage collector. When I think about the effort that has gone into the JVM garbage collectors, I wonder how D will compare. D's garbage collector is also conservative in at least some situations (I'm not sure of the details). Based on my experience with conservative garbage collection in cSuneido, this increases my nervousnous.

D's templates and compile time function evaluation, and mixin's also worry me. They are very powerful and very cool, and they might be way better than C++ templates, but they're still complex. Of course, it's the usual story, you judge a new language when you're a newbie, so you don't necessarily have the basis to form a good opinion. But we seldom have the time to become expert before making a decision. I do have a fair amount of experience with C++ templates to judge by. Templates, as in D and C++, are undeniably powerful. Much more so than the generics in Java or C#. But I wonder if they are on the wrong side of the power / complexity balance. And these advanced features appear to be absorbing much of the energy of the development, to the neglect or even detriment of the overall maturity.

I love the potential of explicit "pure" and "immutable". I wish other languages like Java and C# had more of this. But from the meager exposure I've had, the reality is not as nice. I'm sure some of that is simply learning curve. And some of it may be improved by future language improvements. But when you can't put immutable values in a container without wrapping them in a "Rebindable" template, that again makes me nervous.

Of course, you could use a subset of D and try to ignore templates etc. but this is hard psychologically (I got sucked into mixins just to implement lexer tokens!), and also because the libraries make heavy use of templates. Then you'd primarily be consuming templates, not writing them, but you still need some understanding even to use them.

One of my concerns with Suneido implementation is making it easier for other programmers to get involved in the implementation. (i.e. improving the bus factor) In that respect, I think the jSuneido Java code is much better than the cSuneido C++ code (with templates!). And it's probably easier to find Java programmers than C++ programmers, let alone D programmers. (Of course, the challenge of finding good programmers remains.)

I plan to keep an eye on D and hopefully continue to play with it a bit. I wish the project good luck. But for now, I am going to put on hold any ideas of using it to replace cSuneido.

See also: D-tours and System Programming Languages

D-tours

I've been continuing to spend a little time here and there playing with the D language.  For a sample project I decided I would try implementing Suneido's lexical scanner. It's pretty simple, just a little string and character manipulation so I thought it would be easy.

I based the D version on the Java version since it's newer and cleaner than the C++ version.

The first thing I ran into was that the lexer returns "tokens", which in the Java code are not just integer enums, but actual enum classes with fields and methods. Java went from no enums in early versions, to quite sophisticated enum capabilities.

In typical new user fashion, I tried to reproduce the Java style token enum in D. (It's hard not to avoid "cutting against the grain" when learning a new language.) D does allow you to use structs for enums so it seemed this would work. But if you use structs then you lose the automatic assignment of consecutive integer values, and D struct's are value types so they aren't necessarily unique.

Then I went off an a tangent looking into D's "mixin" ability. D has the ability to run D code at compile time, generate a string, and insert that string into your source code. This is known as CTFE (compile time function evaluation). For example, you can write:

mixin(myfn(arguments...);

where myfn returns a string containing source code to be compiled into your code in place of the mixin "call". This is easier than trying to use C++ templates to do compile time calculations, but not as seamless as Lisp macros because you have to work at the level of strings.

An example of the power of this is the Pegged library for D. This allows you to include a grammar specification in your source file, have a parser generated from it, use it to parse DSL code also included in your source file, and then use the resulting parse tree to generate D code - all of this at compile time, without requiring any external tools.

It was fun looking at what you could do with mixin but it was overkill for what I was trying to do. I realized I could use another feature of D to use simple integer enum tokens, but still work with them as if they were objects with properties.

enum Token { IF, ELSE, AND, OR, ...
bool isKeyword(T token) { ...

D allows you to call functions like isKeyword as if they were methods on their first argument, so you can say token.isKeyword() even though token is an int, not an object.

This also led me to discover D doesn't have a "set" data structure, which surprised me a little. It does have hash maps built-in to the language, which can be used to emulate sets, and it has a red-black-tree implementation which could also be used. But it doesn't have an actual "Set" class. It was easy enough to throw together my own Set so I could make a set of keywords. (Which I didn't end up using because what I really needed was a map of keyword strings to tokens.)

What I initially wrote was:

static auto keywords = new Set!Token(IF, ELSE, ...);

But that won't work because static initializers can only be simple literals. However, you can write:

static Set!Token keywords;
static this() { keywords = new Set!Token(IF, ELSE, ...); }

I'm not sure why the compiler can't do this same translation as simple syntactic sugar. It's extra annoying because you can't use the "auto" type inference.

You could do it with a mixin like:

mixin(stat("keywords", "new Set!Token(...);");

but that's not exactly pretty.

Another minor annoyance was the lack of an equivalent to Java's static import. If I want to refer to the tokens as just IF or ELSE I could make them "anonymous". But then they don't have a specific type. Or I can give the enum a named type, but then I have to always reference them with Token.IF . In Java I could say import static Token.* and then just use the bare names.

I ended up going back to a mixin approach with each Token a static class. I used a class rather than a struct because I wanted to pass tokens around by reference, not by value. See token.d

I also ended up writing multiple versions of the lexer. As with most rewrites, I found a number of improvements I could make. Most of them aren't specific to D, I could apply them to the other versions as well.

After getting it working, I decided to go back and write it in a functional (rather than object oriented style). In this version the lexer is a function that takes a string and returns a struct with the token, its string value, and the remainder of the source string. This version should work with Unicode UTF8 strings, not just ASCII, because I only access the source string with front and popFront.

Along the way I also started a simple version of Hamcrest style asserts. I almost expected something like this to be available, but I couldn't find anything.

One of the features I was excited about with D was immutability. But when I tried to make Token immutable, I ran into all kinds of problems (as others have also). One of them being that you can't store immutable objects in a map! Hopefully this is something they'll figure out. I eventually gave up on immutable and managed to get const to work, although even that was a struggle.

I set up a dsuneido GitHub repository for my experimenting. I haven't used GitHub before, but I keep hearing how great it is so I thought I better give it a try. I used their Mac GUI app, which made it quite painless. However, I'm not sure it's the smartest idea to be using three different version control systems - Subversion for cSuneido, Mercurial for jSuneido, and Git for dSuneido. I chose Mercurial for jSuneido because they had better Windows support than Git at that time, but now that Eclipse comes with built-in Git support, I wonder if I made the right choice.

I'm not sure where I'm going with this, so far I'm just playing and learning D. So far, I like it, but it certainly has some rough edges.

System Programming Languages

Or, more specifically for me, languages I'd consider using to implement Suneido. Here are some opinionated thoughts on C++, Java, Scala, Xtend, D, and Go.

First, I want garbage collection. Other than for things like constrained devices, as far as I'm concerned garbage collection has won.

For me, that's a major drawback to C++. So I wrote my own garbage collector for C++ which we used for a number of years. Eventually I replaced it with the Boehm garbage collector which has worked quite well, given the limitations of conservative garbage collection.

The last few years I've been using Java which gave me great garbage collection and one of the best virtual machines out there.

The big advantage to Java is the rich ecosystem. You have multiple choices for great IDE's (e.g. Eclipse, Netbeans, IntelliJ) with good refactoring support, lots of tools like JConsole and Visual VM, good libraries, both standard and third party (e.g. Guava and Asm), and tons of books and other sources of information.

On the other hand, the Java language itself is nothing to get excited about. Lately I've been looking at Xtend and Scala. Xtend is fairly modest, it's basically a better Java. See my First Impressions of Xtend

Scala is much more ambitious. It has some great features. The claim is that the language is simpler than Java. But I'm more than a little scared of the type system. Although it's that type system that allows some of the great features. Scala strikes me a bit like C++, you can do some amazing things, but it can also get pretty twisted.

I'd be more tempted by Scala, but as system programming languages, JVM language all have a major drawback - you can't write efficient low level code. Don't get me wrong, I have no desire to write "unsafe" code. I don't want to go back to my C and C++ days. Java has primitive (non-heap) types, but they don't work with generic code (without boxing onto the heap). I'd like to have "value" types (e.g. a pair of ints) that I could pass and return by value (not on the heap) and embed (not reference) in other data structures. Scala tries to unify primitives with the other types, but ultimately it comes down to the JVM, and there's still a lot of boxing going on.

Also, Java is primarily in the locks and shared state style of concurrency, a style whose drawbacks are well known. Scala is better on this front with its actor system, which admittedly is also possible from Java.

What else is out there? C# is a possibility, but I have to say I'm leary about Microsoft products. And portability means Mono, which is another question mark in my mind.

Recently I've been rereading the D book by Andrei Alexandrescu (also author of Modern C++). You can get a taste of his writing style in The Case for D. In some respects the D language could be called a better C++. But it has gone beyond that and has some very attractive features. It has value types as well as reference types. You can write efficient code like C++, but still make it safe.

D has garbage collection, although I suspect (with no evidence) that it's not as good as the JVM. Of course, part of the reason so much effort has gone into JVM garbage collection is that everything has to go on the heap. That doesn't apply quite so much in D (depending on the style of your code).

One of the features I really like about D is its support for immutability and pure functions. My feeling is that these are really important, especially for concurrent code. You can, of course, write immutable data structures and pure functions in any language, but D actually lets you enforce them. Unlike C++ const or Java final, D's immutable and pure are "deep". Making an array final in Java doesn't stop you from modifying the contents of the array. Making an array immutable in D does.

One minor disappointment is that D still requires semicolons. I think languages like Scala and Go have shown that it's quite feasible to skip the semicolons. But I could live with this.

I'm not against virtual machines like the JVM or CLR. But they have their drawbacks, especially for deployment. Ensuring that customers have the right version of .Net is bad enough and that's a Microsoft product on a Microsoft platform. We can require our customers install Java, but when they don't update it and there's a security hole, guess who they will blame? One of Suneido's main features is that it is self contained and easy to deploy. Requiring another runtime underneath it is not ideal. So D and Go generating native apps is attractive in this respect.

Another advantage to D is that (I think) it would allow accessing the Win32 API so Suneido's current user interface would still work. (albeit still only on Windows.)

Yet another possibility is Google's Go language. Personally I find it a little quirky. It's got some good features, but it's also missing things that I like (e.g. type safe containers). I'm currently reading Programming in Go but I'm finding it somewhat dry.  I find it a lot easier to learn about a language from a well written book. I didn't really get interested in D until the book came out. And books got me interested in C and C++ and Scala.

One of the things discouraging me from Scala and D and Go is the lack of IDE support. That's gradually improving, but things like refactoring are still pretty limited. I programmed for years with just an editor, but once you get accustomed to a good IDE with refactoring support, it's hard to go back. To me, one of the big advantages of a statically typed language is that it allows great tooling. Unfortunately, Java seems to have a big lead in this area.

Currently, if I was to consider another language for implementing something like Suneido, I think I'd lean towards D.

First Impressions of Xtend

I've been interested in Xtend for a while. It's an improvement over Java, without being a big jump like Scala.

I finally got around to playing with it a little. I took one of my Java classes from jSuneido and rewrote it in Xtend. But I got stuck with some weird errors around array access. I did some searching on the web and found that Xtend doesn't support array indexing e.g. array[i]. I was surprised. You can pass arrays and do for loops over them, but not index them.

I imagine part of the reason for this is that square brackets are used for closures, although in different contexts, so it seems like you could still parse it ok.

Then I saw you can write array.get(i) - more verbose, but more consistent with other containers. (Personally, I'd rather be able to use x[i] or x(i) like Scala for all the containers.)

Then I discovered that the way it handles .get(i) is to wrap the array in another class. This is similar to how Scala does implicit conversions to add methods to existing classes.  There's nothing wrong with this approach, but it means allocating a new object for every array access. Considering arrays are usually used for low level efficiency, this doesn't seem ideal. Granted, in some cases the JIT compiler may be able to eliminate the allocation, and if this technique gets common, it's more likely it will be a target for optimization.

It seems like a better approach would be to recognize array.get(i) and compile it to array access byte codes, without any wrapping. Maybe that's hard to do in their compiler.

Another alternative is to leave array code in Java. Which is fine, but doesn't leave a good first impression.

One of the justifications given for not supporting array indexing is that you should use the higher level containers instead. I think there are times when the extra efficiency of bare arrays is justified, but for the most part I agree. And, in fact, the class that I converted could just as easily use ArrayList's rather than arrays, with likely not too much performance penalty.

Even after I fixed this I still had some weird errors left. It turned out that Xtend doesn't have increment and decrement operators (++ and --) Fair enough, Scala doesn't have them either. But I was more surprised to find that it didn't handle += either, although it is listed as an operator in the documentation. For a language that claims "Less Noise", it seems odd to have to write "i = i + 1" instead of "i++".

Once I got past these issues, I was pretty happy with the results. Even without using any of Xtend's more powerful features (like closures and extension methods) the code was cleaner and less verbose, although not radically so.

One of Xtends claims is "top notch" Eclipse IDE support. Unfortunately, that doesn't mean much in the way of refactoring.

I'm still left with doubts about the future of Xtend. It doesn't have the momentum or backing of Scala. With a "smaller" project like this, it could easily stagnate, mutate, or die.

Immutable and Pure

More and more I find myself wanting a programming language where I could mark classes as immutable and functions as pure (no side-effects) and have this checked statically by the compiler. Being able to mark methods as read-only (like C++ const) would also be nice.

This is coming from a variety of sources:
- reading about functional languages like Haskell and Clojure
- working on concurrency in jSuneido (immutable classes and pure functions make concurrency easier)
- problems in my company's applications where side-effects have been added where they shouldn't

I have been using the javax annotation for Immutable, which in theory can be checked by programs like FindBugs and that's a step in the right direction.

There are a lot of new languages around these days, but so far I haven't seen any with these simple features. Of course, in a "true" functional language like Haskell, "everything" is pure and immutable (except for monads), so this doesn't really apply. But I think for the foreseeable future most of us are going to be using a mixture.

Write Your Own Compiler

I recently listened to a podcast of Scott Hanselman talking to Joel Spolsky.

One of the things they talked about was how Fog Creek (Spolsky's company) wrote their own compiler because it was easier than rewriting their application, which was written in an ancient version of VBScript.

I've always been a little defensive about how my company writes their applications in our own language (Suneido). At best people look at me funny, at worst they think I'm crazy. Our salesmen have to skirt around the issue because it scares people to hear you're not using big name tools. As the old saying goes, "no one gets fired for buying IBM". Nowadays you can substitute Microsoft or Oracle or Java.

So it was heartening to hear Spolsky talk about (aka defend) why they wrote their own compiler. One of his points was that writing a compiler is not that hard. People tend to think it's a big deal, but for a smallish language it's not that big a job. For example, Suneido's compiler makes up a tiny fraction (less than 1%) of the total source code in my company's main application.

I'm the first to admit that part of the reason is simply that I like developing languages (and database servers, and IDE's, and frameworks) better than I like writing applications.

But there are other reasons. Having your own platform has its costs, but it also has its benefits. It's a good way to insulate yourself from the fast pace of change in the computer industry. My company still has customers running software that we originally wrote on MS-DOS. The exact same application code is now running on Vista. There are not many commercial platforms that can claim that. Even things like Java that are supposed to be stable platforms change a lot over the years. And Microsoft changes languages and frameworks faster than you want to keep up with. (Spolsky's problem.)

Of course, that can mean you're not using the latest bleeding edge technology, but that's not such a bad thing for a business application.

Language Connoisseur

It took me a few tries to come up with the right word. Addict? Fanatic? Polyglot? Aficionado?

I think "connoisseur" fits best - "A person of informed and discriminating taste e.g. a connoisseur of fine wines."

I love to read about different programming languages, but I wouldn't say I actually "learn" them. I've only written significant amounts of code in three languages, C, C++, and now Java. It takes years for me to feel like I'm getting "good" at a language.

But I can "taste" other languages, and like a wine connoisseur, critique them (at least in my own mind). Ah yes, made from JVM grapes, definite functional nose, aged in strong typing, goes well with concurrent meals. (Sorry, I'm stretching, but you get the idea.)

It's interesting to note that 4 of the 6 languages that I've read about recently are JVM based. That's partly because I'm interested in JVM based languages because of jSuneido. But I think it also indicates a trend. There are languages based on .Net as well but for some reason they don't seem to be as popular, or at least have as many books.

One of my big interests these days is concurrency, again partly because of having to face it with jSuneido. But it's also something we all have to face as cpu's no longer get faster, and instead we get multiple cores.

It's becoming pretty obvious that threads, mutable data structures, and locking are not the answer. They work, and with a lot of effort it's possible to write programs using them, but it's hard and error prone. I think they will become the "machine language" of concurrency. Present under the covers but not dealt with directly by most people.

So the alternatives in other languages are pretty interesting. Erlang and Scala have "actors". Clojure has software transactional memory.

Pure functions and immutable objects seem to be part of the answer. Transactional memory is promising. All three of these go together.

When I first started reading about Clojure I was confused by the references to "persistent data structures". I was thinking of persistence as saving things in a database. But that's not what they're talking about. Persistent data structures  in this sense are data structures that are optimized for making modified immutable "copies" that share representation. The classic and simplest example is a single linked list (like in Lisp). For example, when you add a new item to the front of a list the new list "shares" the old list.

Suneido strings are immutable and use persistent data structures. Unlike most languages, when you concatenate onto a string in Suneido it doesn't create a whole new string. Instead, behind the scenes, it creates a list of two strings. This is transparent - to the programmer it's just as if it created a new string.

This answered a question that had been lurking in the back of my mind. Isn't it really slow to use large immutable objects in functional languages? Now I can see how it can be done efficiently using some clever algorithms like Clojure's persistent hash map.

I'd been trying to figure out how to implement an efficient concurrent version of the schema cache in jSuneido. Suneido stores the database schema in tables in the database. But for speed it caches it in memory. The schema doesn't tend to change very often so it is well suited to making it immutable (and therefore not requiring locking). But it does change occasionally, and when it does, transactions have to see the appropriate version. (Suneido uses multi-version database transactions.) A persistent data structure is perfect for this because each transaction can have it's own immutable "copy" without physically having to copy the schema (which can be quite large).

However, there's a catch. cSuneido loads the schema lazily (on demand). This adds a twist - the schema cache is logically immutable, but physically mutable. That would require locking, which I'm trying to avoid.

Then I realized that lazy loading was to speed up start up. But jSuneido is primarily (at least initially) aimed at the server, where start up speed is not as important. So I can just load the whole schema at start up, eliminating the problem. Nice.

There is another catch. Java doesn't come with any persistent data structures :-(  and I don't particularly want to write my own.  Java doesn't even have a single linked list.

I did a quick search but I didn't find any third party libraries either.

One possibility is to use the Clojure data structures. (they're written in Java) I took a quick look at the source code and it seems like it might be feasible but I'm not sure how dependent they are on other parts of Clojure. If I'm lucky they'll be usable on their own.

Of course, adopting other code always has it's price. If there are bugs I won't know how to fix them. If Clojure fixes, updates, or improves the code I'll have to decide whether to switch to the new version.

Fun stuff! :-)

Scala Attractions

Here are some of the things I find attractive about Scala. I still haven't written a line of code, though, so don't take it too seriously. Just first impressions of a language connoisseur :-)

user defined operators
In C++ you'd call this operator overloading, but Scala doesn't have a reserved set of operators. Even "+" is just a method. Obviously, this has potential to be abused, like any other feature. But it's a lot nicer to be able to say x + y for a new type of numbers, rather than x.plus(y)

traits
Scala traits are like Java interfaces, but they can include method implementations. They're similar to mix-ins. I struggled to do similar things in C++ so I can see the value.

implicit conversions
Ruby and other dynamic languages have "open classes" where you can modify existing classes at any time. They call it "monkey patching". It can be really useful, but I also find it really scary. Maybe I'm just a wimp, but I don't want standard classes changing under me. If I use some third party library or plug-in, I'm not sure I want the standard string or list class to change as a result.

Scala takes a different approach - you define implicit conversions to your own class. So instead of adding methods to String, you define an implicit conversion to MyString, which then has the extra methods you want. Scala provides a number of "rich wrappers" with implicit conversions for things like RichString and RichInt.

This is a good example of why you need to focus on the end, not the means. If what you want is to be able to write str.myOperation() there's more than one way to do that, and it doesn't necessarily require a dynamic language.

Of course, implicit conversions can cause problems as I know well from C++. But Scala's approach looks like it avoids most of the problems.

everything's an object
In other words, no more "primitive types", like in Java, that have to be handled specially. And no more boxing and unboxing. It's too bad Java didn't take this approach. There are performance issues, but with todays compiler technology they can be overcome.

first class functions
Meaning you can have unnamed function literals and pass functions around as values. To me, this is huge. Sure, you don't need this all the time, but to write higher level code it's invaluable.

lazy
This is a little thing, but I think it's cool that Scala provides a built-in way to do lazy initialization. As opposed to Java, where Effective Java has a whole section on how to do (and not do) lazy initialization.

type inference
Scala seems to combine the best of both worlds - the benefits of static typing, with almost the brevity of dynamic languages. Would you rather write:

val a = Array("one", "two", "three")
val b = new Map[String,String]

or:

String[] a = new String[] { "one", "two", "three" };
Map<int,string> b = new HashMap<string,string>();

I've talked about this before. C++ is just as bad as Java, although they are proposing an improvement in C++0X.

== is equals
I think Java made the wrong choice in making == compare pointers and equals compare values. I've talked about this before. It's great to see Scala break with Java and normally make == the same as equals.

actor model for concurrency
This is actually a Scala library rather than part of the Scala language. But it's a standard library. The actor model is what has made Erlang known for writing concurrent applications. I'm not sure how Scala's implementation compares to Erlang's, but it's nice to see tools for concurrency. Especially coming from C and C++ that basically said "not our problem". I wonder if I can make use of this when I get to making jSuneido's database concurrent?

lists and tuples
These are classic functional data structures. Although I wouldn't say I "know" functional programming, I'm certainly not ignorant of the benefits of imutable data structures and the dangers of side effects. My programmers will tell you I've been know to harp on the evils of side effects. cSuneido makes good use of lists and other imutable data structures.

class parameters
This looks like a great shortcut for writing simple classes. Would you rather write:

class Data(count: Int, name: String)

or:

class Data {
    final public int count;
    final public String name;
    Data(int count, String name) {
        this.count = count;
        this.name = name;
    }
}

optional semicolons
If nothing else, this will save me from the annoyance of typing semicolons in Suneido where I don't need them, and then turning around and omitting them from Java where I do! The joys of switching languages every other day!