Two Steps Forward, One Step Back

Or should that be One Step Forward, Two Steps back?

Yesterday I was back to working on the multi-threaded ACE Suneido server. It started off really well. I got the server actually running, could connect and disconnect, and make simple requests and get responses. A very good milestone.

The obvious next step is to try to start up a Suneido client IDE using the new server. This is a big jump because even to start up requires a large number of requests of different kinds. I didn't really expect it to work, but programmers are eternal optimists :-)

I got a variety of weird errors and memory faults - not surprising. But amongst these errors was one that said something like "GC collecting from unknown thread". Oh yeah, I knew this was going to be an issue but I'd pushed it to the back of my mind. The garbage collector needs to know about all the threads in order to take their stacks and registers etc. into account. The way the Boehm GC normally does this is by requiring you to call their create thread function which is a wrapper around the OS one.

The problem is, ACE is creating the threads. I found where ACE calls create thread and thankfully there was a single point I could modify to call the GC version. But, I was using ACE via a DLL which means it can't call functions in the main program (where the GC one is).

The obvious solution is to not use a DLL, to statically link in the ACE code. Sounds easy. I even found the option "static_libs=1" that would build static libraries. But it doesn't work. It builds a static library alright, but when I try to link it into Suneido I get a bunch of "multiple definitions" and "undefined references". Suspiciously, many of the names were "_imp_..." which seems a lot like the way DLL's work. My guess would be that "static_libs=1" isn't working fully, which isn't too surprising given that the "normal" usage is with shared libraries (DLL). In software, "taking the path less traveled" is often a recipe for frustration.

I started digging into the maze of headers, configs, makefiles, and ifdefs but I ran out of time. Presumably it's solvable. You can see why people like to use things like .Net or Java where at least some of these low level issues are handled for you.

At the same time as I was working on this, I downloaded Ubuntu 7.10 and created a new Parallels VM (on my MacBook while I was working on my main machine). I used the alternate cd with the text based installer as recommended by other people. It went quite smoothly (except for crashing OS X the first time I started the install), and no display problems. But when I tried to install the Parallels Tools, the disk image appeared "corrupted" - only a single file and its name was random garbage characters. I tried rebooting the VM, restarting Parallels, and rebooting the MacBook but it didn't help. I searched on the web but didn't find any references to this problem. I have upgraded my MacBook to Leopard (the new version of OS X) so the problem may be related to that. When I get time I'll try running this VM on my Mac Mini which hasn't been upgraded to Leopard yet.

Still Problems with Ubuntu 7.10 on Parallels on Mac

Larry has been patiently trying to help me with the network problem on my Ubuntu virtual machine (see the comments on Ubuntu on Parallels on Mac) He may have even found the problem.

But when I applied the fix and rebooted I was back to the problems from More Fun With Ubuntu on Parallels - unable to boot because the X display won't start. I did some more thrashing around and some more web searching. Lots of people seem to have this problem and there are various proposed solutions. I wonder if these "solutions" aren't really solutions - it just happened to work as it does for me occasionally. Frustratingly, lots of people also appear to not have this problem - it works fine for them.

This is on a Mac (mini) so you can't blame non-standard hardware. Ubuntu is one of the most common (if not the most common) version of Linux. I haven't loaded up OS X with a lot of junk software. So why am I still faced with these frustrating issues? I don't want to have to be an X Windows expert and mess around with xorg.conf. Running in a virtual machine adds some complexity, but it also reduces the complexity since the virtual machine is more "standard" than real machines.

I don't have a lot installed in my Ubuntu so I am going to try a fresh install. At least with VM's this doesn't mean I have to wipe out my previous one. It would be nice if Parallels had a prebuilt Ubuntu 7.10 VM but they only have 7.04. I could go back to 7.04 but sooner or later I'll want to update.

Some of my recurring frustrations are, no doubt, due to my staying to close to the "bleeding edge". But, (a) I want to keep up with the latest - that's part of my business, and (b) not doing updates has it's own problems with incompatibility, security, etc. and leaves you facing even scarier "big" updates, albeit not so often.

Groovy or JRuby

http://martinfowler.com/bliki/GroovyOrJRuby.html

Slow Progress on Multi-Threading Suneido

Last time I worked on this I had just reached the point where I could successfully compile and link. I resisted the temptation to try running it because I figured there were bound to be problems.

Sure enough, I run it ... and it crashes with a memory fault. No big surprise. I put in a few print's to see where it's getting to. Hmmm ... it's not even getting to my main!?

Download and install MinGW GDB and use it to find it's crashing in the garbage collector code. I am redefining operator new to use the garbage collector so presumably something is calling operator new during static initialization. I comment out my redefinition and it works. I make a tiny test program:

#include "include/gc.h"

void* stat = GC_malloc(10);

int main(int argc, char** argv)
{
return 0;
}

It crashes. But this works in the current version of Suneido. It must be thread related. Yup, my test program runs fine without threads. Now what?

I search the web to see if this is a known problem but I don't find anything.

I'm still using gc-6.5 and the notes for the latest gc-7.0 mention improvements to win32 threads.

So I download gc-7.0 The readme says MinGW is not well tested/supported - ugh. For gc-6.5 I had ended up writing my own makefile but I'd prefer to avoid that. The recommended approach seems to be the standard configure & make so I try this with MinGW MSys.

configure seems to succeed, at least with no obvious errors, but make fails with a bunch of "undefined references". It appears to be trying to make a dll which I don't want - I want a static library. Eventually I hit on configure --enable-shared=0 which avoids the dll stuff but still gives a bunch of "undefined references". This time they all appear to be from win32_threads.c For some reason this isn't getting included in the build. I uncomment am__objects_3 = win32_threads.lo in the generated Makefile to fix this. That's probably not the correct solution but it does the trick and I finally get the build to succeed. gctest runs successfully although it seems slower and in the output the heap is twice as big as with gc-6.5 - not good, but I'll worry about it later!

Thankfully this effort wasn't wasted and my test program runs successfully. And Suneido now manages to get to main! But then it fails with ACE errors saying WSA Startup isn't initialized. This is easily fixed by adding ACE::Init but it's strange because I didn't need it in my previous ACE test program.

After most of a day's work I'm finally back to where I can start debugging my own code! It's great to be able to leverage other people's work (like the Boehm GC and ACE) but it can be extremely frustrating when they don't work and you don't have a clue what's going on. Even the standard configure & make has the same problem. If it works it's great, but if it doesn't you're faced with incomprehensible makefile's.

Positive Feedback for a Change!

It seems like all I do is complain about my frustrations with computers so I thought I should post a positive comment for a change.

I went to check what version of g++ I had on my Ubuntu on Parallels. Here's what I got:

andrew@MacMini-Ubuntu:~$ g++ --version
The program 'g++' can be found in the following packages:
* g++
* pentium-builder
Try: sudo apt-get install <selected>
bash: g++: command not found

This is a vast improvement over just "command not found". Thumbs up to Ubuntu (or Linux or wherever this originated).

Another Mac Printer Annoyance

Just when I thought I had my printer problems figured out, I have a new one.

If I boot up with the printer turned off i.e. turn it on after OS X has booted, then it won't work. Not only won't work, but hangs OS X for several minutes. The first few times I thought OS X had crashed but if I'm patient enough it will come back. It's especially frustrating when I inadvertently leave Lightroom in the Print module because then it hangs when I run it. Or if I forget and switch to the Print module.

It might not be so bad if I could put the printer on the power bar with the Mac so it would get turned on at the same time. But you're not supposed to shut off the power to the printer without turning it off and I'm sure I'd forget.

It's strange (as usual) because the printer is connected to the Mac with USB - which should (and did before) handle turning on later. Maybe it's related to the work around I had to do to access the printer from Parallels since that is network related. (I'm thinking network because that's the only thing I can think of that would hang for several minutes.) I guess I could try disabling or removing the extra printer I have set up for that but I'm not sure I'm in the mood for it. And I'd probably just break something else.

It seems like a lot of stuff these days assumes you're going to leave it turned on all the time. But that's not great for energy efficiency.

More on Multi-Threading Suneido

I am continuing to intermittently work on making a multi-threaded Suneido server (that will also run on Linux as well as Windows).

So far, I have stripped out all the Windows dependencies and just about have the socket stuff converted to ACE. It should not take too much more to get this working - but only single threaded.

In a sense, this is the easy part. The hard part is to add the required locking etc. to allow it to run safely without the multiple threads interfering with each other.

There are two sides to this - the database and the language run time (virtual machine).

On the database side, the data itself should not be a problem because it is immutable (never updated). When records are updated the new version is added to the end of the database file. (This is the main reason the database needs to be "compacted" periodically.) The primary reason for this is to support the multi-version optimistic database transactions but it also ends up being nice for multi-threading.

The indexes are the main issue in the database. The easiest solution is probably locking at a fairly granular level of entire indexes. There are schemes for locking individual index nodes, but this is tricky. It should be easy to use multiple readers OR single writer locking. The downside is that if there is a lot of updating it will end up being single-threaded again.

Ideally, I would like to support multiple readers concurrently with updating (but still only one writer at a time) similar to how multi-version optimistic transactions allow multiple readers to operate independently of updates. But I have not figured out any "easy" way to make the indexes similarly multi-version so readers are not blocked by updates.

This may not be critical because I am pretty sure read's are far more common than update's. I should really measure this instead of guessing!

The other side is the language virtual machine. This should not be too bad because there is not much shared mutable (updatable) data. The main shared data structure is the table of global definitions loaded from libraries. The only time this is modified is when a new definition needs to be loaded from a library. At one time I thought I could use the double checked locking pattern (DCLP) to avoid synchronizing every access, but DCLP has been found to be fatally flawed. In theory, with 32 bit values and an idempotent function it is still workable, but given the history it seems risky. Another way to avoid the synchronization overhead would be for each thread to have its own globals table "cache" and to load this from a shared synchronized table.

I am sure there are many other fun issues lurking in this project. I am still very paranoid about synchronization issues that do not show up until after deploying it to hundreds of customers. My first line defense will be to try to keep the locking simple so I can have a fair chance of convincing myself logically that it is "correct". (Although judging by DCLP, even very smart people can fail to catch concurrency flaws in even simple code.) The next line of defense will be some serious stress testing, probably on something like a quad core machine to increase the chances of conflicts.

The Chumby Has Landed

My Chumby finally arrived. I was happy to see natural cloth packaging instead of yet another frustrating bubble pack. Of course, the first thing it wanted to do after I set it up was to download a bunch of updates. [aside: Automatic updates seem like a great idea, and they would be if they were unobtrusive. But every time I try to do anything on my computers something wants to do an update and disrupt me while it does it. It's especially bad on machines that I don't leave turned on and don't necessarily use every day.]

Here's what I have currently playing on my Chumby:



I'd like to make my own widgets but it looks like that requires using Flash which I haven't done before. For example, the Chumby would make a great (although expensive) status monitor for our automated tests.

Amazon's Kindle Released

Amazon has released its Kindle electronic book reader

And once more, it's US only :-( That may be partly because it uses EV-DO cell phone technology to connect, although you'd think it would make sense to support other methods like regular WiFi. It does have a USB connection but I'm not sure if that means you can load books from your computer.

They have monthly fees for some of the services and that's a little scary. But they don't charge you (directly) for the EV-DO so that's nice.

We'll have to wait and see how much Amazon locks the device to their services and how much they open it up. If they were planning to make their money off the service then you'd expect a lower price (like cell phones). If they expect to sell it for full price AND lock you into their service that won't be too attractive.

Web Services

A few years ago when "web services" were starting to get talked about I read Web Services Essentials which covers XML-RPC, SOAP, UDI, and WSDL. I ended up writing a simple XmlRpc implementation for Suneido and we use it to distribute a particular service for our vertical application.

More recently I just finished reading RESTful Web Services (recommended). I realize now, that for our application there was no reason to use XmlRpc, that a simple RESTful web service (i.e. just using GET and POST) would have worked just as well and been much simpler. On the positive side, at least I didn't try to implement SOAP, UDI, and WSDL!

REST stands for Representational State Transfer which doesn't tell you a whole lot. Basically it is a resource oriented style using basic HTTP GET, POST, PUT, and DELETE.

It seems strange that Web Services Essentials didn't even mention the option of a REST style web service. It's only recently that REST has gained popularity, but you'd think they'd at least mention that you can just use GET and POST. Maybe they assumed that you already knew about that option, but it was all new to me at the time so I assumed the book covered the options.

Of course, if you want to communicate with an existing service you have to use whatever they supply (e.g. SOAP). But for our application we were controlled both the server and the client so we were free to use whatever we wanted.

David Heinemeier Hansson, the originator of the Rails framework for Ruby, is a fan of REST and the latest version of Rails has support for this style.

After reading RESTful Web Services I've been working on improving Suneido's HttpServer to make it easier to implement REST services. We need a new service for our application and it seems like a good opportunity to try a different style.

Bug Labs

I recently listened to a podcast with one of the people from Bug Labs. I like gadgets and this looks like a pretty nifty one.

When I was a kid I did a lot of hardware hacking, building computers and other gadgets. But I haven't done any hardware for a long time. This looks like something where you could build a "gadget" without actually getting your hands dirty.

I have an idea for a hand held gadget I'd like to make with a GPS. Most hand held GPS units aren't programmable so they won't work. Another option would be to use a cellphone with a GPS but from what I've heard software development isn't easy. And I'm not sure I want to be tied to a particular cellphone, especially if I wanted to resell these gadgets.

A "Bug" seems like a great way to develop a prototype. All I'd need for the gadget I'm thinking of would be the base unit plus the GPS module. They even plan to offer a service to convert your Bug "prototype" to a more packaged gadget that they will manufacture.

Unfortunately, it's not available yet, but the web site says 4th quarter of this year so it shouldn't be too long. And they haven't published any prices yet either.

Chumby's Coming

I found someone in the US to order my Chumby for me and it is in the mail on its way to me (hopefully the post office hasn't lost it - it's taking a while).

Dave Winer got his and is pretty positive.

More Fun With Ubuntu on Parallels

I started my Ubuntu virtual machine to get my networking file to continue looking into the network not starting automatically.

A notice came up about upgrading from 7.04 to 7.10. Without really thinking about it (stupid) I went ahead and ran the upgrade. It worked fine till it was finished and it restarted and the X display couldn't start. So I did what I should have done before I started the upgrade and I googled for problems with 7.10 on Parallels. Sure enough, lots of other people were having the same problem. There were various suggestions of how to work around the problem but there didn't seem to be a consensus. Parallels themselves seem to be avoiding the issue. I followed one of the suggestions and booted in recovery mode, which took me to a terminal window but I wasn't sure where to go from there. Then I tried booting the older kernel which seemed to start ok. Then I tried re-installing the Parallels tools (another suggestion). It ended up with the same problem of not being able to restart the display. I stopped and restarted the machine and let it boot normally and it worked!? I have no idea what that means. Is it fixed? Which part of my thrashing around was helpful? Or is the problem intermittent and I just happened to get lucky?

I'd had enough for one day so I just suspended the machine and left it.

By the way, it still has the same problem of not starting the networking automatically. I guess it was too much to hope that problem would go away by itself.

Groovy Style "builders" in Suneido

One of the neat features in Groovy is its "builders". For example, using an XML markup builder:

builder.invoices
  {
  invoice(number: 1234)
      {
      item(type: 'part')
          { product(name: 'widget', cost: 100) }
      }
  }

which produces:

<invoices>
  <invoice number="1234">
      <item type="part">
          <product name="widget" cost="100" />
      </item>
  </invoice>
</invoices>

The builder doesn't actually have methods for "invoice", "part", etc. Instead, dynamic language "tricks" are used to catch "unknown" method calls.

It made me wonder how close I could come to this with Suneido. Here's what I came up with:

builder.invoices()
  {
  .invoice(number: 1234)
      {
      .item(type: 'part')
          { .product(name: 'widget', cost: 100) }
      }
  }

The main difference is that Suneido requires '.' on method calls. Otherwise it's pretty much identical. One thing the Groovy XML builder doesn't handle is tag contents containing a mixture of text and tags. I handled this in Suneido with a special '_' method. For example:

builder.p()
  {
  ._('start ')
  .b() { 'middle' }
  ._(' end')
  }

which produces:

<p>start <b>middle</b> end</p>

Here is the entire implementation of a Suneido XmlBuilder:

class
  {
  New()
      { .s = '' }
  Default(@args)
      {
      .s $= '<' $ args[0]
      for m in args.Members(named:)
          if m isnt #block
              .s $= ' ' $ m $ '="' $ args[m] $ '"'
      if args.Member?(#block)
          {
          .s $= '>'
          result = .Eval2(args.block)
          if result.Size() is 1
              .s $= result[0]
          .s $= '</' $ args[0] $ '>'
          }
      else
          .s $= ' />'
      return
      }
  _(s)
      { .s $= s; return }
  ToString()
      { return .s }
  }

One minor problem with the Suneido version is that certain methods are "built in" (e.g. Size) and therefore can't be used in the builder. [This is a result of trying to make class instances behave the same as generic containers. I'm starting to think this was a mistake, but I'm not sure how to go about changing something so fundamental.]

I had to make to make a slight fix to Suneido to make this work. The approach I used was to use instance.Eval(function) to evaluate the blocks in the "context" of the builder. But I found that Eval didn't work with blocks (only functions). Luckily it was easy to fix. (Actually, I'm using Eval2 which returns the result inside an object so you can determine if there was a return value or not.)

PyPy, LLVM, and Parrot

I recently came across some references to PyPy - a Python version/compiler (and more) implemented in Python. Interesting stuff, but a little hard to follow.

A reference from there led me to LLVM - Low Level Virtual Machine, which is actually a compiler (including JIT) as well as a virtual machine. Check out the tutorial on implementing a language with LLVM - very slick. They discuss garbage collection (including using the Boehm collector that Suneido uses) but this area appears to still be a work in progress.

Another project along these lines is Parrot - the new Perl virtual machine.