Multi-Threading Suneido - Thread Specific Storage

Back to work on multi-threading Suneido. After frittering away my last session, I figured I'd "park downhill" and decided at the end of that session that the next thing I'd work on was replacing my home brew thread specific storage with ACE_TSS. With a clear task I wasted less time.

ACE_TSS is a template class used like: ACE_TSS<mystruct> mytss;

which is then accessed like: mytss->member

It overloads operator-> to do its stuff.

In order to keep ACE specific code as limited as possible I defined functions to access my thread specific storage. For example:

Proc*& tss_proc() { return tss->proc; }

The '&' is needed so the functions can be used as lvalues like:

tss_proc() = newvalue;

Unfortunately, this meant replacing all the instances of proc with tss_proc() but I could use the compiler to do the heavy lifting by removing proc and seeing where I got compile errors.

I could have kept the name as proc, but changing it made the transition clearer, and it doesn't hurt to make it clear in the code that it is thread-specific.

For my current, non-ACE version of Suneido I simply defined the functions to access my home brew thread specific storage.

In the process I discovered that part of the old thread specific storage code was broken. It hadn't caused problems because it was code to clear unused memory during garbage collection, so it just meant the garbage collection was a little less efficient. I fixed it for the current version, but I don't know how to implement it for real threads so it's a good thing it's not critical!

The only real problem I ran into was that the Boehm garbage collector does not appear to "see" the ACE thread specific storage. That's probably fixable but it would require digging into how ACE_TSS is implemented. And that would be platform specific. Yuck. Instead I was able to make the thread specific storage simply point to data that would be seen by the garbage collector. I'll just have to be careful I don't forget about this issue in the future.

As far as I can tell, it's all working, both the current version and the ACE version. Next I have to start tackling the crux of the problem - locking global data structures. I think I'll start with the symbol table and the global values table. These are fairly loosely coupled and shouldn't be too hard to handle. That should give me a little experience to help tackle the hard parts like the database.

PS. I started this session using NetBeans but I gave up part way through and went back to using Scite. This was partly because I haven't got NetBeans set up to build, which meant I had to compile in a separate command line window, which meant there was no easy way to click on the errors and go to that part of the source. (which I can do in Scite, although it's mostly a straight editor). I could have messed around with getting the build environment set up, but then I would have ended up like last time, with no real work accomplished.

Another thing I found disappointing in NetBeans is that it didn't seem to have the kind of powerful tools that I want/expect from an IDE. For instance, there didn't seem to be any way to find all the uses of a function or variable. Maybe I just missed it. But I wouldn't be surprised if it wasn't there, C++ tends to get second-class support in primarily Java IDE's. The Eclipse CDT used to be fairly minimal too, but I think they're supposed to have made significant improvements in the latest version. I'll have to go back and take another stab at trying to use it.

A final annoyance in NetBeans was that it didn't treat underscores as word characters. e.g. if you choose "Whole Words" and search for "proc" you find "tss_proc". I looked through the configuration options but couldn't find anywhere to change this. It seems odd because that's not C++ specific. If anyone knows how to "fix" this, let me know.

Finally!

Finally some good progress on the ACE version of the Suneido server. It's actually working well enough to run a client IDE from it, a major milestone. The last couple of problems were minor mistakes of mine. ACE and the Boehm GC seem to be working together.

Of course, this is just the start, now comes the "fun" part - actually making my code thread safe.

ACE + GC Progress

I spent a few more frustrating hours thrashing around trying to build and link with ACE statically.

Finally, I decided to start from scratch. Strangely "make clean" didn't clean up (as I discovered when a make after make clean didn't recompile!). (Note: Don't run make clean from the top level ACE_wrappers directory - it takes forever recursing into all the examples and tests.)

I rebuilt and ... it worked! Somehow I had still been getting left over shared library stuff. Another requirement is to #define ACE_AS_STATIC_LIBS before include the ACE headers.

Boy, that shouldn't have been so hard! But I can't really blame anyone but myself :-)

But ... now Suneido crashes right away on startup, which seemed more like a step backwards not forwards!

I created a small test program that used ACE and GC. It crashed the same way. Eventually, after another few hours of flailing I hit on the right combination. The key seems to be to initialize GC first, then ACE. But to achieve that, you have to prevent ACE from redefining "main" to do their startup. Here's my successful test program:

#define ACE_AS_STATIC_LIBS 1
#include "ace/Thread_Manager.h"

static ACE_THR_FUNC_RETURN thread_func(void* arg)
{
for (int i = 0; i <>
operator new(10000);
return 0;
}

extern "C" { void GC_init(); }
#undef main
int main(int argc, char**argv)
{
GC_init();
ACE::init();
ACE_Thread_Manager::instance()->spawn_n(2, thread_func);
ACE_Thread_Manager::instance()->wait();
}

At this point I'm quitting for the day. It should be easy to incorporate what I've learned into Suneido, but then I'll just run into the next problem. I'd rather end the day on a positive note!

Two Steps Forward, One Step Back

Or should that be One Step Forward, Two Steps back?

Yesterday I was back to working on the multi-threaded ACE Suneido server. It started off really well. I got the server actually running, could connect and disconnect, and make simple requests and get responses. A very good milestone.

The obvious next step is to try to start up a Suneido client IDE using the new server. This is a big jump because even to start up requires a large number of requests of different kinds. I didn't really expect it to work, but programmers are eternal optimists :-)

I got a variety of weird errors and memory faults - not surprising. But amongst these errors was one that said something like "GC collecting from unknown thread". Oh yeah, I knew this was going to be an issue but I'd pushed it to the back of my mind. The garbage collector needs to know about all the threads in order to take their stacks and registers etc. into account. The way the Boehm GC normally does this is by requiring you to call their create thread function which is a wrapper around the OS one.

The problem is, ACE is creating the threads. I found where ACE calls create thread and thankfully there was a single point I could modify to call the GC version. But, I was using ACE via a DLL which means it can't call functions in the main program (where the GC one is).

The obvious solution is to not use a DLL, to statically link in the ACE code. Sounds easy. I even found the option "static_libs=1" that would build static libraries. But it doesn't work. It builds a static library alright, but when I try to link it into Suneido I get a bunch of "multiple definitions" and "undefined references". Suspiciously, many of the names were "_imp_..." which seems a lot like the way DLL's work. My guess would be that "static_libs=1" isn't working fully, which isn't too surprising given that the "normal" usage is with shared libraries (DLL). In software, "taking the path less traveled" is often a recipe for frustration.

I started digging into the maze of headers, configs, makefiles, and ifdefs but I ran out of time. Presumably it's solvable. You can see why people like to use things like .Net or Java where at least some of these low level issues are handled for you.

At the same time as I was working on this, I downloaded Ubuntu 7.10 and created a new Parallels VM (on my MacBook while I was working on my main machine). I used the alternate cd with the text based installer as recommended by other people. It went quite smoothly (except for crashing OS X the first time I started the install), and no display problems. But when I tried to install the Parallels Tools, the disk image appeared "corrupted" - only a single file and its name was random garbage characters. I tried rebooting the VM, restarting Parallels, and rebooting the MacBook but it didn't help. I searched on the web but didn't find any references to this problem. I have upgraded my MacBook to Leopard (the new version of OS X) so the problem may be related to that. When I get time I'll try running this VM on my Mac Mini which hasn't been upgraded to Leopard yet.

Slow Progress on Multi-Threading Suneido

Last time I worked on this I had just reached the point where I could successfully compile and link. I resisted the temptation to try running it because I figured there were bound to be problems.

Sure enough, I run it ... and it crashes with a memory fault. No big surprise. I put in a few print's to see where it's getting to. Hmmm ... it's not even getting to my main!?

Download and install MinGW GDB and use it to find it's crashing in the garbage collector code. I am redefining operator new to use the garbage collector so presumably something is calling operator new during static initialization. I comment out my redefinition and it works. I make a tiny test program:

#include "include/gc.h"

void* stat = GC_malloc(10);

int main(int argc, char** argv)
{
return 0;
}

It crashes. But this works in the current version of Suneido. It must be thread related. Yup, my test program runs fine without threads. Now what?

I search the web to see if this is a known problem but I don't find anything.

I'm still using gc-6.5 and the notes for the latest gc-7.0 mention improvements to win32 threads.

So I download gc-7.0 The readme says MinGW is not well tested/supported - ugh. For gc-6.5 I had ended up writing my own makefile but I'd prefer to avoid that. The recommended approach seems to be the standard configure & make so I try this with MinGW MSys.

configure seems to succeed, at least with no obvious errors, but make fails with a bunch of "undefined references". It appears to be trying to make a dll which I don't want - I want a static library. Eventually I hit on configure --enable-shared=0 which avoids the dll stuff but still gives a bunch of "undefined references". This time they all appear to be from win32_threads.c For some reason this isn't getting included in the build. I uncomment am__objects_3 = win32_threads.lo in the generated Makefile to fix this. That's probably not the correct solution but it does the trick and I finally get the build to succeed. gctest runs successfully although it seems slower and in the output the heap is twice as big as with gc-6.5 - not good, but I'll worry about it later!

Thankfully this effort wasn't wasted and my test program runs successfully. And Suneido now manages to get to main! But then it fails with ACE errors saying WSA Startup isn't initialized. This is easily fixed by adding ACE::Init but it's strange because I didn't need it in my previous ACE test program.

After most of a day's work I'm finally back to where I can start debugging my own code! It's great to be able to leverage other people's work (like the Boehm GC and ACE) but it can be extremely frustrating when they don't work and you don't have a clue what's going on. Even the standard configure & make has the same problem. If it works it's great, but if it doesn't you're faced with incomprehensible makefile's.