Wednesday, May 14, 2008

jSuneido - implementing classes & more

So far, so good. I've written about 850 lines of Java that equates to about 2000 lines of C++ in two and a bit days. I've been porting the basic Suneido data types.

I'm lucky that Java is pretty close to C++, and Suneido's byte code is pretty close to Java byte code.

I'm gradually getting an idea of how to implement Suneido on top of Java and the JVM. Here's how I envision Suneido classes being compiled:
class SuClass { // ultimate base class - static, not generated
public SuValue invoke(int method, SuValue ... args) {
return invoke2(method, args);
}
public SuValue invoke2(int method, SuValue[] args) {
if (method == DEFAULT)
throw SuException("method not found");
return invoke2(DEFAULT, args);
}
}

class xxx extends SuClass { // generated
public SuValue invoke2(int method, SuValue[] args) {
switch (method) {
case 8346:
return mymethod(massage(6, args, 4746, 9846, 3836));
...
default:
return super.invoke2(method, args);
}
}
public SuValue mymethod(SuValue[] locals) {
if (locals[2] == null) locals[2] = FALSE;
invoke(8347, locals[1]); // call a method in "this"
locals[4] = globals[123].invoke(7472, locals[0], locals[2]);
locals[3] = new SuValue(
JavaClass.a_static_method(locals[1].string()));
}
...
}
Note: I'm showing this as Java source code, but I plan to compile directly to Java byte code.

The explanation:
  • int method is an index into the Suneido symbol table - since it's faster to dispatch on int instead of string (and it also makes the compiled code smaller)
  • a variable number of arguments are received into the args array
  • a switch is used to dispatch to the correct method, in this example 8346 is the symbol index of "mymethod"
  • Suneido's extra argument passing features are implemented by "massage" which also allocates an array for the local variables of the method, the "6" is the size of this array, the remaining (variable number) arguments are symbol indexes for the methods parameter names (required to handle named/keyword arguments)
  • if the method is not found in the current class, invoke2 is called on the parent class, if this ends up at the ultimate base class (SuClass) then Suneido's Default method-not-found will be called, if this also ends up back at SuClass then an exception is thrown
  • methods receive a "locals" array containing the arguments in the first part of the array
  • default argument values are compiled to code, in this example the default value of the third argument is false
  • classes and functions are stored in the globals array
  • blocks are compiled into separate methods, context is passed via the locals array (this is one of the reasons for using the locals array instead of native Java local variables)
  • Java classes can be called directly, with the appropriate conversions
This should be relatively fast - method dispatch is just a couple of extra function calls and a switch - no reflection or table lookup. Of course, methods deep inside class hierarchies will take longer due to the "chaining".

So far I'm using unchecked exceptions. Partly because it's simpler and partly because that's what I'm used to from C++. The traditional advice (e.g. from Sun) is to use checked exceptions, but this is starting to be questioned.

I'm using the standard Java indent/curly style. It hasn't been a problem but it's a bit of a transition after 25 years of C and C++ (and Suneido) using the Whitesmiths style.

I did figure out how to configure the Home and End keys to be beginning and end of line in Eclipse. That's one thing I miss on the Mac. The "standard" appears to be Apple + left/right arrow, but I find that more awkward, it doesn't appear to be supported everywhere, and those keys are sometimes used for other purposes like switching OS X virtual desktops.

I notice that C++0x is proposing another approach to the problem I talked about in my last post. I suggested replacing:
Type var = new Type(...);
with:
Type var = new(...);
For C++0x they are proposing:
auto var = new Type(...);
This is better than my suggestion because it also allows things like:
auto var = func(...);
where the type of var is taken from the type of the return value of func.

Tuesday, May 13, 2008

jSuneido - good news, bad news

I just had a depressing thought. Suneido consists of about 60,000 lines of C++ code. Yesterday I wrote 200 lines of Java. Even if the Java is half the size of the C++ (?) that's 150 straight days of programming. I can probably do better than 200 lines once I get more up to speed on Java. But some of the code is going to be a lot harder to port than what I did yesterday. It's obviously do-able, but it's not going to be "quick" even if it's "easy" (which it probably won't be).

I'm starting to find some resources e.g. the golden spike by John Rose @ Sun and the JVM Languages Google Group. (there's some interesting stuff if you go back to the beginning of the group) It's good to know there's information out there (although these days that's almost a given) but the complexity of some of it is a little daunting.

Another C++ feature that doesn't port straight across is operator overloading. Not a big deal other than recognizing when it's used in the C++ code.

A Java annoyance - the duplication in statements like this bugs me:

HashMap hashmap = new HashMap();

C++ is similar but you notice it less because you can allocate on the stack with just:

HashMap hashmap;

And you can use typedef to make abbreviations so you can write:

Map hashmap = new Map;

Hah! Two days programming in Java and I'm already redesigning the language! Here's my proposed shortcut (syntactic sugar):

HashMap hashmap = new; // or new(args...)

On the positive side, writing a multi-threaded socket server seems a lot simpler in Java than in C++. It didn't hurt that one of the books I found at the bookstore was TCP/IP Sockets in Java.

Monday, May 12, 2008

More on jSuneido

When I wrote the jSuneido post a few days ago it was just a vague idea. But since then it's been buzzing around in my head and it seems more and more like something that should at least be investigated.

I think there are good "business" reasons for doing it, but I have to admit it's the technical challenge that excites me as much as anything!

Of course, the overwhelming temptation is to use this opportunity to "fix" parts of Suneido that in hindsight seem less than ideal. But if it's only the server part of Suneido, then it will have to stay compatible with the existing client, so that should stop me from getting too carried away!

I went through my library for Java books but they're all fairly out of date so Friday I ordered some new ones from Amazon and Chapters, including a couple on the JVM. I was out of town for the weekend but Sunday on the way home I stopped at the local bookstore and picked up a few more.

A lot of my questions revolve around how to "map" Suneido onto Java and the JVM. Luckily, Suneido is not as "dynamic" as some dynamic languages. For example, it doesn't allow adding/removing methods on classes on the fly as Ruby does. (This was a deliberate choice - Suneido's predecessor had this ability and I felt it led to more harm than good.)

Here are some highpoints:
  • Suneido exception => Java exception
  • Suneido class => Java class
  • Suneido function => Java class with static method
  • Suneido number => Java BigDecimal
  • Suneido instance members => Java HashMap
  • Suneido method call => custom method lookup (until Java invokedynamic is released)
Suneido has more argument passing options e.g. keyword parameters so this will have to be handled on top of the Java argument passing.

Suneido blocks (closures) will be one of the harder parts. The code itself can be private methods, but the challenge is to provide access to the local variables of the defining context.

I'm not sure how to implement calling Java code from Suneido code. But judging from Groovy, I assume it's do-able, presumably using Java reflection. For now, I'm not too worried about the reverse (calling Suneido code from Java).

I'm not sure whether I'll port my own regular expression code or use Java's. It depends on how compatible they are.

Another issue is how to "map" (port) my C++ code to Java. Things like reference variables and templates make this tricky in spots.

Sunday evening I installed Eclipse on my MacBook and started to get my feet wet. I have to say that starting a new project, and in Eclipse's "native" language, I had a much more positive experience than when I tried to use it for C++ recently. Of course, I fumbled around a bit as you would expect with an unfamiliar tool but overall it went pretty smoothly and I'm enjoying the Eclipse features.

I chose to start by implementing the Suneido data types. I've written about 200 lines of Java code (as reported by the Eclipse Metrics plugin) including jUnit tests. So far it's been a straightforward translation of the C++ code to Java.

Already the portability is nice, I was able to work in Mac OS X at home and then copy the project directly to Windows at work and continue working.

It's hard to say how big a project this is. Sometimes it doesn't seem too bad, other times it seems like a huge job. Time will tell I guess.

Friday, May 09, 2008

New Trick for SciTE

I just discovered that SciTE, the programming text editor that I use, has an autocomplete feature that I never knew about.

You turn it on by adding to the properties file:

autocompleteword.automatic=1

From then on, if you type a prefix of a word that has only one match within the current file it will suggest that word and you can hit Tab to accept it. This is really handy for long names.

I've seen this feature in other editors but didn't realize it was available in SciTE. (I was aware that there was a facility for setting up API's but this is simpler and more generic.)

Note: SciTE is based on Scintilla which is the code editing component used by Suneido.

Thursday, May 08, 2008

Suneido Broken & Fixed

When I was making the changes to handle thread specific storage on ACE I encountered some "ugly" code.

The first thing was a lot of instances of:
proc->stack.top()
which got even worse when they became:
tss_proc()->stack.top()
(and similar for push, pop, etc.)

In interp.cpp I had defined inline functions for these, but I hadn't extended this to elsewhere. I moved them to interp.h and converted all the instances to:
TOP()
While doing this I noticed code like:
Value* sp = GETSP();
...
SETSP(sp);
This seemed like a perfect application for an automatic destructor. So I added this to interp.h:
struct KeepSp
{
KeepSp()
{ sp = GETSP(); }
~KeepSp()
{ SETSP(sp); }
Value* sp;
};
#define KEEPSP KeepSp keep_sp;
Using this, the code became simply:
KEEPSP
...
This was even better where I'd had:
Value* sp = GETSP();
Value result = ...
SETSP(sp);
return result;
because it could now be simplified to:
KEEPSP
return ...
The next thing that caught my eye was code like:
try
{
...
x->close();
}
catch
{
x->close();
throw ;
}
Another good application of automatic destructors. So I added this to std.h
template<class T> struct Closer
{
Closer(T t) : x(t)
{ }
~Closer()
{ x->close(); }
T x;
};
This let me rewrite the code to simply:
Closer<mytype> closer(x);
...
(without any try-catch)

I was feeling pretty good about this refactoring, until I found that one of my tests was broken. I reviewed all my changes to see if I'd made any inadvertant typos or other mistakes. It all looked good. I started tediously reversing my changes one at a time until I found where I'd broken the test. (Theoretically I should have compiled and run the tests after each little change, but unfortunately that takes a little too long to be tolerable.)

I found the spot, but it looked fine. It almost seemed like the KeepSp destructor wasn't getting run. At one point it seemed like it worked on Visual C++ so I started looking for bugs related to destructors with GCC 3.4.2 But I'd been mistaken, it was actually broken on Visual C++ as well.

I tried using GDB but it didn't really help. In the end I added some "prints" and that gave me the clue that there was an exception being thrown, but the problem wasn't that the destructor wasn't being run - the problem was that the destructor was being run. The old way the code had been written, the stack pointer wasn't being restored if there was an exception, now it was.

Why would that cause a problem? Looking at the interp.cpp code I saw a key comment - returns from a block (that actually return from the containing function) are implemented as exceptions and pass the return value on the stack. My new code that restored the stack pointer even when there was an exception was losing the block return value.

Hmmm... now what? Do I put the code back to the way it was? But it seems kind of ugly to depend on cleanup code not getting run when there's an exception. Instead I changed block returns to store the value in the proc rather than on the stack. Now all the tests pass.

Thank goodness for tests. Only one test (out of hundreds) failed. Without that test it could have been quite a while till I ran into this bug, and it would have been a lot harder to track down.

You could argue that if I'd just left well enough alone I wouldn't have run into the problem in the first place. After all, if it's not broken, don't fix it. I don't agree. Down that path lies chaos. You have to fight entropy. Otherwise each successive change makes the code slightly worse, until eventually it's too incomprehensible and fragile to dare touch.

The day was a bit of a roller coaster. I start off feeling pretty good about the cleanups. Then I got frustrated when I couldn't figure out why it was broken. Then when I thought I'd have to undo my changes I got depressed. I know, rationally it's pretty silly to get depressed about something like that. But I guess if I didn't care enough for it to be depressing, then I would never have gotten this far in the first place. Still, you'd think after 30 plus years of programming I'd be a little less emotional about it!

At this point I decided to take a break and go for a run. That usually improves my mood. And while I was running I figured out that I could make it work with the improvements. So between the endorphins and not having to undo my changes, suddenly I was in a good mood again. That's probably a good place to leave things for the day!

OpenOffice 3 beta

OpenOffice 3 beta is available. The big feature for me was that it finally runs native on Mac OS X (instead of via X windows) but there are a bunch of other features.

I've been using NeoOffice to get a Mac version, but it's nice that this is now standard. (Although it leaves the NeoOffice project with less of a reason for existence.)

Wednesday, May 07, 2008

jSuneido ?

As I work on multi-threading Suneido I wonder if I shouldn't be shifting direction.

Does it really make sense to implement your own VM these days? Wouldn't it make more sense to re-implement Suneido on top of an existing VM such as Java?

For example, jRuby (Ruby re-implemented on the Java VM) in some cases outperforms the original C version of Ruby.

And Groovy also demonstrates that it is possible to implement a highly dynamic language on top of the Java VM.

The JVM would handle low level issues like garbage collection and JIT and multi-threading and 64 bit. (Although you'd still have to deal with higher level multi-threading issues.)

It could provide access to Java libraries so we didn't have to continually "re-invent the wheel".

It would not solve the current issue of a non-portable Windows UI. As with my current project, it could be server only to start with.

It would allow portability to everywhere that the JVM is available.

I'd have to not only learn Java, but also learn JVM issues that a normal Java programmer doesn't have to worry about. That's a little intimidating.

Of course, the other big question is why not just use an existing language/database? The big reason now, is that we have a big (to us) application written with Suneido's language and database. The thought of re-writing or porting it all to another platform is way too scary.

Moving to something like the JVM that allowed interoperability with other JVM based languages (and other databases) might actually allow us to incrementally move to another language if we chose to.

Obviously, this wouldn't be a minor decision, but it is food for thought.

More Concurrency

While I was waiting for the iMac transfer I was working on my project to multi-thread Suneido. The next piece I decided to tackle are the global hash tables for symbols and global names.

One obvious solution is to simply use many-readers/single-writer locking.

That still leaves the question of how to implement it. Do I mix the locking into the existing code? Or do I write a "wrapper" class that handles the locking and then delegates to the existing class. Neither is ideal - mixing in the locking will complicate the code but a wrapper will add overhead (although probably minimal).

But I'm not completely happy with adding locking overhead to readers. Especially for these tables. Once the server is "warmed up" these tables will almost never be written to. It would be really nice to avoid any overhead for readers.

My next thought was lock-free hash tables. I've heard various noises about this so I thought it was worth seeing what was available. As far as I could find, the answer was: not much.

The Intel Threading Building Blocks have some concurrent C++ data structures. But Boost doesn't. And STL doesn't (yet).

The most interesting thing I came across was Cliff Click's work. If you've got an hour to spare, check out the video of him giving a presentation about it. I see from his blog that he's extended his work to other data structures. Apart from the impressive benchmarks, what I really like about his approach is that it's simple and relatively understandable. (Although I still have a hard time wrapping my head around re-ordering of memory accesses and the required "fencing".)

Unfortunately, his work is all in Java. The only mention of C++ is someone commenting on the blog that they'd like a C++ version. Part of me is really tempted to implement it. It seems like it wouldn't be that hard. Famous last words.

Side note: I ran across some references to the advantages of power-of-two size + AND hash tables versus prime size + MOD (more traditional and what I'm using). One of the reasons is that MOD is slow (relative to AND). But I wonder how much that really affects overall speed.

Another thought I had was that I could just give each thread their own tables. The downside is the extra memory and redundant work maintaining the tables. This approach is closer to using separate processes instead of threads.

That led me to consider two copies - one for reading and one for writing. The reading one would be lock free. The write copy would be protected by a mutex. To write you would lock the write copy, update it, swap the two copies using an atomic compare-and-swap, and then update the other copy to keep in sync. This seems simple to implement, and simple enough to be reasonably sure it's "correct". It uses double the memory, but only for the hash table of pointers, not for the data itself. (Both copies can point to the same data.)

Strangely, I haven't seen any mention of this approach. Perhaps because it's not general purpose - it would not scale to many writers. But this shouldn't matter for my purposes. Or maybe there is some flaw that I'm missing? (concurrency issues can be like that)

The writing bottleneck could slow down startup. If so, (and obviously I'd want to measure that) then it would be fairly easy to write out symbol tables from a "warmed up" server and use these during startup to initialize.

As far as the atomic compare-and-swap, that seems readily available: in Win32 it's InterlockedCompareExchange. ACE has ACE_Atomic_Op, although it doesn't appear to have compare-and-swap. As of version 4.1 GCC has built-in atomic operations.

Given all the attention on concurrency these days, it's surprising that there aren't more tools available.

Tuesday, May 06, 2008

New iMac

My new iMac arrived today. There's a bit of a story behind it's arrival.

Originally I was going to buy it through the local store (not an actual Apple store, but an Apple retailer and service center). Of course, they didn't have the model I wanted in stock. But they had some similar models coming in a few days and they could upgrade the memory and hard disk to what I wanted. Except the person that took down my phone number mumbled that he didn't think they could upgrade the hard drive, leaving me wondering whether they could or not.

A week passed and no word. Finally I phoned them to find out what was happening. "Soon", I was told. Given the uncertainty about getting the right configuration, I decided I might as well just order on-line.

But my credit card was declined!? It was paid off and I hadn't had any problems with it. I had let Apple use the credit card info on file - maybe it was out of date. I re-entered the credit card information. It was declined again. Finally I phoned the credit card company - they had declined it because they (or probably their software) thought it was "suspicious". I'm not sure why. I've purchased from Apple before. Sure, it was a larger amount but surely that doesn't mean it automatically gets declined? And couldn't they have contacted me? Oh well, no point expecting rational behavior from something like a bank.

One the positive side, while I was messing around with the credit card I got an email from Apple saying they had come out with upgraded models and since my order hadn't been shipped I'd be getting the new model. Lucky - I'd have been really annoyed if I got the old model just a few days before the new one came out.

I got a 24" (1920 x 1200) iMac with 3 ghz Core 2 Duo with 6 mb L2, 4 gb of memory and a 1 tb hard disk. A decent improvement over my 2.2 ghz Mac mini with 2 gb of memory and 160 gb disk

I pulled it out of the box, plugged it into the power and network, put batteries in the mouse and keyboard, and fired it up. I hesitated when the migration tool wanted Firewire, but I hadn't set the machines up side by side. I decided to use the secondary Ethernet choice. (wouldn't that be clearer if they called it "Network"?) This required installing CD and DVD Sharing software on the old machine.

Soon after it started it came up with an error about losing its connection. But when I re-tried it seemed fine. Not sure what caused it. It started off saying it was going to take 10 hours, but that estimate soon dropped and it ended up taking a couple of hours to transfer about 110 gb. That seemed a little slow. I searched on the web and found some people saying Firewire was faster. I wondered about starting over but the migration didn't even have a cancel button and I was a little nervous about "pulling the plug" while it was transferring so I let it go. The nice thing about having multiple machines is that when one is tied up you can still work on another (the MacBook).

The transfer finished and at first it seemed like everything was great. But then I started running into problems. Some of the System Preferences wouldn't open. iTunes refused to start (with an error 1000). Hmmm... I searched on the internet for migration problems. One person recommended running Software Update on the new machine before migrating (so you didn't get newer settings on older software) But unfortunately, it was a little too late for that.

After a few tries and a few system restarts (reminiscent of Windows!) I got Software Update to run and that seemed to clean up the problems.

I calibrated the monitor with my Huey Pro. This seemed to make a bigger difference than on my last monitor, making the colors quite a bit "warmer" than the default.

Then I noticed that I now had a Huey icon on the menu bar. Strange, that hadn't shown up on my old machine. Then after I restarted the machine it was gone. I got on the Huey web site to see if there were any software updates. In their support section I found that the automatic startup was broken and you have to manually install. Which means it hadn't been running on my old machine. Ouch. I'm not sure if this affected the entire calibration or just the ambient light adjustment. They hope to fix their software in 2008. I wonder how many other people have this problem?

Next problem was that it had installed my Epson R1800 printer with the Gutenprint drivers. This works ok for simple printing, but not so good with printing photographs from Lightroom. I downloaded the drivers from Epson and installed them. I thought this would create the printer but it didn't, just installed the drivers. So I added the printer and said "yes" I really wanted to install this printer a second time. But when I printed a test page, nothing came out. Here we go again.

I poked around and happened to right click on the printer. One of the options was "Reset printing system". That seemed like it might help so I chose it. I got a warning saying this would delete all my printer queues and pending jobs. That seemed alright so I clicked on OK. Oops! That deleted all the printers. I guess that's what they meant by "printer queues". I re-install the printer and run a test page again. Still no errors, but nothing comes out. I notice that it appears to be printing a Postscript file. But I don't think the Epson driver uses Postscript. I try printing from another program and it works fine. It was probably working all along. Sigh. Not a very good test page. I guess it's intended for Gutenprint drivers, but why use it for other drivers?

I also had problems getting my Parallels virtual machines to work. I thought I might have to re-install the software, but after hanging up a few times it decided to start working. Except that my virtual machines had lost their network adapter. But all I had to do was re-enable it.

Of course, I'd lost the twisted printer setup that let me print from Windows with the Epson driver. But I noticed I now had Apple Bonjour on Windows. I can't remember why I installed it, but I figured it was worth a try. My first attempt failed but after enabling Printer Sharing on OS X and sharing the printer I got it to work. Even the test page printed :-) Much easier than the previous setup.

I order the iMac with the new wireless keyboard. It's tiny! Especially next to the 24" iMac. It's taking a little getting used to but the feel is nice. I won't miss the numeric keypad, but I'll miss the larger arrow keys and Home/End/Page Up/Page Down/forward Delete etc. But it's essentially identical to my MacBook keyboard so it's not too bad.

And that pretty much took up my whole day. Still, probably the most painless computer switch I've done.

Monday, May 05, 2008

Old Lessons

One of the common suggestions for testing is to test around limits or boundaries. e.g. try 0 or 1 or -1. I was painfully reminded of this by a recent debugging session.

Some background. Originally Suneido memory mapped the entire database file into memory. That was simple and fast but limited the size of the database to less than 2 gb (the amount of address space a Win32 process gets).

When our customers' databases started to approach this size I had to change Suneido to memory map a limited number of segments of the database file and re-use address space in an LRU fashion.

Considering how major this change was it went amazingly well. I was quite nervous about weird bugs from things like dangling pointers caused by re-mapping. So I did quite a lot of testing before we rolled it out. And we kept a close eye on it at first.

This was quite a while ago and I figured everything was good. Until our client with the largest database started having problems. Their database was right around 4 gb. So I did some testing and sure enough when the database reached 4 gb MapViewOfFile would fail with "Access Denied". The "Access Denied" part threw me off since it sounded like some kind of security/permissions issue. I dug through MSDN trying to find some mention of a 4 gb limit but couldn't find anything.

As with so many of these things, the error turned out to be really stupid. Memory mapping a file with Win32 is a two step process, first CreateFileMapping, then MapViewOfFile. Since it was MapViewOfFile failing I hadn't paid much attention to the CreateFileMapping step.

Both API calls accept 64 bit file offsets as two 32 bit values. I was only passing the low 32 bit value to CreateFileMapping. Stupid! Of course, it worked up to 4 gb. But over 4 gb it wasn't passing the correct offset. Whereas I was passing both halves of the offset to MapViewOfFile. It was trying to map the correct offset, but was failing because the file mapping was for a different, incorrect offset. (A little confusing because the call with the wrong arguments was succeeding and the call with the right arguments was failing.)

Why hadn't I caught this with my testing? Because the old database size limit was under 2 gb so I tested with over 2 gb. The bigger the database the slower the testing so I never bothered going as big as 4 gb. Oops.

I should have realized that the limit for 32 bit values would be an obvious failure point and I should have tested up to and over 4 gb.

After I fixed this bug I was able to grow a database past 4 gb. For a minute I thought I was done. But a little more testing revealed that it wasn't accessing the data past 4 gb properly.

This was a little tougher to track down. I got fed up with waiting so long to create big databases and came up with a way to quickly expand the temporary databases used by the tests to over 4 gb. (I could have saved considerable time in the long run if I'd done this at the start!)

The problem turned out to be a place in the code where I mistakenly stored a 64 bit file offset in a 32 bit integer. Again, this worked up to 4 gb but not beyond.

After this fix all the tests ran successfully.

Next time maybe I'll remember to test around obvious limits and boundaries.

PS. Not that it's any excuse, but it seems odd that the compiler wouldn't give a warning about assigning a 64 bit value to a 32 bit variable. I'm using -Wall with GCC and the only warning I have turned off is -Wno-sign-compare. Maybe there are warnings I need to turn on that aren't included in -Wall. I should probably look into this.

PPS. Along the way I got an excellent demonstration of the effects of locality on multi-level storage. If I created a big database by doing: for each table, add 10000 records, then it was reasonably fast. If I turned it around to: 10000 times, for each table, add 1 record, then it was about 10 times slower. Dealing with one table at a time kept all the data and index nodes for that table together so they could all be mapped into memory. Looping through each table and adding one record to each causes the data and index nodes for all the tables to be mixed together over too large an area to be mapped all at once, causing continuous un-mapping and re-mapping.

Saturday, April 26, 2008

The Best UI is No UI

The other day I watched someone unlock their car with a remote key. No big deal, a common occurrence. But for some reason I noticed it and thought that's cool. Maybe because I never had a remote key. I went from a regular, mechanical, non-electronic key, to a Prius with a "Smartkey" system.

With the Prius, I don't need to touch the key. I walk up to the locked car and open the door. As long as I have my key in my pocket or backpack it's all automatic. In other words, no user interface. (I'm now spoiled - I resent having to pull my office key-card out of my pocket, and even worse to have to use a mechanical key on my house.)

Too often in software (and other areas) we think the ultimate "solution" to a problem is some clever UI, like a remote key. But people don't want a clever UI, they want to get some job done, and often the less UI the better - none if possible.

Similarly, the best solution for errors is to prevent them in the first place. (again, no UI) You can't lock your key in the Prius because it won't let you lock the car if the key is inside.

Thursday, April 24, 2008

Multi-Threading Suneido - Thread Specific Storage

Back to work on multi-threading Suneido. After frittering away my last session, I figured I'd "park downhill" and decided at the end of that session that the next thing I'd work on was replacing my home brew thread specific storage with ACE_TSS. With a clear task I wasted less time.

ACE_TSS is a template class used like: ACE_TSS<mystruct> mytss;

which is then accessed like: mytss->member

It overloads operator-> to do its stuff.

In order to keep ACE specific code as limited as possible I defined functions to access my thread specific storage. For example:

Proc*& tss_proc() { return tss->proc; }

The '&' is needed so the functions can be used as lvalues like:

tss_proc() = newvalue;

Unfortunately, this meant replacing all the instances of proc with tss_proc() but I could use the compiler to do the heavy lifting by removing proc and seeing where I got compile errors.

I could have kept the name as proc, but changing it made the transition clearer, and it doesn't hurt to make it clear in the code that it is thread-specific.

For my current, non-ACE version of Suneido I simply defined the functions to access my home brew thread specific storage.

In the process I discovered that part of the old thread specific storage code was broken. It hadn't caused problems because it was code to clear unused memory during garbage collection, so it just meant the garbage collection was a little less efficient. I fixed it for the current version, but I don't know how to implement it for real threads so it's a good thing it's not critical!

The only real problem I ran into was that the Boehm garbage collector does not appear to "see" the ACE thread specific storage. That's probably fixable but it would require digging into how ACE_TSS is implemented. And that would be platform specific. Yuck. Instead I was able to make the thread specific storage simply point to data that would be seen by the garbage collector. I'll just have to be careful I don't forget about this issue in the future.

As far as I can tell, it's all working, both the current version and the ACE version. Next I have to start tackling the crux of the problem - locking global data structures. I think I'll start with the symbol table and the global values table. These are fairly loosely coupled and shouldn't be too hard to handle. That should give me a little experience to help tackle the hard parts like the database.

PS. I started this session using NetBeans but I gave up part way through and went back to using Scite. This was partly because I haven't got NetBeans set up to build, which meant I had to compile in a separate command line window, which meant there was no easy way to click on the errors and go to that part of the source. (which I can do in Scite, although it's mostly a straight editor). I could have messed around with getting the build environment set up, but then I would have ended up like last time, with no real work accomplished.

Another thing I found disappointing in NetBeans is that it didn't seem to have the kind of powerful tools that I want/expect from an IDE. For instance, there didn't seem to be any way to find all the uses of a function or variable. Maybe I just missed it. But I wouldn't be surprised if it wasn't there, C++ tends to get second-class support in primarily Java IDE's. The Eclipse CDT used to be fairly minimal too, but I think they're supposed to have made significant improvements in the latest version. I'll have to go back and take another stab at trying to use it.

A final annoyance in NetBeans was that it didn't treat underscores as word characters. e.g. if you choose "Whole Words" and search for "proc" you find "tss_proc". I looked through the configuration options but couldn't find anywhere to change this. It seems odd because that's not C++ specific. If anyone knows how to "fix" this, let me know.

Firefox 3 Beta

I've been holding off on trying the Firefox 3 beta. I used to be an earlier adopter but now that so much of my computing life goes through my Firefox browser I'm a little more reluctant to live on the bleeding edge.

But beta 5 is supposed to be the final beta before the release candidate so I thought I'd give it a try - as long as I could install it alongside my current Firefox. On Windows I used Portable Firefox but I couldn't find anything similar for the Mac. Instead I created a second profile and have Firefox prompt for the profile when it starts up. This works but is a bit of a hassle and I'm afraid I'll choose the wrong profile at some point.

A pleasant surprise on the Mac is that it finally has a more Mac-like theme. e.g. buttons on web pages are rounded instead of ugly rectangles.

Unfortunately, it still doesn't show the favicons on the bookmark bar. I guess this is to be compatible with Safari, but that seems pretty silly to me. But I found the Mac bookmark toolbar favicons style for Stylish which solved this issue.

If you asked me if I needed a customizable browser my first response would be "no". But when I go back to a "bare" browser I realize how much I depend on the add-ons I use. Of course, not all the add-ons have Firefox 3 versions yet. dragdropupload (which I use all the time for email attachments) was available.

But the Google Firefox Browser Sync isn't, which was a real hassle because it meant I didn't have any of my bookmarks or passwords. I don't have many bookmarks since I keep the bulk of them on del.icio.us so it was easy to export from Firefox 2 and import into Firefox 3 (once on Windows and once on Mac). I haven't found a way to export and import my passwords.

Passwords was another pleasant change in Firefox 3. I have a hard time remembering all my passwords for all the web sites I use, especially since I try not to use the same password for all of them. On Firefox 2 after you enter a password a dialog pops up asking if you want to remember it. But at this point you don't know if it's right or not. Either you say "remember" and then if it's wrong end up storing a bad password. Or you say "don't remember", in which case, if it works, if you want to remember it, you have to log out and log back in. Either way it's not ideal. But in Firefox 3, the "remember" prompt is inserted in a bar at the top of the screen mode-lessly - meaning you can wait to see if the password is accepted before choosing to remember it or not. Great improvement.

The Delicious Bookmarks add-on for Firefox 3 is a beta version that requires you to join the Yahoo group before you can download it. No big deal, but more hassle than necessary.

Firefox 3 is supposed to be a lot faster than Firefox 2. I can't say I've really noticed a big difference, but I haven't timed anything.

I haven't had any real problems with the beta so far. I crashed once in a couple of days, which isn't any worse than Firefox 2. I'll continue testing for a bit longer, but it seems good enough to switch over.

Tuesday, April 22, 2008

Wireless Security

Sometimes it seems like I go out of my way to make things hard for myself!

I got home from work to find a note from Shelley that her laptop wireless wasn't working. Oh yeah, I hadn't got it working with the new Time Capsule.

I run the Airport Utility, change the Wireless Security to WEP, and try to set the password to what I was using previously (so I didn't have to mess with the laptops or other devices).

Nope, it will only take an exactly 13 character password. Okay, I make up a new password and go to connect Shelley's Windows laptop. But it won't work. I mess around with the settings, stupidly failing to record the original settings. No luck.

I search on the internet. Some people say that on Windows you have to use the hex version of the password (you get that from the AirPort Utility from Equivalent Network Password on the Base Station menu). I type in a 26 character hex password multiple times. No luck.

More digging reveals that this is 128 bit WEP. (More or less. It's also Transitional Network Security that is also WPA compatible. Whatever that means.) The older 40 bit WEP that I was using previously is actually available - you just have to turn off the new 802.11n :-( and then hold down the option key while you pull down the Radio Mode list. (how could I have missed that!)

Now I can enter the old 10 digit hex password (with a dollar sign). And now the Windows laptop connects.

I pull out the Chumby to make sure it can still connect. It can, but in the process I find that it does handle WPA! So I don't need to use WEP after all. Argh! (It must have been my old Squeezebox audio player that was WEP only, but I'm not using it any more - too much hassle to run its proprietary server.)

Back to the Airport Utility, re-enable 802.11n, set it back to the default WPA. Back to the Windows laptop, enter the new password, it connects fine. Back to the Chumby, enter the new password, it works fine. Same with my MacBook.

So I think I'm all good. I'm glad my Mac mini is hard wired to the Time Capsule. Trying to configure wireless through a wireless connection would have added even more fun to the process!

Monday, April 21, 2008

New Time Capsule

I just replaced my Linksys wireless router and Lacie network drive with a 1 tb Apple Time Capsule. My reasons for the change included:

- The Linksys router periodically "dies". Even my wife knows to unplug it, wait a few seconds, and plug it back in, but it's still a hassle. Since this fixes the problem I'm assuming it's the router, but I suppose it's possible it's something like the ISP DHCP. I'm crossing my fingers that a new router will fix the problem.

- The Lacie network drive was getting full.

- I was using a second Firewire Lacie drive for my Time Machine backups from my Mac mini. This drive was also running out of space.

- I wanted to make Time Machine backups of my MacBook, and the only way to do that without plugging in an external drive is with Time Capsule. (Time Machine won't work to a regular network drive. I'm not sure if that's just a marketing decision or whether there is some technical reason as well.)

- I could replace two boxes (router & network drive) with one - the less cables the better, if you ask me.

- I occasionally had range problems with wireless. I'm hoping the new router will help.

As usual, the packaging was slick and the hardware is attractive. I plugged it in, installed the software, and it worked.

At first I couldn't see the drive from the Windows machine but after I set the right workgroup it appeared.

The setup wizard only allowed WPA wireless security. I was still using WEP since I have some wireless devices that don't support WPA (like my Chumby). I realize WEP is minimal security but it's enough to stop my neighbors from accidentally using my network. I have used MAC address filtering in addition but it's a hassle when you add new devices and I always seem to end up turning it off. I see the regular configuration allows WEP but I haven't got around to changing it yet.

To keep things simple (at least I assume that's the reason), OS X only seems to offer a single "key" entry field. I'm never quite sure if this is hex or text or what. If you were Apple/Mac only it wouldn't matter - it just works. But when you're connecting to other things it can be confusing. I've seen references to using a dollar sign or 0x prefix to enter hex keys but it doesn't seem to be clearly documented (that I've seen).

Once I had it set up I switched my Mac mini Time Machine to go to the Time Capsule. Of course, the first initial backup is huge and takes a long time. And there's no way to move your existing Time Machine backups to the new drive.

I probably should have know better, but while that was chugging away I turned on Time Machine on my MacBook and pointed it at the Time Capsule. Of course, it also was a huge initial backup.

At the same time (I know, asking for trouble!) I was playing with the Time Machine options to exclude certain files and directories. Then I realized every time I changed the settings the initial backup started all over again!

Just to stress it a little more, I started copying the other files (pictures and music) from the old network drive to the new one. It took me a few minutes to get the network drive working again - it wouldn't show up until I re-ran the configuration utility. This copy ran for quite a while (an hour maybe?) and then aborted with an Error 50. I suspect my impromptu stress test uncovered some bug in the Time Capsule software.

When I went to bed the two Time Machine backups were still running. When I got up, the MacBook had finished. Unfortunately, I'd forgotten I had the Mac mini set to power off at night. When I started it up, it had to restart the backup from the very beginning! I realize this is just the initial backup, but given how long it takes, you'd think they would have made the software handle resuming a backup. This time, without all the other concurrent activity, the backup went smoothly and finished.

Or maybe I should say, more or less finished. For several more hours it popped up windows about backup up large quantities of files. I'm not sure what this was - I hadn't modified or added any significant amount of files. Why didn't it get this stuff on the initial backup? But eventually it seemed to settle down. The problem with this kind of system that just invisibly does stuff in the background is that you're never quite sure if it's working properly (at least if you're a cynical techie).

Once the backups were done I went back and copied the other data from the old network drive. Again, now that I wasn't trying to do too many things at once, it went smoothly. (Theoretically, if the software is "correct" it shouldn't matter how much stuff you do at once. But no software is "correct". As the saying goes, in theory, practice should be the same as theory, but in practice, it's not.)

So far so good. My only (minor) complaint is that, judging by the temperature of the case, the Time Capsule doesn't seem to go to "sleep" - even overnight, with no computers active (or even turned on). There's probably continuous activity on the internet side, but if nothing is awake on the LAN side, I would think it could still be smart enough to go to sleep.

One nice side benefit is that the Time Capsule hard drive is a lot quieter than the Lacie.

I know it's showing my age but I can't help continuing to be a little mind boggled by gigabytes of memory and terabytes of hard drive space. (I just listened to a podcast that said that in many cases algorithms had progress more than hardware and in these cases you'd be better off with a modern algorithm on old hardware than old algorithms on modern hardware. In theory that might be true in some cases, but modern software is never going to fit on old hardware. I can't even fit a single digital picture on a floppy disk, let alone something like Open Office.)

My next plan is to replace my Mac mini with a 24" iMac. I considered a quad core Mac Pro for not a lot more money, but I decided I'd prefer the reduced "clutter" of the iMac. And if I wanted an Apple monitor, then the Pro would end up quite a bit more expensive. When I bought the mini it was more in the nature of an experiment so I bought the cheapest Mac I could. Now that I've pretty much converted to Mac I want something a little "bigger" i.e. 4 gb ram, 1 tb disk. Nothing like RAW photos and virtual machine images to eat up disk space!

Tools, NetBeans, and Eclipse

I've been pushing my Ruby on Rails programmer to use some kind of IDE instead of just an editor. I suggested NetBeans since I'd read some good things about it. He installed it, but so far I haven't convinced him to switch. I can sympathize, when you've got a comfortable routine it's a hassle to change. And productivity tends to take a dip at first while you're learning new tools. But in the end, better tools can make a big difference. Many of the new tools we've added to Suneido have become so useful that I'd be really annoyed to lose them. (Even though we went years without them, and without knowing what we were missing.)

Soon after, I found myself working on the Suneido C++ source code with ... just an editor. Hmmm. I decided maybe I should give NetBeans a try myself. I also figured I should see what progress there had been on the C++ tools (CDT) in Eclipse. Both NetBeans and Eclipse have had many improvements. I downloaded, installed, and tried both. My conclusions were similar to Eclipse 3.3 or NetBeans 6.0.

As much as I like the concepts of Eclipse, it's got some awkward aspects (like "workspaces"). With NetBeans I was able to get the Suneido code loaded with little trouble. I ended up giving up on Eclipse. I'm sure I could have figured it out in the end (I have in the past) but I wasn't in the mood to spend the time.

It's interesting that both these IDE's are written in Java and run on Windows, Linux, and Mac.

Of course, in the end, I used up the time I had messing with IDE's and didn't make much progress on my code. Was this an excuse to goof off or was it an investment in better tools? Depends whether it pays off in the future I guess.

Tuesday, April 08, 2008

Too many heap sections

Recently, a couple of our clients with big databases (for us that's over 4 gb) have been unable to repair their database after crashing. (Why they crashed in the first place is another question - there may be hardware problems.) The repair aborts with "Too many heap sections". Luckily Thankfully, the software does automatic backups twice a day and the customer didn't lose much data. But the repair is still preferable as it generally only loses incomplete transactions that were in progress at the time of the crash.

This error comes from the Boehm memory manager/garbage collector we use. The repair process does keep a lot of information in memory and the amount of information is relative to the size of the database. The question was how to fix it? Did I need to rewrite the repair process to keep less information in memory, maybe use a temporary file? (Although that would make it slower.) Or was there something that could be adjusted in the Boehm code? There's also a newer version of the Boehm code - we're on 6.5 and the latest is 7.0

I searched on the web but didn't find any useful information about this error. Most mentions of it were pretty old. (I did find a reference to the Boehm code with the Mac OS X code - I wonder what part of OS X uses it?)

I searched the Boehm code for the error message and found it in several places (not very DRY). The error is caused when MAX_HEAP_SECTS is exceeded. I searched for where that is defined and the value seemed to depend on whether SMALL_CONFIG or LARGE_CONFIG (or neither) was defined. I wasn't specifically defining either, presumably leading to a medium setting.

I figured it was worth a try re-compiling with LARGE_CONFIG. I modified the makefile and re-built. Then I read a magazine while I tried running the repair process. (It takes a while to process a 4 gb database, even with a fast computer with lots of memory.) It reminded me of the "old" days when I'd have time to catch up on my reading while I compiled what today would be regarded as tiny C programs.

Damn, it still crashed with the same error. Oh well, should have known it wouldn't be that easy.

I went back to remove the setting from the makefile and glanced at the file name. Hey! I was editing the makefile for the MinGW version, but I'd been testing the VC7 version. Doh!

Try again, this time build the right version & test the version I build.

Eureka! It completed successfully.

Now we'll just have to test this version enough to be relatively sure that the change doesn't have any unwanted side effects. Maybe it will be that easy after all!

Sidenote: Considering the number of people and projects using the Boehm code (e.g. Mono) it seems odd that there isn't more documentation. I can understand Boehm not writing it, I'd rather he spent his time on the code. But you'd think someone along the way would have written some. Maybe no one else understands it well enough. Despite having written my own, I know I don't - I just treat it as a black box.

More Software Frustrations

Here's a few software things that have bugged me lately:

I'm currently reading Why Software Sucks by David Platt. He praises how Google senses the country you are connecting from and uses the language of that country e.g. Spanish in Mexico. The problem is, it does this even if I am logged in to my account. Just because I am in another country doesn't mean I've changed my language. It's possible to get back to English but it's annoying nonetheless.

Next, I'm on my bank web site paying some bills (a feature I do appreciate). A message shows up telling me I can now receive one of my bills electronically, rather than having it mailed to me. Great! I click on the link and get presented with a blank form, without the name of the company filled in, forcing me to pick it from a huge list. (Most of the list is not applicable, e.g. for other provinces.) Next there is a "Name" field - who's name? mine? but surely it knows my name? or the company name? but I just picked it from a list? I leave it blank and continue - it doesn't complain so I'm still not sure what it wanted. Next it tells me I have to sign up for an ePost account. It's free, but it's annoying to have to sign up for yet another on-line account and come up with yet another password. Can't the company send bills to my bank without every account holder having to sign up for ePost?

But the best is yet to come. I get through the ePost signup, only to be told that I have to contact the company personally, by phone, before I can complete the process. Yeah, right. I pay bills on the evenings and weekends, so I have to wait till business hours to phone. Then I can look forward to the "pleasure" of an automated phone system and being on hold forever. Then I'll likely talk to some poor customer service person who has never heard of sending bills electronically. No thanks. And they wonder why so many on line transactions are abandoned part way through.

The next incident was minor, but it's a good example of a GUI blooper. I was ordering books through Lulu (copies of Getting Real for my programmers). I'm entering my address and I get to the field for "State/Province". Except there are no provinces. That wouldn't surprise me except that for the prompt. I skip it and move to the next field, "Country", and enter Canada. The screen jumps around a little bit. Hmmm... sure enough, State/Province now lists provinces. This is a good trick, some programmer obviously applied some Javascript. But they didn't consider that people generally enter fields in order, pretty much guaranteeing that Canadian customers will be frustrated. And most people wouldn't notice the slight screen jump and figure out that they could go back to a previous field and it would now magically let them do what it wouldn't just a minute ago.

We run into this issue in our own software. The rule of thumb is that if a field "depends" on another field it should be "after" it in the normal entry order. Usually this can be handled simply by changing the order of the fields. But in this case, it seems "wrong" to put Country before State/Province. My suggestion would be to remove the fancy Javascript - just have a combined list. The programmer can apply their Javascript by filling in the country based on the State/Province choice.

We use Snagit for our screenshots. It's a good program. But it has one really annoying "feature". Quite frequently, when I start it up, it pops up a dialog saying "You have the most recent version of Snagit". I always have to read this a couple of times because the normal expectation is that it would pop up to tell you there's a newer version.

I haven't quite figured out the logic behind this one. Why do I need to be told I have the latest version? I don't think it comes up every time, so maybe it only comes up after it does a periodic check for a new version. (Of course, I only use it periodically, so it happens more often than not.) It may just be that the programmer thought that since they'd gone to all the work of checking for a new version there should be some recognition of this. Programmers are often strangely reluctant to just have the program quietly do it's stuff. Maybe we need pop up blockers for more than just our web browsers.

There is a link on the dialog to the release notes. So perhaps the purpose of the pop up is to give you a chance to read the release notes. But surely if that was the purpose it would only pop up once after installing a new version. How many times (if any) do I want to read the release notes?

Enough being critical, I'd better go and try to do better myself. It's always easier to criticize. (Why Software Sucks does actually suggest solutions or at least, better alternatives, to most of the things it complains about.)

Monday, April 07, 2008

MindMeister Online Mind Mapping

I came across MindMeister from a list of web applications that use Google Gears to run off-line. Check out the demo screencast for a quick overview. It's free for a basic account.

From a quick look MindMeister looks like a good example of a Web 2.0 application.

I haven't got into using mind mapping but some people swear by it.

Monday, March 31, 2008

Lightroom & Photo Workflow

I seem to have settled into a reasonable work flow with my photography and Lightroom.

I have multiple memory cards for each camera. Two 2 gb cards for the Canon SD700 IS - shooting jpeg gives me about 700 photos per card or about 15 min. of video at the highest quality (640x480, 30fps). I can't remember the last time I filled up one of these cards. I have four 4 gb cards for the Pentax K10D - shooting DNG (raw) I get about 240 photos per card and I do fill these up sometimes. For the Pentax I always carry at least one spare memory card and a spare battery.

I have spare batteries for both cameras but seldom need them. The last a long time and most of the time it's easy to recharge them. The exception is in more remote locations where power isn't readily available.

When traveling I usually download/import once a day to Lightroom on my MacBook. I take advantage of Lightroom's option to make a backup as it imports, to an external hard drive. I also have Lightroom set to back up its catalog to the external drive daily. When I fly I keep the MacBook and the external drive separate to improve my chances of not losing both!

I also separate the cameras (and their chargers and other accessories) so even if one bag gets lost I'll still have a usable camera. I worry as much about the chargers since they'd be impossible to replace in most places and without them the camera would soon become useless. In this respect I like the use of standard AA batteries in my Canon S3. I still use rechargeables but I can use readily available disposables in an emergency, and there's also a much better chance of replacing AA rechargeables and recharger.

After I download I do NOT erase the memory cards and I rotate them so I put in the least recently used card. That way the cards provide a third short term backup in case I mess up an import. I erase a card when I put it into the camera. The only drawback to this system is that occasionally I forget to erase the card when I put it in the camera. I then run out of space soon after I start shooting, but I can't erase the card at this point because I've put new pictures on it. At least Lightroom has an option to ignore already imported photos.

My primary Lightroom catalog and photos are on my desktop system at home. For each trip I start a new catalog on the MacBook just for that trip. When I get home I import that catalog into my main system. One minor drawback of this is that I start fresh with keyword tags and sometimes they don't match with the ones I'm using in my main catalog. But this is minor and easily fixed using Lightroom's ability to rename keyword tags.

I keep both the main and the MacBook catalog organized primarily by date with the yyyy/yyyy-mm-dd format. I rename as I import to yyyymmdd - filename. When I first starting using Lightroom I didn't rename but the straight camera generated file names aren't very helpful, and there's always the risk of duplicates. In my main catalog I add another level of folders for major trips e.g. 2008/2008 Baja/yyyy-mm-dd. This makes it easier to select all the photos for a trip e.g. to put together a slide show.

Some people keep their photos in folders by subject or type rather than date. To me recording subject or type is better done using tags. Otherwise you're continually faced with questions like which folder do I put a picture of a bird on a beach at sunset? In birds, beaches, or sunsets? That's the power of the ability to apply multiple tags, in this case birds, beaches, and sunsets. And it's just as easy to click on a birds tag as it is to click on a birds folder. Of course, this assumes you're inside Lightroom. Outside Lightroom you'd have no easy way to select the birds tag.

I'm not very systematic about my tagging. I'll do a little tagging when I first look at the photos. And I might go through and apply a tag (e.g. birds) if I want to make a selection of a certain type or subject. But lots of my photos aren't tagged at all, which makes it hard to find them later. I could do better in this area.

Another choice is whether to delete "bad" pictures e.g. badly exposed or out of focus. Personally, I don't delete, no matter how bad the photo is. I just flag it as "Rejected" and normally I filter rejected photos out of view. This is similar to Gmail's philosophy of keeping everything. Storage is cheap. It's not so much that you'll ever want a totally ruined photo, it's the time and mental effort needed to decide on the marginal cases. You can always change your mind about a Rejected photo, if you delete it, it's gone.

One case where I might be tempted to delete is when I'm bracketing exposures. Since they're more or less duplicates, why not just keep the best exposure? But again, it's not always clear which is "best". You might actually want to combine several exposures for a HDR image. And Lightroom's ability to "stack" photos avoids having to look at the "duplicates" most of the time.

I'm finding the K10D tends to overexpose bright scenes like snow or sand. I need to figure out when I need to override the exposure, by how much, and what the best way to do that is. I've been bracketing when I suspect there will be a problem. So far it looks like -1 stop does the trick. Lightroom's Recovery slider (and shooting raw) lets you fix a certain amount of overexposure, but it can't recover what really isn't there. This is where composing in the display in the small camera helps. I need to get in the habit of reviewing photos on the K10 and maybe even checking the histogram. Of course, that only works for static subjects where you have the option of another shot.

I also find that for certain shots e.g. closeups I need to manually focus the K10, especially when low light means a small depth of focus. The auto-focus is pretty good, but it can't read your mind and doesn't always pick the right thing to focus on.

On my main system I use Chronosync to automatically back up my photos and catalog to a network drive. And I also have TimeMachine running to an external drive. But currently I don't have a good offsite backup system. I keep a lot of my photos on my work computer, but I don't have a convenient (or automatic) way to do this so it's not too reliable. I've tried a few ways to sync my home and work computers (especially my photos and music), but it hasn't been too successful, partly because of the sheer volume of data to sync, especially at the start. An online internet backup is another option, but again, it's tough with the volume of data, again especially at the start. And not just at the start - when I come back from this trip I'll be adding 40 gb in one shot. That's a lot to transfer over the internet.

When traveling, I try to upload photos to the web every few days so my family and friends can keep up with me. So far I've been using Google Picasa Web Albums. Before Lightroom I was using Picassa to organize my photos, so it was natural to use it's web albums. It works well enough so I've just kept using it. And the Lightroom plugin to export directly to a web album makes it very easy. Now I keep thinking I should give Flickr a try.

I do a certain amount of tweaking of the photos I choose to post - mostly cropping and exposure. For this, and for general organizing, it's great to use the same software (Lightroom) traveling as well as at home.

You can check out some of the photos I've posted at : http://picasaweb.google.com/apmckinlay

Monday, March 24, 2008

Publishing Video

Everyone is posting their video these days, how hard can it be?

I shot some short video clips of a school of dolphins on my Canon SD700 IS.

The first hurdle was to combine them together. I fired up iMovie for the very first time and managed to put in the clips, crop them, and add transitions. So far so good.

I exported the movie and uploaded it to YouTube. The QuickTime mov file was about 100mb for 30 seconds so it took a long time (almost 2 hours) to upload it over the hotel DSL. It would be nice if you got some progress feedback - I started to wonder if it was working or not.

Finally I can play the YouTube video. Yuck. Shrunken down from 640x480 to YouTube size and highly compressed, it looked like crap.

I start looking for alternatives and I decide to try Google Video. One of the reasons is that it recommends 640x480. I try the Google Uploader this time. It still takes forever. And the Google Video version still looks like crap and it appears to have shrunk it even smaller than YouTube (pick "Original Size" on the pop up menu).

The most recommended format seemed to be mpeg4 so I started to look for a way to convert the file. One of the suggestions I found said to use iMovie. Sure enough, although it is far from obvious that iMovie can do this, there are instructions in the help. The key is to choose "Expert Settings". I guess only experts would want something other than QuickTime! It took some trial and error experimentation to get a file that looked decent. I ended up choosing Lan/Intranet, even though that isn't what I want. (I hate it when the "right" settings are obviously "wrong".) This mpeg4 file is under 6mb, a little better than the "default" QuickTime file.

Next I decided to try blip.tv only because I'd recently seen some videos published there. Their FAQ doesn't really say much about resolution or compression, but then again, neither do YouTube or Google Video. Their web upload gives good progress feedback - very nice. (And I'm at the coffee shop this time and the upload rate is better.) I watch the video and, eureka, it looks good. But it says it's still being processed (converted to Flash). When it finishes I watch the Flash version. But at least you can choose the format and if you pick the MPEG4 version the quality is better. I'm not sure if that's a limitation of the Flash format or if they just choose a higher compression. At least even the Flash version is the full 640x480 unlike YouTube and Google Video.

Google actually has a third option - uploading to Picasa Web Albums. But you have to use Picasa to upload video and they don't have a Mac version. I could have installed Picasa inside my Windows VM in Parallels but that seemed like too much hassle.

I'm a newbie at publishing video, and probably pickier than most people. But it seems like this technology is still pretty rough around the edges. Maybe there are ways to make YouTube or Google Video work better. Oh well, in the end I accomplished what I set out to.

Remote Access with LogMeIn

I've been using LogMeIn to access my office computer while I'm traveling (to get access to things that are internal to the office). So far it's worked well, even accessing a Windows machine from my MacBook.

The only problem I've run into is that the Java file transfer doesn't work under Firefox - it locks up so you have to kill Firefox. This is mentioned on their support forum, but so far no solution. It works from Safari though, so it's not a big deal.

The basic version is free. I decided to pay for the Pro version since it was relatively cheap and I could use some of the Pro features.

There's a bit of a lag in typing, but that's not surprising considering I'm using wireless to dsl from Mexico to Canada. Occasionally it trips me up, but for my usage it's fine.

Thursday, March 20, 2008

First Time Skype

I finally got around to using Skype to talk to Shelley back in Saskatoon, Canada from Loreto, Baja Mexico where I am currently.

The first time we connected we couldn't hear each other, although we both heard the "ringing" in our headsets. At least we had the Chat facility to try to debug the problem. It was a little harder since I was on the Mac version and Shelley was on the Windows version, so the menu options are different. On the Windows version the settings are in a "Personalize" option - which seems like a rather odd choice of name. (On the Mac it's the standard Skype > Preferences)

Everything seemed ok so we tried again and this time it worked. Not sure why it didn't work the first time. The only thing I can think of was that Windows hadn't finished recognizing the USB headset (Shelley had only just plugged it in).

I could hear Shelley great. She said my sound quality wasn't so good. That might have been due to my bluetooth headset. The microphone is quite far from my mouth. And on top of that, I was outside on the deck (to reach the wireless) and there was a breeze and traffic noises. There's a definite lag time and you pretty much had to take turns talking, but it was definitely usable.

I wouldn't want to use it as my regular phone, but it beats dealing with long distance when I already have an internet connection.

Pretty amazing - bluetooth from headset to laptop, wireless from laptop, and then by internet across North America. And all for "free" :-) And more amazingly, given all the software involved, it works.

You Weren't Meant to Have a Boss

The latest essay by Paul Graham.

I've been my own boss for over 20 years now. And the only job I had before that was for a pretty small company where I had a lot of freedom. Nowadays, I can't imagine working for someone else - it's a frightening thought.

On the other hand, I've known lots of programmers (and other people) that want nothing more than to settle into a secure job in a large organization. To each his own, I guess.

Sunday, March 16, 2008

Customer Service

I've been traveling recently. Travel gives you a great perspective on customer service. You're continually dealing with airlines, restaurants, hotels, etc. One of the things I've noticed lately is that you can divide people/companies into two types. One type wants to go by the book. If you ask them for something that doesn't fit into their predefined "boxes" their response is "sorry, can't help you". They might be really friendly about it, but they don't help you. The other type will look for a way to help you, even though you're asking for something unusual or something that isn't really their business.

For example, we go into several agencies and say we want to go out snorkeling to a different island than the usual one. Most of them give us the official response of "we don't go there" or "we don't have enough people who want to go there" and that's the end of it. But then we go to another agency and they say "sure, we can work something out, it may depend on the weather and what else we have booked, but we'll figure it out". That's the kind of service that brings me back for repeat business.

I just hope that my business is more like the second type. Obviously, you can't answer every single request but you can try to come up with something to offer. My staff will come to me and say "so and so wants xyz, what should I tell them". If possible, I'll say "give it to them". Lots of times it's not possible, and you can tell that the staff person just wants to say no. But often we can offer a partial solution or an alternative or a suggestion. It may or may not totally satisfy the customer, but at least you've tried.

Here's an interesting video from a customer service conference: (note: it's an hour long)

Friday, March 07, 2008

Day 4 at Etech 2008

Started out with XO Laptop Hacks. I just got my OLPC XO not that long ago and I haven't had time to do much with it so it was interesting to hear more.

Next I went to the presentation on CouchDB. It's an interesting project - written in Erlang, primary interface JSON via HTTP, started by one guy on his own, but now IBM pays his salary.

I couldn't miss Jeff Jonas's talk on behind the scenes in Las Vegas - he's an entertaining presenter. His talk didn't have as much software content as last year, but it was still good.

Synthetic Neurobiology was largely on the theme of "body hacking". One of their more clever hacks was to introduce genes into neurons (via a virus) that make the cell sensitive to light. You can then trigger the neurons to fire (or suppress them from firing) using light. Wow.

The day (and the conference) ended with more keynotes. First by Alex Steffen of WorldChanging (who I was disappointed to see carrying a disposable plastic bottle of water!) Then a presentation on Twine - yet another social network app, but at least with some interesting semantic web aspects. A brief talk on Digital Democracy about the use of the web (and social networks, of course!) in politics.

And the final talk was by Timothy Ferriss of the (4-Hour Workweek). I'd seen Tim around during the conference, attending talks. My first impression was that he was short. A bit like seeing an famous actor and finding out how short they are. He also blended in pretty well carrying his backpack around, although without the ever present laptop of most people. But when he gave his talk he definitely had a bigger presence and charisma. Unlike every other speaker, he did not have any powerpoint slides - he just talked. If you've read his book and blog it was nothing new. Despite being a wing nut in many ways, I still find him "inspiring" if that's the right word. The gist of his message is that you don't have to be powerlessly overwhelmingly "busy". You can re-engineer your life to do what you want to do.

Wednesday, March 05, 2008

Day 3 at ETech 2008

When I saw the first speaker of the day was some old guy instead of the scheduled Kathy Sierra, I was a little disappointed. The "old guy" turned out to be John McCarthy, the inventor of Lisp and "artificial intelligence". To me it was a pleasant surprise to see someone like this along with all the "tech kiddies". Unfortunately, his talk moved really slowly and they had to cut him off. I felt bad for him.

Kathy Sierra did speak later. She talked about how we all want to be "good" at something and what it takes to do that. Surprisingly, natural ability counts less than sheer focus and concentration. Of course, that requires motivation and it's less clear how to get that. Lately I've been thinking about when I developed Suneido and my focus (perhaps fanaticism would be more accurate). I seldom achieve that kind of focus anymore and I miss it. I just haven't quite figured out how to get it back. Then again, do I want to get it back? Spending the majority of your waking moments working on something isn't exactly a balanced life, no matter how challenging/rewarding/addictive it is.

Looking at today's sessions it seemed like there wasn't that much that was attractive to me. But sometimes it's a good thing to be be forced to go to talks that I might otherwise have skipped. After all, the whole point of the conference (for me) is to get exposed to new ideas.

Brain Imaging and I Sing the Body Electric surprisingly turned out to have a lot of common ground. There's a definite feeling around that hacking humans is the next frontier. I've never been keen on the idea of "hacking" myself, maybe because I know how easy it is to screw up complex systems.

Hackers Built My Motorcycle started out by saying his talk had nothing to do with the title, and sure enough he never mentioned motorcycles. Maybe that was an example of "hacking" the conference schedule. Nevertheless, it was a fun talk. The problem with talks about security, like talks about the environment, is that they are somewhat depressing! He proceeded to explain how easy it is to hack into cell phones, web sites, house locks, RFID credit cards ... scary stuff.

The next talk, about technology in Cuba, had the potential to be quite interesting but the two speakers read a pre-written speech and that seldom works well. I've really found that a good presenter is worth going to regardless of the topic, and no matter how interesting the topic, a bad presenter will kill it.

My final session was on OpenCV, an open source computer vision library. I don't know much about this area but it was pretty amazing what's possible these days.

Tuesday, March 04, 2008

Day 2 at ETech 2008

Saul Griffith's talk on Energy Literacy was well done but pretty depressing. I can't dredge up much optimism that a) people will massively reduce their energy use, and b) we'll shut down much of our fossil fuel energy production and replace it via a massive construction of new green energy sources. I can't see it happening. People won't take such drastic action until there's a crisis. So it was further depressing to hear that the time lag on carbon reduction affects can be hundreds of years. i.e. By the time there's a crisis it'll be way too late to do anything. Of course, it's a contentious subject. Someone at my lunch table argued that technology would save us. Hmmm ... maybe it'll save us humans (we're good at that) but what about the rest of our ecosystem? We don't have near as good a track record there.

The presentation by MegaPhone on Collaborative Gaming in Public Spaces was entertaining. The two presenters were the founders and looked to be still in their teens! They develop ways for people to interact with public video displays, primarily by cell phone. They ended up in the debugger on the big screen a few times, but it seemed to work in the end. I'm not sure how they got to be keynote speakers but it was refreshingly "innocent".

The session on the Future of Mind Hacks was pretty interesting. You always wonder if you've picked the right session, but when Timothy Ferriss is a couple of rows ahead and Tim O'Reilly is a couple of rows behind I figure I've picked a good one.

Two Microsoft employees were sitting beside me, one had a MacBook, the other an iPhone (and worked in the mobile division). Hmmm ... is that getting to know the competition?

After the keynotes I went to Tap is the New Click on gestural interfaces (e.g. the iPhone). It was ok, but not great, mostly just a dry overview.

At lunch I counted 10 tables with a single person at them (one of them was mine). Good to know I'm not the only anti-social geek. But most geeks must be more social electronically than me. One of the big topics is social applications. Frankly, I don't have enough friends to need an application to keep track of them. Then again, I don't even have a cell phone so I'm obviously abnormal.

After lunch I went to Green Nano by an HP researcher. I've been excited by nanotech since Eric Drexler's Engines of Creation so it was nice to hear about progress. But the talk seemed a little dry.

For some variety I next went to a talk on Digital Activism by Ethan Zuckerman. It turned out to be pretty interesting. It's good to hear about some "positive" uses of technology.

Then DIY Drones by Chris Anderson (of Long Tail fame) - fun.

And finally, Personal Productivity by Gina Trapani of LifeHacker, another good talk. (Of course, Timothy Ferriss was at this one as well.)

All in all, a pretty good day.

Monday, March 03, 2008

Day 1 at ETech 2008

The first day at ETech is tutorials.

First up was Live, Vast and Deep: Web-native Information Visualization by Tom Carden of Stamen Design. Although it was pretty high level and didn't get into too many details, there were lots of thought provoking examples and links to things to investigate - and that's what I'm looking for.

It was a tough choice between this and Storyboarding for Nonfiction by Kathy Sierra (or Creating Passionate Users). I decided that I might get more useful ideas (for me) from visualization. But I bet Kathy's talk was good too.

In the afternoon I went to Debugging Hacks: What They Never Taught You About Solving Hard Bugs by Marc Hedlund of Wesabe. Nothing really new but a good talk on how to solve hard bugs. One point that resonated with me: "the goal is not to suppress the symptoms, it's to understand the problem". I have this discussion with my own programmers on occasion - removing an assert is not "solving" the problem! And conversely, adding the assert did not "cause" the problem.

Saturday, March 01, 2008

San Diego ETech

I'll be in San Diego this week for ETech.

Drop me an email if you're in the area and interested in meeting up.

Wednesday, February 27, 2008

Creative Uses for the Wii

This is very cool, maybe I need to buy a Wii.



Keep it Simple

37signals response to an article about them in Wired.

I don't necessarily agree with all of 37signal's philosophy but it's nice to see someone fighting against complexity and for simplicity.

Tuesday, February 26, 2008

The Future is Free?

An interesting article by Chris Anderson (also know for the "long tail") in Wired on how and why more and more things are "free".

Sunday, February 24, 2008

Skype, Headsets, and Bluetooth

I'm going to be spending some time away from home without Shelley so I thought I should set up Skype so we could talk. It was no problem to download and install the software on my MacBook and on Shelley's Windows XP PC.

I bought a USB headset for the PC but I wanted something smaller for the MacBook and since it has Bluetooth I figured I could get a Bluetooth headset. I couldn't find a computer specific bluetooth headset, only cell phone ones. But would they work with the Mac? I did some research on the web and read about lots of problems, but mostly old issues that were supposedly fixed with Leopard. I didn't even bother trying to ask a clerk at Staples which bluetooth headsets worked with Skype on OS X on a MacBook. Although I guess if you were lucky you might get some kid who was an expert on the issue.

I chose a Bluetrek Tattoo headset, more or less at random. I charged it up, and managed to pair it with the MacBook, but I couldn't get any sound in or out of it. I couldn't even tell if it was "on" or not. It was supposed to have a green light when it was switched on but I got no lights. I played around for a while but it just seemed to be dead. I took it back and picked a Motorola H800, again for no particular reason. This time it worked fine, no problems. I still don't know if the Tattoo is incompatible, or if I just got a dud.

So now I have Skype working. The sound quality of the "echo" test call wasn't great but it says it's going to the UK so that might be part of the reason.

Of course, what I paid for the two headsets would have paid for more than enough regular long distance phone calls, but what would be the fun in that!

Tuesday, February 19, 2008

Suneido Build Frustrations

I made a minor improvement to the Suneido source code this morning, ran make, no problems, built-in tests ran successfully.

But when I tried to use the new executable I got an obscure database error. What's going on?

I had built with MinGW so I switched to Visual C++. Exact same error!?

This was at home and my last builds had been at work but I can't see why that would matter.

Remove all the object files and build from scratch. No good, same problem.

Check version control to see what I'd changed lately - very little, and nothing that seemed related to the error.

The error is from the database btree code. Maybe the database is corrupted. But all the exe's, old and new, say the database is fine.

Try creating a new database with just the standard library. Now I get a different error related to the Scintilla source code editing component.

Build a MinGW debug version and run it under GDB. That gives me a clue to what query is leading to the error. It had appeared to be outputting to the database, which seemed odd for start up, but it was actually building a temporary index for a query that it was reading. Although that query in the old working executable doesn't require a temporary index.

Turn on the query tracing at the start of the standard library Init to see where the query is coming from. It's loading the plugins.

Aha, that's why the database with only stdlib gets a different error - because it only has to look for plugins in a single library and therefore no temporary index. Yeah, if I disable the plugin loading then I get the other error.

Two unresolved questions
- why the temporary index in the new builds but not the older build?
- why the later UI error?

And how are these two questions related? (assuming a single cause) It seems like it would have to be something low level, like the garbage collector, to affect such unrelated areas.

Of course, it could be something like an uninitialized variable that happens to get a different value on my home system. But it seems too consistent for that. And something like would likely have been encountered before now.

What is different between my work and home machines? I call the office and have them install LogMeIn on my work computer so I can access it. I try building with MinGW and it works fine. The exe is a different size though. Something is different.

Transfer the home exe to work to see if it's the environment. Same error message, so at least it's not because of my Vista on Parallels on Mac setup at home.

md5sum the files at work and at home and compare. The only real difference is the change I made this morning.

But ... that couldn't be the problem could it? Revert that file and build.

Oh no, this is really embarrassing! It works. The problem was the most obvious first place I should have looked - the change I just made.

I'm really tempted not to post this - it just makes me look stupid.

Why did I go off on a wild goose chase? I guess because the error seemed to be so totally unrelated to my change, and I hadn't built for a while so it seemed likely that there could be a problem. And the change I did seemed trivial so I didn't suspect it. And because it seemed trivial I didn't write any tests. (The bug was also obvious, once I looked for it.)

Ouch. There goes a few hours down the drain. Maybe I learned a lesson, but sadly it's one I should have learned a long time ago.

Monday, February 18, 2008

ZENN and the art of slow progress

One of my recurring complaints is gadgets that are only available in the US. But at least it makes a certain amount of sense when they're made in the US. But here's an example where it's made in Canada and it's still only available in the US!

Saturday, February 16, 2008

Wednesday, February 13, 2008

VirtualBox

In Tim Bray's ongoing blog he mentions Sun's acquisition of Innotek, the developers of VirtualBox - virtualization software for Windows, Linux, and OS X. I haven't tried it, but it looks interesting - and free for personal use.