One of the common suggestions for testing is to test around limits or boundaries. e.g. try 0 or 1 or -1. I was painfully reminded of this by a recent debugging session.
Some background. Originally Suneido memory mapped the entire database file into memory. That was simple and fast but limited the size of the database to less than 2 gb (the amount of address space a Win32 process gets).
When our customers' databases started to approach this size I had to change Suneido to memory map a limited number of segments of the database file and re-use address space in an LRU fashion.
Considering how major this change was it went amazingly well. I was quite nervous about weird bugs from things like dangling pointers caused by re-mapping. So I did quite a lot of testing before we rolled it out. And we kept a close eye on it at first.
This was quite a while ago and I figured everything was good. Until our client with the largest database started having problems. Their database was right around 4 gb. So I did some testing and sure enough when the database reached 4 gb MapViewOfFile would fail with "Access Denied". The "Access Denied" part threw me off since it sounded like some kind of security/permissions issue. I dug through MSDN trying to find some mention of a 4 gb limit but couldn't find anything.
As with so many of these things, the error turned out to be really stupid. Memory mapping a file with Win32 is a two step process, first CreateFileMapping, then MapViewOfFile. Since it was MapViewOfFile failing I hadn't paid much attention to the CreateFileMapping step.
Both API calls accept 64 bit file offsets as two 32 bit values. I was only passing the low 32 bit value to CreateFileMapping. Stupid! Of course, it worked up to 4 gb. But over 4 gb it wasn't passing the correct offset. Whereas I was passing both halves of the offset to MapViewOfFile. It was trying to map the correct offset, but was failing because the file mapping was for a different, incorrect offset. (A little confusing because the call with the wrong arguments was succeeding and the call with the right arguments was failing.)
Why hadn't I caught this with my testing? Because the old database size limit was under 2 gb so I tested with over 2 gb. The bigger the database the slower the testing so I never bothered going as big as 4 gb. Oops.
I should have realized that the limit for 32 bit values would be an obvious failure point and I should have tested up to and over 4 gb.
After I fixed this bug I was able to grow a database past 4 gb. For a minute I thought I was done. But a little more testing revealed that it wasn't accessing the data past 4 gb properly.
This was a little tougher to track down. I got fed up with waiting so long to create big databases and came up with a way to quickly expand the temporary databases used by the tests to over 4 gb. (I could have saved considerable time in the long run if I'd done this at the start!)
The problem turned out to be a place in the code where I mistakenly stored a 64 bit file offset in a 32 bit integer. Again, this worked up to 4 gb but not beyond.
After this fix all the tests ran successfully.
Next time maybe I'll remember to test around obvious limits and boundaries.
PS. Not that it's any excuse, but it seems odd that the compiler wouldn't give a warning about assigning a 64 bit value to a 32 bit variable. I'm using -Wall with GCC and the only warning I have turned off is -Wno-sign-compare. Maybe there are warnings I need to turn on that aren't included in -Wall. I should probably look into this.
PPS. Along the way I got an excellent demonstration of the effects of locality on multi-level storage. If I created a big database by doing: for each table, add 10000 records, then it was reasonably fast. If I turned it around to: 10000 times, for each table, add 1 record, then it was about 10 times slower. Dealing with one table at a time kept all the data and index nodes for that table together so they could all be mapped into memory. Looping through each table and adding one record to each causes the data and index nodes for all the tables to be mixed together over too large an area to be mapped all at once, causing continuous un-mapping and re-mapping.
Monday, May 05, 2008
Saturday, April 26, 2008
The Best UI is No UI
The other day I watched someone unlock their car with a remote key. No big deal, a common occurrence. But for some reason I noticed it and thought that's cool. Maybe because I never had a remote key. I went from a regular, mechanical, non-electronic key, to a Prius with a "Smartkey" system.
With the Prius, I don't need to touch the key. I walk up to the locked car and open the door. As long as I have my key in my pocket or backpack it's all automatic. In other words, no user interface. (I'm now spoiled - I resent having to pull my office key-card out of my pocket, and even worse to have to use a mechanical key on my house.)
Too often in software (and other areas) we think the ultimate "solution" to a problem is some clever UI, like a remote key. But people don't want a clever UI, they want to get some job done, and often the less UI the better - none if possible.
Similarly, the best solution for errors is to prevent them in the first place. (again, no UI) You can't lock your key in the Prius because it won't let you lock the car if the key is inside.
With the Prius, I don't need to touch the key. I walk up to the locked car and open the door. As long as I have my key in my pocket or backpack it's all automatic. In other words, no user interface. (I'm now spoiled - I resent having to pull my office key-card out of my pocket, and even worse to have to use a mechanical key on my house.)
Too often in software (and other areas) we think the ultimate "solution" to a problem is some clever UI, like a remote key. But people don't want a clever UI, they want to get some job done, and often the less UI the better - none if possible.
Similarly, the best solution for errors is to prevent them in the first place. (again, no UI) You can't lock your key in the Prius because it won't let you lock the car if the key is inside.
Labels:
user experience
Thursday, April 24, 2008
Multi-Threading Suneido - Thread Specific Storage
Back to work on multi-threading Suneido. After frittering away my last session, I figured I'd "park downhill" and decided at the end of that session that the next thing I'd work on was replacing my home brew thread specific storage with ACE_TSS. With a clear task I wasted less time.
ACE_TSS is a template class used like: ACE_TSS<mystruct> mytss;
which is then accessed like: mytss->member
It overloads operator-> to do its stuff.
In order to keep ACE specific code as limited as possible I defined functions to access my thread specific storage. For example:
Proc*& tss_proc() { return tss->proc; }
The '&' is needed so the functions can be used as lvalues like:
tss_proc() = newvalue;
Unfortunately, this meant replacing all the instances of proc with tss_proc() but I could use the compiler to do the heavy lifting by removing proc and seeing where I got compile errors.
I could have kept the name as proc, but changing it made the transition clearer, and it doesn't hurt to make it clear in the code that it is thread-specific.
For my current, non-ACE version of Suneido I simply defined the functions to access my home brew thread specific storage.
In the process I discovered that part of the old thread specific storage code was broken. It hadn't caused problems because it was code to clear unused memory during garbage collection, so it just meant the garbage collection was a little less efficient. I fixed it for the current version, but I don't know how to implement it for real threads so it's a good thing it's not critical!
The only real problem I ran into was that the Boehm garbage collector does not appear to "see" the ACE thread specific storage. That's probably fixable but it would require digging into how ACE_TSS is implemented. And that would be platform specific. Yuck. Instead I was able to make the thread specific storage simply point to data that would be seen by the garbage collector. I'll just have to be careful I don't forget about this issue in the future.
As far as I can tell, it's all working, both the current version and the ACE version. Next I have to start tackling the crux of the problem - locking global data structures. I think I'll start with the symbol table and the global values table. These are fairly loosely coupled and shouldn't be too hard to handle. That should give me a little experience to help tackle the hard parts like the database.
PS. I started this session using NetBeans but I gave up part way through and went back to using Scite. This was partly because I haven't got NetBeans set up to build, which meant I had to compile in a separate command line window, which meant there was no easy way to click on the errors and go to that part of the source. (which I can do in Scite, although it's mostly a straight editor). I could have messed around with getting the build environment set up, but then I would have ended up like last time, with no real work accomplished.
Another thing I found disappointing in NetBeans is that it didn't seem to have the kind of powerful tools that I want/expect from an IDE. For instance, there didn't seem to be any way to find all the uses of a function or variable. Maybe I just missed it. But I wouldn't be surprised if it wasn't there, C++ tends to get second-class support in primarily Java IDE's. The Eclipse CDT used to be fairly minimal too, but I think they're supposed to have made significant improvements in the latest version. I'll have to go back and take another stab at trying to use it.
A final annoyance in NetBeans was that it didn't treat underscores as word characters. e.g. if you choose "Whole Words" and search for "proc" you find "tss_proc". I looked through the configuration options but couldn't find anywhere to change this. It seems odd because that's not C++ specific. If anyone knows how to "fix" this, let me know.
ACE_TSS is a template class used like: ACE_TSS<mystruct> mytss;
which is then accessed like: mytss->member
It overloads operator-> to do its stuff.
In order to keep ACE specific code as limited as possible I defined functions to access my thread specific storage. For example:
Proc*& tss_proc() { return tss->proc; }
The '&' is needed so the functions can be used as lvalues like:
tss_proc() = newvalue;
Unfortunately, this meant replacing all the instances of proc with tss_proc() but I could use the compiler to do the heavy lifting by removing proc and seeing where I got compile errors.
I could have kept the name as proc, but changing it made the transition clearer, and it doesn't hurt to make it clear in the code that it is thread-specific.
For my current, non-ACE version of Suneido I simply defined the functions to access my home brew thread specific storage.
In the process I discovered that part of the old thread specific storage code was broken. It hadn't caused problems because it was code to clear unused memory during garbage collection, so it just meant the garbage collection was a little less efficient. I fixed it for the current version, but I don't know how to implement it for real threads so it's a good thing it's not critical!
The only real problem I ran into was that the Boehm garbage collector does not appear to "see" the ACE thread specific storage. That's probably fixable but it would require digging into how ACE_TSS is implemented. And that would be platform specific. Yuck. Instead I was able to make the thread specific storage simply point to data that would be seen by the garbage collector. I'll just have to be careful I don't forget about this issue in the future.
As far as I can tell, it's all working, both the current version and the ACE version. Next I have to start tackling the crux of the problem - locking global data structures. I think I'll start with the symbol table and the global values table. These are fairly loosely coupled and shouldn't be too hard to handle. That should give me a little experience to help tackle the hard parts like the database.
PS. I started this session using NetBeans but I gave up part way through and went back to using Scite. This was partly because I haven't got NetBeans set up to build, which meant I had to compile in a separate command line window, which meant there was no easy way to click on the errors and go to that part of the source. (which I can do in Scite, although it's mostly a straight editor). I could have messed around with getting the build environment set up, but then I would have ended up like last time, with no real work accomplished.
Another thing I found disappointing in NetBeans is that it didn't seem to have the kind of powerful tools that I want/expect from an IDE. For instance, there didn't seem to be any way to find all the uses of a function or variable. Maybe I just missed it. But I wouldn't be surprised if it wasn't there, C++ tends to get second-class support in primarily Java IDE's. The Eclipse CDT used to be fairly minimal too, but I think they're supposed to have made significant improvements in the latest version. I'll have to go back and take another stab at trying to use it.
A final annoyance in NetBeans was that it didn't treat underscores as word characters. e.g. if you choose "Whole Words" and search for "proc" you find "tss_proc". I looked through the configuration options but couldn't find anywhere to change this. It seems odd because that's not C++ specific. If anyone knows how to "fix" this, let me know.
Labels:
ace,
boehm gc,
concurrency,
suneido
Firefox 3 Beta
I've been holding off on trying the Firefox 3 beta. I used to be an earlier adopter but now that so much of my computing life goes through my Firefox browser I'm a little more reluctant to live on the bleeding edge.
But beta 5 is supposed to be the final beta before the release candidate so I thought I'd give it a try - as long as I could install it alongside my current Firefox. On Windows I used Portable Firefox but I couldn't find anything similar for the Mac. Instead I created a second profile and have Firefox prompt for the profile when it starts up. This works but is a bit of a hassle and I'm afraid I'll choose the wrong profile at some point.
A pleasant surprise on the Mac is that it finally has a more Mac-like theme. e.g. buttons on web pages are rounded instead of ugly rectangles.
Unfortunately, it still doesn't show the favicons on the bookmark bar. I guess this is to be compatible with Safari, but that seems pretty silly to me. But I found the Mac bookmark toolbar favicons style for Stylish which solved this issue.
If you asked me if I needed a customizable browser my first response would be "no". But when I go back to a "bare" browser I realize how much I depend on the add-ons I use. Of course, not all the add-ons have Firefox 3 versions yet. dragdropupload (which I use all the time for email attachments) was available.
But the Google Firefox Browser Sync isn't, which was a real hassle because it meant I didn't have any of my bookmarks or passwords. I don't have many bookmarks since I keep the bulk of them on del.icio.us so it was easy to export from Firefox 2 and import into Firefox 3 (once on Windows and once on Mac). I haven't found a way to export and import my passwords.
Passwords was another pleasant change in Firefox 3. I have a hard time remembering all my passwords for all the web sites I use, especially since I try not to use the same password for all of them. On Firefox 2 after you enter a password a dialog pops up asking if you want to remember it. But at this point you don't know if it's right or not. Either you say "remember" and then if it's wrong end up storing a bad password. Or you say "don't remember", in which case, if it works, if you want to remember it, you have to log out and log back in. Either way it's not ideal. But in Firefox 3, the "remember" prompt is inserted in a bar at the top of the screen mode-lessly - meaning you can wait to see if the password is accepted before choosing to remember it or not. Great improvement.
The Delicious Bookmarks add-on for Firefox 3 is a beta version that requires you to join the Yahoo group before you can download it. No big deal, but more hassle than necessary.
Firefox 3 is supposed to be a lot faster than Firefox 2. I can't say I've really noticed a big difference, but I haven't timed anything.
I haven't had any real problems with the beta so far. I crashed once in a couple of days, which isn't any worse than Firefox 2. I'll continue testing for a bit longer, but it seems good enough to switch over.
But beta 5 is supposed to be the final beta before the release candidate so I thought I'd give it a try - as long as I could install it alongside my current Firefox. On Windows I used Portable Firefox but I couldn't find anything similar for the Mac. Instead I created a second profile and have Firefox prompt for the profile when it starts up. This works but is a bit of a hassle and I'm afraid I'll choose the wrong profile at some point.
A pleasant surprise on the Mac is that it finally has a more Mac-like theme. e.g. buttons on web pages are rounded instead of ugly rectangles.
Unfortunately, it still doesn't show the favicons on the bookmark bar. I guess this is to be compatible with Safari, but that seems pretty silly to me. But I found the Mac bookmark toolbar favicons style for Stylish which solved this issue.
If you asked me if I needed a customizable browser my first response would be "no". But when I go back to a "bare" browser I realize how much I depend on the add-ons I use. Of course, not all the add-ons have Firefox 3 versions yet. dragdropupload (which I use all the time for email attachments) was available.
But the Google Firefox Browser Sync isn't, which was a real hassle because it meant I didn't have any of my bookmarks or passwords. I don't have many bookmarks since I keep the bulk of them on del.icio.us so it was easy to export from Firefox 2 and import into Firefox 3 (once on Windows and once on Mac). I haven't found a way to export and import my passwords.
Passwords was another pleasant change in Firefox 3. I have a hard time remembering all my passwords for all the web sites I use, especially since I try not to use the same password for all of them. On Firefox 2 after you enter a password a dialog pops up asking if you want to remember it. But at this point you don't know if it's right or not. Either you say "remember" and then if it's wrong end up storing a bad password. Or you say "don't remember", in which case, if it works, if you want to remember it, you have to log out and log back in. Either way it's not ideal. But in Firefox 3, the "remember" prompt is inserted in a bar at the top of the screen mode-lessly - meaning you can wait to see if the password is accepted before choosing to remember it or not. Great improvement.
The Delicious Bookmarks add-on for Firefox 3 is a beta version that requires you to join the Yahoo group before you can download it. No big deal, but more hassle than necessary.
Firefox 3 is supposed to be a lot faster than Firefox 2. I can't say I've really noticed a big difference, but I haven't timed anything.
I haven't had any real problems with the beta so far. I crashed once in a couple of days, which isn't any worse than Firefox 2. I'll continue testing for a bit longer, but it seems good enough to switch over.
Labels:
mac
Tuesday, April 22, 2008
Wireless Security
Sometimes it seems like I go out of my way to make things hard for myself!
I got home from work to find a note from Shelley that her laptop wireless wasn't working. Oh yeah, I hadn't got it working with the new Time Capsule.
I run the Airport Utility, change the Wireless Security to WEP, and try to set the password to what I was using previously (so I didn't have to mess with the laptops or other devices).
Nope, it will only take an exactly 13 character password. Okay, I make up a new password and go to connect Shelley's Windows laptop. But it won't work. I mess around with the settings, stupidly failing to record the original settings. No luck.
I search on the internet. Some people say that on Windows you have to use the hex version of the password (you get that from the AirPort Utility from Equivalent Network Password on the Base Station menu). I type in a 26 character hex password multiple times. No luck.
More digging reveals that this is 128 bit WEP. (More or less. It's also Transitional Network Security that is also WPA compatible. Whatever that means.) The older 40 bit WEP that I was using previously is actually available - you just have to turn off the new 802.11n :-( and then hold down the option key while you pull down the Radio Mode list. (how could I have missed that!)
Now I can enter the old 10 digit hex password (with a dollar sign). And now the Windows laptop connects.
I pull out the Chumby to make sure it can still connect. It can, but in the process I find that it does handle WPA! So I don't need to use WEP after all. Argh! (It must have been my old Squeezebox audio player that was WEP only, but I'm not using it any more - too much hassle to run its proprietary server.)
Back to the Airport Utility, re-enable 802.11n, set it back to the default WPA. Back to the Windows laptop, enter the new password, it connects fine. Back to the Chumby, enter the new password, it works fine. Same with my MacBook.
So I think I'm all good. I'm glad my Mac mini is hard wired to the Time Capsule. Trying to configure wireless through a wireless connection would have added even more fun to the process!
I got home from work to find a note from Shelley that her laptop wireless wasn't working. Oh yeah, I hadn't got it working with the new Time Capsule.
I run the Airport Utility, change the Wireless Security to WEP, and try to set the password to what I was using previously (so I didn't have to mess with the laptops or other devices).
Nope, it will only take an exactly 13 character password. Okay, I make up a new password and go to connect Shelley's Windows laptop. But it won't work. I mess around with the settings, stupidly failing to record the original settings. No luck.
I search on the internet. Some people say that on Windows you have to use the hex version of the password (you get that from the AirPort Utility from Equivalent Network Password on the Base Station menu). I type in a 26 character hex password multiple times. No luck.
More digging reveals that this is 128 bit WEP. (More or less. It's also Transitional Network Security that is also WPA compatible. Whatever that means.) The older 40 bit WEP that I was using previously is actually available - you just have to turn off the new 802.11n :-( and then hold down the option key while you pull down the Radio Mode list. (how could I have missed that!)
Now I can enter the old 10 digit hex password (with a dollar sign). And now the Windows laptop connects.
I pull out the Chumby to make sure it can still connect. It can, but in the process I find that it does handle WPA! So I don't need to use WEP after all. Argh! (It must have been my old Squeezebox audio player that was WEP only, but I'm not using it any more - too much hassle to run its proprietary server.)
Back to the Airport Utility, re-enable 802.11n, set it back to the default WPA. Back to the Windows laptop, enter the new password, it connects fine. Back to the Chumby, enter the new password, it works fine. Same with my MacBook.
So I think I'm all good. I'm glad my Mac mini is hard wired to the Time Capsule. Trying to configure wireless through a wireless connection would have added even more fun to the process!
Monday, April 21, 2008
New Time Capsule
I just replaced my Linksys wireless router and Lacie network drive with a 1 tb Apple Time Capsule. My reasons for the change included:
- The Linksys router periodically "dies". Even my wife knows to unplug it, wait a few seconds, and plug it back in, but it's still a hassle. Since this fixes the problem I'm assuming it's the router, but I suppose it's possible it's something like the ISP DHCP. I'm crossing my fingers that a new router will fix the problem.
- The Lacie network drive was getting full.
- I was using a second Firewire Lacie drive for my Time Machine backups from my Mac mini. This drive was also running out of space.
- I wanted to make Time Machine backups of my MacBook, and the only way to do that without plugging in an external drive is with Time Capsule. (Time Machine won't work to a regular network drive. I'm not sure if that's just a marketing decision or whether there is some technical reason as well.)
- I could replace two boxes (router & network drive) with one - the less cables the better, if you ask me.
- I occasionally had range problems with wireless. I'm hoping the new router will help.
As usual, the packaging was slick and the hardware is attractive. I plugged it in, installed the software, and it worked.
At first I couldn't see the drive from the Windows machine but after I set the right workgroup it appeared.
The setup wizard only allowed WPA wireless security. I was still using WEP since I have some wireless devices that don't support WPA (like my Chumby). I realize WEP is minimal security but it's enough to stop my neighbors from accidentally using my network. I have used MAC address filtering in addition but it's a hassle when you add new devices and I always seem to end up turning it off. I see the regular configuration allows WEP but I haven't got around to changing it yet.
To keep things simple (at least I assume that's the reason), OS X only seems to offer a single "key" entry field. I'm never quite sure if this is hex or text or what. If you were Apple/Mac only it wouldn't matter - it just works. But when you're connecting to other things it can be confusing. I've seen references to using a dollar sign or 0x prefix to enter hex keys but it doesn't seem to be clearly documented (that I've seen).
Once I had it set up I switched my Mac mini Time Machine to go to the Time Capsule. Of course, the first initial backup is huge and takes a long time. And there's no way to move your existing Time Machine backups to the new drive.
I probably should have know better, but while that was chugging away I turned on Time Machine on my MacBook and pointed it at the Time Capsule. Of course, it also was a huge initial backup.
At the same time (I know, asking for trouble!) I was playing with the Time Machine options to exclude certain files and directories. Then I realized every time I changed the settings the initial backup started all over again!
Just to stress it a little more, I started copying the other files (pictures and music) from the old network drive to the new one. It took me a few minutes to get the network drive working again - it wouldn't show up until I re-ran the configuration utility. This copy ran for quite a while (an hour maybe?) and then aborted with an Error 50. I suspect my impromptu stress test uncovered some bug in the Time Capsule software.
When I went to bed the two Time Machine backups were still running. When I got up, the MacBook had finished. Unfortunately, I'd forgotten I had the Mac mini set to power off at night. When I started it up, it had to restart the backup from the very beginning! I realize this is just the initial backup, but given how long it takes, you'd think they would have made the software handle resuming a backup. This time, without all the other concurrent activity, the backup went smoothly and finished.
Or maybe I should say, more or less finished. For several more hours it popped up windows about backup up large quantities of files. I'm not sure what this was - I hadn't modified or added any significant amount of files. Why didn't it get this stuff on the initial backup? But eventually it seemed to settle down. The problem with this kind of system that just invisibly does stuff in the background is that you're never quite sure if it's working properly (at least if you're a cynical techie).
Once the backups were done I went back and copied the other data from the old network drive. Again, now that I wasn't trying to do too many things at once, it went smoothly. (Theoretically, if the software is "correct" it shouldn't matter how much stuff you do at once. But no software is "correct". As the saying goes, in theory, practice should be the same as theory, but in practice, it's not.)
So far so good. My only (minor) complaint is that, judging by the temperature of the case, the Time Capsule doesn't seem to go to "sleep" - even overnight, with no computers active (or even turned on). There's probably continuous activity on the internet side, but if nothing is awake on the LAN side, I would think it could still be smart enough to go to sleep.
One nice side benefit is that the Time Capsule hard drive is a lot quieter than the Lacie.
I know it's showing my age but I can't help continuing to be a little mind boggled by gigabytes of memory and terabytes of hard drive space. (I just listened to a podcast that said that in many cases algorithms had progress more than hardware and in these cases you'd be better off with a modern algorithm on old hardware than old algorithms on modern hardware. In theory that might be true in some cases, but modern software is never going to fit on old hardware. I can't even fit a single digital picture on a floppy disk, let alone something like Open Office.)
My next plan is to replace my Mac mini with a 24" iMac. I considered a quad core Mac Pro for not a lot more money, but I decided I'd prefer the reduced "clutter" of the iMac. And if I wanted an Apple monitor, then the Pro would end up quite a bit more expensive. When I bought the mini it was more in the nature of an experiment so I bought the cheapest Mac I could. Now that I've pretty much converted to Mac I want something a little "bigger" i.e. 4 gb ram, 1 tb disk. Nothing like RAW photos and virtual machine images to eat up disk space!
- The Linksys router periodically "dies". Even my wife knows to unplug it, wait a few seconds, and plug it back in, but it's still a hassle. Since this fixes the problem I'm assuming it's the router, but I suppose it's possible it's something like the ISP DHCP. I'm crossing my fingers that a new router will fix the problem.
- The Lacie network drive was getting full.
- I was using a second Firewire Lacie drive for my Time Machine backups from my Mac mini. This drive was also running out of space.
- I wanted to make Time Machine backups of my MacBook, and the only way to do that without plugging in an external drive is with Time Capsule. (Time Machine won't work to a regular network drive. I'm not sure if that's just a marketing decision or whether there is some technical reason as well.)
- I could replace two boxes (router & network drive) with one - the less cables the better, if you ask me.
- I occasionally had range problems with wireless. I'm hoping the new router will help.
As usual, the packaging was slick and the hardware is attractive. I plugged it in, installed the software, and it worked.
At first I couldn't see the drive from the Windows machine but after I set the right workgroup it appeared.
The setup wizard only allowed WPA wireless security. I was still using WEP since I have some wireless devices that don't support WPA (like my Chumby). I realize WEP is minimal security but it's enough to stop my neighbors from accidentally using my network. I have used MAC address filtering in addition but it's a hassle when you add new devices and I always seem to end up turning it off. I see the regular configuration allows WEP but I haven't got around to changing it yet.
To keep things simple (at least I assume that's the reason), OS X only seems to offer a single "key" entry field. I'm never quite sure if this is hex or text or what. If you were Apple/Mac only it wouldn't matter - it just works. But when you're connecting to other things it can be confusing. I've seen references to using a dollar sign or 0x prefix to enter hex keys but it doesn't seem to be clearly documented (that I've seen).
Once I had it set up I switched my Mac mini Time Machine to go to the Time Capsule. Of course, the first initial backup is huge and takes a long time. And there's no way to move your existing Time Machine backups to the new drive.
I probably should have know better, but while that was chugging away I turned on Time Machine on my MacBook and pointed it at the Time Capsule. Of course, it also was a huge initial backup.
At the same time (I know, asking for trouble!) I was playing with the Time Machine options to exclude certain files and directories. Then I realized every time I changed the settings the initial backup started all over again!
Just to stress it a little more, I started copying the other files (pictures and music) from the old network drive to the new one. It took me a few minutes to get the network drive working again - it wouldn't show up until I re-ran the configuration utility. This copy ran for quite a while (an hour maybe?) and then aborted with an Error 50. I suspect my impromptu stress test uncovered some bug in the Time Capsule software.
When I went to bed the two Time Machine backups were still running. When I got up, the MacBook had finished. Unfortunately, I'd forgotten I had the Mac mini set to power off at night. When I started it up, it had to restart the backup from the very beginning! I realize this is just the initial backup, but given how long it takes, you'd think they would have made the software handle resuming a backup. This time, without all the other concurrent activity, the backup went smoothly and finished.
Or maybe I should say, more or less finished. For several more hours it popped up windows about backup up large quantities of files. I'm not sure what this was - I hadn't modified or added any significant amount of files. Why didn't it get this stuff on the initial backup? But eventually it seemed to settle down. The problem with this kind of system that just invisibly does stuff in the background is that you're never quite sure if it's working properly (at least if you're a cynical techie).
Once the backups were done I went back and copied the other data from the old network drive. Again, now that I wasn't trying to do too many things at once, it went smoothly. (Theoretically, if the software is "correct" it shouldn't matter how much stuff you do at once. But no software is "correct". As the saying goes, in theory, practice should be the same as theory, but in practice, it's not.)
So far so good. My only (minor) complaint is that, judging by the temperature of the case, the Time Capsule doesn't seem to go to "sleep" - even overnight, with no computers active (or even turned on). There's probably continuous activity on the internet side, but if nothing is awake on the LAN side, I would think it could still be smart enough to go to sleep.
One nice side benefit is that the Time Capsule hard drive is a lot quieter than the Lacie.
I know it's showing my age but I can't help continuing to be a little mind boggled by gigabytes of memory and terabytes of hard drive space. (I just listened to a podcast that said that in many cases algorithms had progress more than hardware and in these cases you'd be better off with a modern algorithm on old hardware than old algorithms on modern hardware. In theory that might be true in some cases, but modern software is never going to fit on old hardware. I can't even fit a single digital picture on a floppy disk, let alone something like Open Office.)
My next plan is to replace my Mac mini with a 24" iMac. I considered a quad core Mac Pro for not a lot more money, but I decided I'd prefer the reduced "clutter" of the iMac. And if I wanted an Apple monitor, then the Pro would end up quite a bit more expensive. When I bought the mini it was more in the nature of an experiment so I bought the cheapest Mac I could. Now that I've pretty much converted to Mac I want something a little "bigger" i.e. 4 gb ram, 1 tb disk. Nothing like RAW photos and virtual machine images to eat up disk space!
Tools, NetBeans, and Eclipse
I've been pushing my Ruby on Rails programmer to use some kind of IDE instead of just an editor. I suggested NetBeans since I'd read some good things about it. He installed it, but so far I haven't convinced him to switch. I can sympathize, when you've got a comfortable routine it's a hassle to change. And productivity tends to take a dip at first while you're learning new tools. But in the end, better tools can make a big difference. Many of the new tools we've added to Suneido have become so useful that I'd be really annoyed to lose them. (Even though we went years without them, and without knowing what we were missing.)
Soon after, I found myself working on the Suneido C++ source code with ... just an editor. Hmmm. I decided maybe I should give NetBeans a try myself. I also figured I should see what progress there had been on the C++ tools (CDT) in Eclipse. Both NetBeans and Eclipse have had many improvements. I downloaded, installed, and tried both. My conclusions were similar to Eclipse 3.3 or NetBeans 6.0.
As much as I like the concepts of Eclipse, it's got some awkward aspects (like "workspaces"). With NetBeans I was able to get the Suneido code loaded with little trouble. I ended up giving up on Eclipse. I'm sure I could have figured it out in the end (I have in the past) but I wasn't in the mood to spend the time.
It's interesting that both these IDE's are written in Java and run on Windows, Linux, and Mac.
Of course, in the end, I used up the time I had messing with IDE's and didn't make much progress on my code. Was this an excuse to goof off or was it an investment in better tools? Depends whether it pays off in the future I guess.
Soon after, I found myself working on the Suneido C++ source code with ... just an editor. Hmmm. I decided maybe I should give NetBeans a try myself. I also figured I should see what progress there had been on the C++ tools (CDT) in Eclipse. Both NetBeans and Eclipse have had many improvements. I downloaded, installed, and tried both. My conclusions were similar to Eclipse 3.3 or NetBeans 6.0.
As much as I like the concepts of Eclipse, it's got some awkward aspects (like "workspaces"). With NetBeans I was able to get the Suneido code loaded with little trouble. I ended up giving up on Eclipse. I'm sure I could have figured it out in the end (I have in the past) but I wasn't in the mood to spend the time.
It's interesting that both these IDE's are written in Java and run on Windows, Linux, and Mac.
Of course, in the end, I used up the time I had messing with IDE's and didn't make much progress on my code. Was this an excuse to goof off or was it an investment in better tools? Depends whether it pays off in the future I guess.
Tuesday, April 08, 2008
Too many heap sections
Recently, a couple of our clients with big databases (for us that's over 4 gb) have been unable to repair their database after crashing. (Why they crashed in the first place is another question - there may be hardware problems.) The repair aborts with "Too many heap sections". Luckily Thankfully, the software does automatic backups twice a day and the customer didn't lose much data. But the repair is still preferable as it generally only loses incomplete transactions that were in progress at the time of the crash.
This error comes from the Boehm memory manager/garbage collector we use. The repair process does keep a lot of information in memory and the amount of information is relative to the size of the database. The question was how to fix it? Did I need to rewrite the repair process to keep less information in memory, maybe use a temporary file? (Although that would make it slower.) Or was there something that could be adjusted in the Boehm code? There's also a newer version of the Boehm code - we're on 6.5 and the latest is 7.0
I searched on the web but didn't find any useful information about this error. Most mentions of it were pretty old. (I did find a reference to the Boehm code with the Mac OS X code - I wonder what part of OS X uses it?)
I searched the Boehm code for the error message and found it in several places (not very DRY). The error is caused when MAX_HEAP_SECTS is exceeded. I searched for where that is defined and the value seemed to depend on whether SMALL_CONFIG or LARGE_CONFIG (or neither) was defined. I wasn't specifically defining either, presumably leading to a medium setting.
I figured it was worth a try re-compiling with LARGE_CONFIG. I modified the makefile and re-built. Then I read a magazine while I tried running the repair process. (It takes a while to process a 4 gb database, even with a fast computer with lots of memory.) It reminded me of the "old" days when I'd have time to catch up on my reading while I compiled what today would be regarded as tiny C programs.
Damn, it still crashed with the same error. Oh well, should have known it wouldn't be that easy.
I went back to remove the setting from the makefile and glanced at the file name. Hey! I was editing the makefile for the MinGW version, but I'd been testing the VC7 version. Doh!
Try again, this time build the right version & test the version I build.
Eureka! It completed successfully.
Now we'll just have to test this version enough to be relatively sure that the change doesn't have any unwanted side effects. Maybe it will be that easy after all!
Sidenote: Considering the number of people and projects using the Boehm code (e.g. Mono) it seems odd that there isn't more documentation. I can understand Boehm not writing it, I'd rather he spent his time on the code. But you'd think someone along the way would have written some. Maybe no one else understands it well enough. Despite having written my own, I know I don't - I just treat it as a black box.
This error comes from the Boehm memory manager/garbage collector we use. The repair process does keep a lot of information in memory and the amount of information is relative to the size of the database. The question was how to fix it? Did I need to rewrite the repair process to keep less information in memory, maybe use a temporary file? (Although that would make it slower.) Or was there something that could be adjusted in the Boehm code? There's also a newer version of the Boehm code - we're on 6.5 and the latest is 7.0
I searched on the web but didn't find any useful information about this error. Most mentions of it were pretty old. (I did find a reference to the Boehm code with the Mac OS X code - I wonder what part of OS X uses it?)
I searched the Boehm code for the error message and found it in several places (not very DRY). The error is caused when MAX_HEAP_SECTS is exceeded. I searched for where that is defined and the value seemed to depend on whether SMALL_CONFIG or LARGE_CONFIG (or neither) was defined. I wasn't specifically defining either, presumably leading to a medium setting.
I figured it was worth a try re-compiling with LARGE_CONFIG. I modified the makefile and re-built. Then I read a magazine while I tried running the repair process. (It takes a while to process a 4 gb database, even with a fast computer with lots of memory.) It reminded me of the "old" days when I'd have time to catch up on my reading while I compiled what today would be regarded as tiny C programs.
Damn, it still crashed with the same error. Oh well, should have known it wouldn't be that easy.
I went back to remove the setting from the makefile and glanced at the file name. Hey! I was editing the makefile for the MinGW version, but I'd been testing the VC7 version. Doh!
Try again, this time build the right version & test the version I build.
Eureka! It completed successfully.
Now we'll just have to test this version enough to be relatively sure that the change doesn't have any unwanted side effects. Maybe it will be that easy after all!
Sidenote: Considering the number of people and projects using the Boehm code (e.g. Mono) it seems odd that there isn't more documentation. I can understand Boehm not writing it, I'd rather he spent his time on the code. But you'd think someone along the way would have written some. Maybe no one else understands it well enough. Despite having written my own, I know I don't - I just treat it as a black box.
More Software Frustrations
Here's a few software things that have bugged me lately:
I'm currently reading Why Software Sucks by David Platt. He praises how Google senses the country you are connecting from and uses the language of that country e.g. Spanish in Mexico. The problem is, it does this even if I am logged in to my account. Just because I am in another country doesn't mean I've changed my language. It's possible to get back to English but it's annoying nonetheless.
Next, I'm on my bank web site paying some bills (a feature I do appreciate). A message shows up telling me I can now receive one of my bills electronically, rather than having it mailed to me. Great! I click on the link and get presented with a blank form, without the name of the company filled in, forcing me to pick it from a huge list. (Most of the list is not applicable, e.g. for other provinces.) Next there is a "Name" field - who's name? mine? but surely it knows my name? or the company name? but I just picked it from a list? I leave it blank and continue - it doesn't complain so I'm still not sure what it wanted. Next it tells me I have to sign up for an ePost account. It's free, but it's annoying to have to sign up for yet another on-line account and come up with yet another password. Can't the company send bills to my bank without every account holder having to sign up for ePost?
But the best is yet to come. I get through the ePost signup, only to be told that I have to contact the company personally, by phone, before I can complete the process. Yeah, right. I pay bills on the evenings and weekends, so I have to wait till business hours to phone. Then I can look forward to the "pleasure" of an automated phone system and being on hold forever. Then I'll likely talk to some poor customer service person who has never heard of sending bills electronically. No thanks. And they wonder why so many on line transactions are abandoned part way through.
The next incident was minor, but it's a good example of a GUI blooper. I was ordering books through Lulu (copies of Getting Real for my programmers). I'm entering my address and I get to the field for "State/Province". Except there are no provinces. That wouldn't surprise me except that for the prompt. I skip it and move to the next field, "Country", and enter Canada. The screen jumps around a little bit. Hmmm... sure enough, State/Province now lists provinces. This is a good trick, some programmer obviously applied some Javascript. But they didn't consider that people generally enter fields in order, pretty much guaranteeing that Canadian customers will be frustrated. And most people wouldn't notice the slight screen jump and figure out that they could go back to a previous field and it would now magically let them do what it wouldn't just a minute ago.
We run into this issue in our own software. The rule of thumb is that if a field "depends" on another field it should be "after" it in the normal entry order. Usually this can be handled simply by changing the order of the fields. But in this case, it seems "wrong" to put Country before State/Province. My suggestion would be to remove the fancy Javascript - just have a combined list. The programmer can apply their Javascript by filling in the country based on the State/Province choice.
We use Snagit for our screenshots. It's a good program. But it has one really annoying "feature". Quite frequently, when I start it up, it pops up a dialog saying "You have the most recent version of Snagit". I always have to read this a couple of times because the normal expectation is that it would pop up to tell you there's a newer version.
I haven't quite figured out the logic behind this one. Why do I need to be told I have the latest version? I don't think it comes up every time, so maybe it only comes up after it does a periodic check for a new version. (Of course, I only use it periodically, so it happens more often than not.) It may just be that the programmer thought that since they'd gone to all the work of checking for a new version there should be some recognition of this. Programmers are often strangely reluctant to just have the program quietly do it's stuff. Maybe we need pop up blockers for more than just our web browsers.
There is a link on the dialog to the release notes. So perhaps the purpose of the pop up is to give you a chance to read the release notes. But surely if that was the purpose it would only pop up once after installing a new version. How many times (if any) do I want to read the release notes?
Enough being critical, I'd better go and try to do better myself. It's always easier to criticize. (Why Software Sucks does actually suggest solutions or at least, better alternatives, to most of the things it complains about.)
I'm currently reading Why Software Sucks by David Platt. He praises how Google senses the country you are connecting from and uses the language of that country e.g. Spanish in Mexico. The problem is, it does this even if I am logged in to my account. Just because I am in another country doesn't mean I've changed my language. It's possible to get back to English but it's annoying nonetheless.
Next, I'm on my bank web site paying some bills (a feature I do appreciate). A message shows up telling me I can now receive one of my bills electronically, rather than having it mailed to me. Great! I click on the link and get presented with a blank form, without the name of the company filled in, forcing me to pick it from a huge list. (Most of the list is not applicable, e.g. for other provinces.) Next there is a "Name" field - who's name? mine? but surely it knows my name? or the company name? but I just picked it from a list? I leave it blank and continue - it doesn't complain so I'm still not sure what it wanted. Next it tells me I have to sign up for an ePost account. It's free, but it's annoying to have to sign up for yet another on-line account and come up with yet another password. Can't the company send bills to my bank without every account holder having to sign up for ePost?
But the best is yet to come. I get through the ePost signup, only to be told that I have to contact the company personally, by phone, before I can complete the process. Yeah, right. I pay bills on the evenings and weekends, so I have to wait till business hours to phone. Then I can look forward to the "pleasure" of an automated phone system and being on hold forever. Then I'll likely talk to some poor customer service person who has never heard of sending bills electronically. No thanks. And they wonder why so many on line transactions are abandoned part way through.
The next incident was minor, but it's a good example of a GUI blooper. I was ordering books through Lulu (copies of Getting Real for my programmers). I'm entering my address and I get to the field for "State/Province". Except there are no provinces. That wouldn't surprise me except that for the prompt. I skip it and move to the next field, "Country", and enter Canada. The screen jumps around a little bit. Hmmm... sure enough, State/Province now lists provinces. This is a good trick, some programmer obviously applied some Javascript. But they didn't consider that people generally enter fields in order, pretty much guaranteeing that Canadian customers will be frustrated. And most people wouldn't notice the slight screen jump and figure out that they could go back to a previous field and it would now magically let them do what it wouldn't just a minute ago.
We run into this issue in our own software. The rule of thumb is that if a field "depends" on another field it should be "after" it in the normal entry order. Usually this can be handled simply by changing the order of the fields. But in this case, it seems "wrong" to put Country before State/Province. My suggestion would be to remove the fancy Javascript - just have a combined list. The programmer can apply their Javascript by filling in the country based on the State/Province choice.
We use Snagit for our screenshots. It's a good program. But it has one really annoying "feature". Quite frequently, when I start it up, it pops up a dialog saying "You have the most recent version of Snagit". I always have to read this a couple of times because the normal expectation is that it would pop up to tell you there's a newer version.
I haven't quite figured out the logic behind this one. Why do I need to be told I have the latest version? I don't think it comes up every time, so maybe it only comes up after it does a periodic check for a new version. (Of course, I only use it periodically, so it happens more often than not.) It may just be that the programmer thought that since they'd gone to all the work of checking for a new version there should be some recognition of this. Programmers are often strangely reluctant to just have the program quietly do it's stuff. Maybe we need pop up blockers for more than just our web browsers.
There is a link on the dialog to the release notes. So perhaps the purpose of the pop up is to give you a chance to read the release notes. But surely if that was the purpose it would only pop up once after installing a new version. How many times (if any) do I want to read the release notes?
Enough being critical, I'd better go and try to do better myself. It's always easier to criticize. (Why Software Sucks does actually suggest solutions or at least, better alternatives, to most of the things it complains about.)
Labels:
user experience
Monday, April 07, 2008
MindMeister Online Mind Mapping
I came across MindMeister from a list of web applications that use Google Gears to run off-line. Check out the demo screencast for a quick overview. It's free for a basic account.
From a quick look MindMeister looks like a good example of a Web 2.0 application.
I haven't got into using mind mapping but some people swear by it.
From a quick look MindMeister looks like a good example of a Web 2.0 application.
I haven't got into using mind mapping but some people swear by it.
Monday, March 31, 2008
Lightroom & Photo Workflow
I seem to have settled into a reasonable work flow with my photography and Lightroom.
I have multiple memory cards for each camera. Two 2 gb cards for the Canon SD700 IS - shooting jpeg gives me about 700 photos per card or about 15 min. of video at the highest quality (640x480, 30fps). I can't remember the last time I filled up one of these cards. I have four 4 gb cards for the Pentax K10D - shooting DNG (raw) I get about 240 photos per card and I do fill these up sometimes. For the Pentax I always carry at least one spare memory card and a spare battery.
I have spare batteries for both cameras but seldom need them. The last a long time and most of the time it's easy to recharge them. The exception is in more remote locations where power isn't readily available.
When traveling I usually download/import once a day to Lightroom on my MacBook. I take advantage of Lightroom's option to make a backup as it imports, to an external hard drive. I also have Lightroom set to back up its catalog to the external drive daily. When I fly I keep the MacBook and the external drive separate to improve my chances of not losing both!
I also separate the cameras (and their chargers and other accessories) so even if one bag gets lost I'll still have a usable camera. I worry as much about the chargers since they'd be impossible to replace in most places and without them the camera would soon become useless. In this respect I like the use of standard AA batteries in my Canon S3. I still use rechargeables but I can use readily available disposables in an emergency, and there's also a much better chance of replacing AA rechargeables and recharger.
After I download I do NOT erase the memory cards and I rotate them so I put in the least recently used card. That way the cards provide a third short term backup in case I mess up an import. I erase a card when I put it into the camera. The only drawback to this system is that occasionally I forget to erase the card when I put it in the camera. I then run out of space soon after I start shooting, but I can't erase the card at this point because I've put new pictures on it. At least Lightroom has an option to ignore already imported photos.
My primary Lightroom catalog and photos are on my desktop system at home. For each trip I start a new catalog on the MacBook just for that trip. When I get home I import that catalog into my main system. One minor drawback of this is that I start fresh with keyword tags and sometimes they don't match with the ones I'm using in my main catalog. But this is minor and easily fixed using Lightroom's ability to rename keyword tags.
I keep both the main and the MacBook catalog organized primarily by date with the yyyy/yyyy-mm-dd format. I rename as I import to yyyymmdd - filename. When I first starting using Lightroom I didn't rename but the straight camera generated file names aren't very helpful, and there's always the risk of duplicates. In my main catalog I add another level of folders for major trips e.g. 2008/2008 Baja/yyyy-mm-dd. This makes it easier to select all the photos for a trip e.g. to put together a slide show.
Some people keep their photos in folders by subject or type rather than date. To me recording subject or type is better done using tags. Otherwise you're continually faced with questions like which folder do I put a picture of a bird on a beach at sunset? In birds, beaches, or sunsets? That's the power of the ability to apply multiple tags, in this case birds, beaches, and sunsets. And it's just as easy to click on a birds tag as it is to click on a birds folder. Of course, this assumes you're inside Lightroom. Outside Lightroom you'd have no easy way to select the birds tag.
I'm not very systematic about my tagging. I'll do a little tagging when I first look at the photos. And I might go through and apply a tag (e.g. birds) if I want to make a selection of a certain type or subject. But lots of my photos aren't tagged at all, which makes it hard to find them later. I could do better in this area.
Another choice is whether to delete "bad" pictures e.g. badly exposed or out of focus. Personally, I don't delete, no matter how bad the photo is. I just flag it as "Rejected" and normally I filter rejected photos out of view. This is similar to Gmail's philosophy of keeping everything. Storage is cheap. It's not so much that you'll ever want a totally ruined photo, it's the time and mental effort needed to decide on the marginal cases. You can always change your mind about a Rejected photo, if you delete it, it's gone.
One case where I might be tempted to delete is when I'm bracketing exposures. Since they're more or less duplicates, why not just keep the best exposure? But again, it's not always clear which is "best". You might actually want to combine several exposures for a HDR image. And Lightroom's ability to "stack" photos avoids having to look at the "duplicates" most of the time.
I'm finding the K10D tends to overexpose bright scenes like snow or sand. I need to figure out when I need to override the exposure, by how much, and what the best way to do that is. I've been bracketing when I suspect there will be a problem. So far it looks like -1 stop does the trick. Lightroom's Recovery slider (and shooting raw) lets you fix a certain amount of overexposure, but it can't recover what really isn't there. This is where composing in the display in the small camera helps. I need to get in the habit of reviewing photos on the K10 and maybe even checking the histogram. Of course, that only works for static subjects where you have the option of another shot.
I also find that for certain shots e.g. closeups I need to manually focus the K10, especially when low light means a small depth of focus. The auto-focus is pretty good, but it can't read your mind and doesn't always pick the right thing to focus on.
On my main system I use Chronosync to automatically back up my photos and catalog to a network drive. And I also have TimeMachine running to an external drive. But currently I don't have a good offsite backup system. I keep a lot of my photos on my work computer, but I don't have a convenient (or automatic) way to do this so it's not too reliable. I've tried a few ways to sync my home and work computers (especially my photos and music), but it hasn't been too successful, partly because of the sheer volume of data to sync, especially at the start. An online internet backup is another option, but again, it's tough with the volume of data, again especially at the start. And not just at the start - when I come back from this trip I'll be adding 40 gb in one shot. That's a lot to transfer over the internet.
When traveling, I try to upload photos to the web every few days so my family and friends can keep up with me. So far I've been using Google Picasa Web Albums. Before Lightroom I was using Picassa to organize my photos, so it was natural to use it's web albums. It works well enough so I've just kept using it. And the Lightroom plugin to export directly to a web album makes it very easy. Now I keep thinking I should give Flickr a try.
I do a certain amount of tweaking of the photos I choose to post - mostly cropping and exposure. For this, and for general organizing, it's great to use the same software (Lightroom) traveling as well as at home.
You can check out some of the photos I've posted at : http://picasaweb.google.com/apmckinlay
I have multiple memory cards for each camera. Two 2 gb cards for the Canon SD700 IS - shooting jpeg gives me about 700 photos per card or about 15 min. of video at the highest quality (640x480, 30fps). I can't remember the last time I filled up one of these cards. I have four 4 gb cards for the Pentax K10D - shooting DNG (raw) I get about 240 photos per card and I do fill these up sometimes. For the Pentax I always carry at least one spare memory card and a spare battery.
I have spare batteries for both cameras but seldom need them. The last a long time and most of the time it's easy to recharge them. The exception is in more remote locations where power isn't readily available.
When traveling I usually download/import once a day to Lightroom on my MacBook. I take advantage of Lightroom's option to make a backup as it imports, to an external hard drive. I also have Lightroom set to back up its catalog to the external drive daily. When I fly I keep the MacBook and the external drive separate to improve my chances of not losing both!
I also separate the cameras (and their chargers and other accessories) so even if one bag gets lost I'll still have a usable camera. I worry as much about the chargers since they'd be impossible to replace in most places and without them the camera would soon become useless. In this respect I like the use of standard AA batteries in my Canon S3. I still use rechargeables but I can use readily available disposables in an emergency, and there's also a much better chance of replacing AA rechargeables and recharger.
After I download I do NOT erase the memory cards and I rotate them so I put in the least recently used card. That way the cards provide a third short term backup in case I mess up an import. I erase a card when I put it into the camera. The only drawback to this system is that occasionally I forget to erase the card when I put it in the camera. I then run out of space soon after I start shooting, but I can't erase the card at this point because I've put new pictures on it. At least Lightroom has an option to ignore already imported photos.
My primary Lightroom catalog and photos are on my desktop system at home. For each trip I start a new catalog on the MacBook just for that trip. When I get home I import that catalog into my main system. One minor drawback of this is that I start fresh with keyword tags and sometimes they don't match with the ones I'm using in my main catalog. But this is minor and easily fixed using Lightroom's ability to rename keyword tags.
I keep both the main and the MacBook catalog organized primarily by date with the yyyy/yyyy-mm-dd format. I rename as I import to yyyymmdd - filename. When I first starting using Lightroom I didn't rename but the straight camera generated file names aren't very helpful, and there's always the risk of duplicates. In my main catalog I add another level of folders for major trips e.g. 2008/2008 Baja/yyyy-mm-dd. This makes it easier to select all the photos for a trip e.g. to put together a slide show.
Some people keep their photos in folders by subject or type rather than date. To me recording subject or type is better done using tags. Otherwise you're continually faced with questions like which folder do I put a picture of a bird on a beach at sunset? In birds, beaches, or sunsets? That's the power of the ability to apply multiple tags, in this case birds, beaches, and sunsets. And it's just as easy to click on a birds tag as it is to click on a birds folder. Of course, this assumes you're inside Lightroom. Outside Lightroom you'd have no easy way to select the birds tag.
I'm not very systematic about my tagging. I'll do a little tagging when I first look at the photos. And I might go through and apply a tag (e.g. birds) if I want to make a selection of a certain type or subject. But lots of my photos aren't tagged at all, which makes it hard to find them later. I could do better in this area.
Another choice is whether to delete "bad" pictures e.g. badly exposed or out of focus. Personally, I don't delete, no matter how bad the photo is. I just flag it as "Rejected" and normally I filter rejected photos out of view. This is similar to Gmail's philosophy of keeping everything. Storage is cheap. It's not so much that you'll ever want a totally ruined photo, it's the time and mental effort needed to decide on the marginal cases. You can always change your mind about a Rejected photo, if you delete it, it's gone.
One case where I might be tempted to delete is when I'm bracketing exposures. Since they're more or less duplicates, why not just keep the best exposure? But again, it's not always clear which is "best". You might actually want to combine several exposures for a HDR image. And Lightroom's ability to "stack" photos avoids having to look at the "duplicates" most of the time.
I'm finding the K10D tends to overexpose bright scenes like snow or sand. I need to figure out when I need to override the exposure, by how much, and what the best way to do that is. I've been bracketing when I suspect there will be a problem. So far it looks like -1 stop does the trick. Lightroom's Recovery slider (and shooting raw) lets you fix a certain amount of overexposure, but it can't recover what really isn't there. This is where composing in the display in the small camera helps. I need to get in the habit of reviewing photos on the K10 and maybe even checking the histogram. Of course, that only works for static subjects where you have the option of another shot.
I also find that for certain shots e.g. closeups I need to manually focus the K10, especially when low light means a small depth of focus. The auto-focus is pretty good, but it can't read your mind and doesn't always pick the right thing to focus on.
On my main system I use Chronosync to automatically back up my photos and catalog to a network drive. And I also have TimeMachine running to an external drive. But currently I don't have a good offsite backup system. I keep a lot of my photos on my work computer, but I don't have a convenient (or automatic) way to do this so it's not too reliable. I've tried a few ways to sync my home and work computers (especially my photos and music), but it hasn't been too successful, partly because of the sheer volume of data to sync, especially at the start. An online internet backup is another option, but again, it's tough with the volume of data, again especially at the start. And not just at the start - when I come back from this trip I'll be adding 40 gb in one shot. That's a lot to transfer over the internet.
When traveling, I try to upload photos to the web every few days so my family and friends can keep up with me. So far I've been using Google Picasa Web Albums. Before Lightroom I was using Picassa to organize my photos, so it was natural to use it's web albums. It works well enough so I've just kept using it. And the Lightroom plugin to export directly to a web album makes it very easy. Now I keep thinking I should give Flickr a try.
I do a certain amount of tweaking of the photos I choose to post - mostly cropping and exposure. For this, and for general organizing, it's great to use the same software (Lightroom) traveling as well as at home.
You can check out some of the photos I've posted at : http://picasaweb.google.com/apmckinlay
Monday, March 24, 2008
Publishing Video
Everyone is posting their video these days, how hard can it be?
I shot some short video clips of a school of dolphins on my Canon SD700 IS.
The first hurdle was to combine them together. I fired up iMovie for the very first time and managed to put in the clips, crop them, and add transitions. So far so good.
I exported the movie and uploaded it to YouTube. The QuickTime mov file was about 100mb for 30 seconds so it took a long time (almost 2 hours) to upload it over the hotel DSL. It would be nice if you got some progress feedback - I started to wonder if it was working or not.
Finally I can play the YouTube video. Yuck. Shrunken down from 640x480 to YouTube size and highly compressed, it looked like crap.
I start looking for alternatives and I decide to try Google Video. One of the reasons is that it recommends 640x480. I try the Google Uploader this time. It still takes forever. And the Google Video version still looks like crap and it appears to have shrunk it even smaller than YouTube (pick "Original Size" on the pop up menu).
The most recommended format seemed to be mpeg4 so I started to look for a way to convert the file. One of the suggestions I found said to use iMovie. Sure enough, although it is far from obvious that iMovie can do this, there are instructions in the help. The key is to choose "Expert Settings". I guess only experts would want something other than QuickTime! It took some trial and error experimentation to get a file that looked decent. I ended up choosing Lan/Intranet, even though that isn't what I want. (I hate it when the "right" settings are obviously "wrong".) This mpeg4 file is under 6mb, a little better than the "default" QuickTime file.
Next I decided to try blip.tv only because I'd recently seen some videos published there. Their FAQ doesn't really say much about resolution or compression, but then again, neither do YouTube or Google Video. Their web upload gives good progress feedback - very nice. (And I'm at the coffee shop this time and the upload rate is better.) I watch the video and, eureka, it looks good. But it says it's still being processed (converted to Flash). When it finishes I watch the Flash version. But at least you can choose the format and if you pick the MPEG4 version the quality is better. I'm not sure if that's a limitation of the Flash format or if they just choose a higher compression. At least even the Flash version is the full 640x480 unlike YouTube and Google Video.
Google actually has a third option - uploading to Picasa Web Albums. But you have to use Picasa to upload video and they don't have a Mac version. I could have installed Picasa inside my Windows VM in Parallels but that seemed like too much hassle.
I'm a newbie at publishing video, and probably pickier than most people. But it seems like this technology is still pretty rough around the edges. Maybe there are ways to make YouTube or Google Video work better. Oh well, in the end I accomplished what I set out to.
I shot some short video clips of a school of dolphins on my Canon SD700 IS.
The first hurdle was to combine them together. I fired up iMovie for the very first time and managed to put in the clips, crop them, and add transitions. So far so good.
I exported the movie and uploaded it to YouTube. The QuickTime mov file was about 100mb for 30 seconds so it took a long time (almost 2 hours) to upload it over the hotel DSL. It would be nice if you got some progress feedback - I started to wonder if it was working or not.
Finally I can play the YouTube video. Yuck. Shrunken down from 640x480 to YouTube size and highly compressed, it looked like crap.
I start looking for alternatives and I decide to try Google Video. One of the reasons is that it recommends 640x480. I try the Google Uploader this time. It still takes forever. And the Google Video version still looks like crap and it appears to have shrunk it even smaller than YouTube (pick "Original Size" on the pop up menu).
The most recommended format seemed to be mpeg4 so I started to look for a way to convert the file. One of the suggestions I found said to use iMovie. Sure enough, although it is far from obvious that iMovie can do this, there are instructions in the help. The key is to choose "Expert Settings". I guess only experts would want something other than QuickTime! It took some trial and error experimentation to get a file that looked decent. I ended up choosing Lan/Intranet, even though that isn't what I want. (I hate it when the "right" settings are obviously "wrong".) This mpeg4 file is under 6mb, a little better than the "default" QuickTime file.
Next I decided to try blip.tv only because I'd recently seen some videos published there. Their FAQ doesn't really say much about resolution or compression, but then again, neither do YouTube or Google Video. Their web upload gives good progress feedback - very nice. (And I'm at the coffee shop this time and the upload rate is better.) I watch the video and, eureka, it looks good. But it says it's still being processed (converted to Flash). When it finishes I watch the Flash version. But at least you can choose the format and if you pick the MPEG4 version the quality is better. I'm not sure if that's a limitation of the Flash format or if they just choose a higher compression. At least even the Flash version is the full 640x480 unlike YouTube and Google Video.
Google actually has a third option - uploading to Picasa Web Albums. But you have to use Picasa to upload video and they don't have a Mac version. I could have installed Picasa inside my Windows VM in Parallels but that seemed like too much hassle.
I'm a newbie at publishing video, and probably pickier than most people. But it seems like this technology is still pretty rough around the edges. Maybe there are ways to make YouTube or Google Video work better. Oh well, in the end I accomplished what I set out to.
Remote Access with LogMeIn
I've been using LogMeIn to access my office computer while I'm traveling (to get access to things that are internal to the office). So far it's worked well, even accessing a Windows machine from my MacBook.
The only problem I've run into is that the Java file transfer doesn't work under Firefox - it locks up so you have to kill Firefox. This is mentioned on their support forum, but so far no solution. It works from Safari though, so it's not a big deal.
The basic version is free. I decided to pay for the Pro version since it was relatively cheap and I could use some of the Pro features.
There's a bit of a lag in typing, but that's not surprising considering I'm using wireless to dsl from Mexico to Canada. Occasionally it trips me up, but for my usage it's fine.
The only problem I've run into is that the Java file transfer doesn't work under Firefox - it locks up so you have to kill Firefox. This is mentioned on their support forum, but so far no solution. It works from Safari though, so it's not a big deal.
The basic version is free. I decided to pay for the Pro version since it was relatively cheap and I could use some of the Pro features.
There's a bit of a lag in typing, but that's not surprising considering I'm using wireless to dsl from Mexico to Canada. Occasionally it trips me up, but for my usage it's fine.
Thursday, March 20, 2008
First Time Skype
I finally got around to using Skype to talk to Shelley back in Saskatoon, Canada from Loreto, Baja Mexico where I am currently.
The first time we connected we couldn't hear each other, although we both heard the "ringing" in our headsets. At least we had the Chat facility to try to debug the problem. It was a little harder since I was on the Mac version and Shelley was on the Windows version, so the menu options are different. On the Windows version the settings are in a "Personalize" option - which seems like a rather odd choice of name. (On the Mac it's the standard Skype > Preferences)
Everything seemed ok so we tried again and this time it worked. Not sure why it didn't work the first time. The only thing I can think of was that Windows hadn't finished recognizing the USB headset (Shelley had only just plugged it in).
I could hear Shelley great. She said my sound quality wasn't so good. That might have been due to my bluetooth headset. The microphone is quite far from my mouth. And on top of that, I was outside on the deck (to reach the wireless) and there was a breeze and traffic noises. There's a definite lag time and you pretty much had to take turns talking, but it was definitely usable.
I wouldn't want to use it as my regular phone, but it beats dealing with long distance when I already have an internet connection.
Pretty amazing - bluetooth from headset to laptop, wireless from laptop, and then by internet across North America. And all for "free" :-) And more amazingly, given all the software involved, it works.
The first time we connected we couldn't hear each other, although we both heard the "ringing" in our headsets. At least we had the Chat facility to try to debug the problem. It was a little harder since I was on the Mac version and Shelley was on the Windows version, so the menu options are different. On the Windows version the settings are in a "Personalize" option - which seems like a rather odd choice of name. (On the Mac it's the standard Skype > Preferences)
Everything seemed ok so we tried again and this time it worked. Not sure why it didn't work the first time. The only thing I can think of was that Windows hadn't finished recognizing the USB headset (Shelley had only just plugged it in).
I could hear Shelley great. She said my sound quality wasn't so good. That might have been due to my bluetooth headset. The microphone is quite far from my mouth. And on top of that, I was outside on the deck (to reach the wireless) and there was a breeze and traffic noises. There's a definite lag time and you pretty much had to take turns talking, but it was definitely usable.
I wouldn't want to use it as my regular phone, but it beats dealing with long distance when I already have an internet connection.
Pretty amazing - bluetooth from headset to laptop, wireless from laptop, and then by internet across North America. And all for "free" :-) And more amazingly, given all the software involved, it works.
You Weren't Meant to Have a Boss
The latest essay by Paul Graham.
I've been my own boss for over 20 years now. And the only job I had before that was for a pretty small company where I had a lot of freedom. Nowadays, I can't imagine working for someone else - it's a frightening thought.
On the other hand, I've known lots of programmers (and other people) that want nothing more than to settle into a secure job in a large organization. To each his own, I guess.
I've been my own boss for over 20 years now. And the only job I had before that was for a pretty small company where I had a lot of freedom. Nowadays, I can't imagine working for someone else - it's a frightening thought.
On the other hand, I've known lots of programmers (and other people) that want nothing more than to settle into a secure job in a large organization. To each his own, I guess.
Sunday, March 16, 2008
Customer Service
I've been traveling recently. Travel gives you a great perspective on customer service. You're continually dealing with airlines, restaurants, hotels, etc. One of the things I've noticed lately is that you can divide people/companies into two types. One type wants to go by the book. If you ask them for something that doesn't fit into their predefined "boxes" their response is "sorry, can't help you". They might be really friendly about it, but they don't help you. The other type will look for a way to help you, even though you're asking for something unusual or something that isn't really their business.
For example, we go into several agencies and say we want to go out snorkeling to a different island than the usual one. Most of them give us the official response of "we don't go there" or "we don't have enough people who want to go there" and that's the end of it. But then we go to another agency and they say "sure, we can work something out, it may depend on the weather and what else we have booked, but we'll figure it out". That's the kind of service that brings me back for repeat business.
I just hope that my business is more like the second type. Obviously, you can't answer every single request but you can try to come up with something to offer. My staff will come to me and say "so and so wants xyz, what should I tell them". If possible, I'll say "give it to them". Lots of times it's not possible, and you can tell that the staff person just wants to say no. But often we can offer a partial solution or an alternative or a suggestion. It may or may not totally satisfy the customer, but at least you've tried.
Here's an interesting video from a customer service conference: (note: it's an hour long)
For example, we go into several agencies and say we want to go out snorkeling to a different island than the usual one. Most of them give us the official response of "we don't go there" or "we don't have enough people who want to go there" and that's the end of it. But then we go to another agency and they say "sure, we can work something out, it may depend on the weather and what else we have booked, but we'll figure it out". That's the kind of service that brings me back for repeat business.
I just hope that my business is more like the second type. Obviously, you can't answer every single request but you can try to come up with something to offer. My staff will come to me and say "so and so wants xyz, what should I tell them". If possible, I'll say "give it to them". Lots of times it's not possible, and you can tell that the staff person just wants to say no. But often we can offer a partial solution or an alternative or a suggestion. It may or may not totally satisfy the customer, but at least you've tried.
Here's an interesting video from a customer service conference: (note: it's an hour long)
Friday, March 07, 2008
Day 4 at Etech 2008
Started out with XO Laptop Hacks. I just got my OLPC XO not that long ago and I haven't had time to do much with it so it was interesting to hear more.
Next I went to the presentation on CouchDB. It's an interesting project - written in Erlang, primary interface JSON via HTTP, started by one guy on his own, but now IBM pays his salary.
I couldn't miss Jeff Jonas's talk on behind the scenes in Las Vegas - he's an entertaining presenter. His talk didn't have as much software content as last year, but it was still good.
Synthetic Neurobiology was largely on the theme of "body hacking". One of their more clever hacks was to introduce genes into neurons (via a virus) that make the cell sensitive to light. You can then trigger the neurons to fire (or suppress them from firing) using light. Wow.
The day (and the conference) ended with more keynotes. First by Alex Steffen of WorldChanging (who I was disappointed to see carrying a disposable plastic bottle of water!) Then a presentation on Twine - yet another social network app, but at least with some interesting semantic web aspects. A brief talk on Digital Democracy about the use of the web (and social networks, of course!) in politics.
And the final talk was by Timothy Ferriss of the (4-Hour Workweek). I'd seen Tim around during the conference, attending talks. My first impression was that he was short. A bit like seeing an famous actor and finding out how short they are. He also blended in pretty well carrying his backpack around, although without the ever present laptop of most people. But when he gave his talk he definitely had a bigger presence and charisma. Unlike every other speaker, he did not have any powerpoint slides - he just talked. If you've read his book and blog it was nothing new. Despite being a wing nut in many ways, I still find him "inspiring" if that's the right word. The gist of his message is that you don't have to be powerlessly overwhelmingly "busy". You can re-engineer your life to do what you want to do.
Next I went to the presentation on CouchDB. It's an interesting project - written in Erlang, primary interface JSON via HTTP, started by one guy on his own, but now IBM pays his salary.
I couldn't miss Jeff Jonas's talk on behind the scenes in Las Vegas - he's an entertaining presenter. His talk didn't have as much software content as last year, but it was still good.
Synthetic Neurobiology was largely on the theme of "body hacking". One of their more clever hacks was to introduce genes into neurons (via a virus) that make the cell sensitive to light. You can then trigger the neurons to fire (or suppress them from firing) using light. Wow.
The day (and the conference) ended with more keynotes. First by Alex Steffen of WorldChanging (who I was disappointed to see carrying a disposable plastic bottle of water!) Then a presentation on Twine - yet another social network app, but at least with some interesting semantic web aspects. A brief talk on Digital Democracy about the use of the web (and social networks, of course!) in politics.
And the final talk was by Timothy Ferriss of the (4-Hour Workweek). I'd seen Tim around during the conference, attending talks. My first impression was that he was short. A bit like seeing an famous actor and finding out how short they are. He also blended in pretty well carrying his backpack around, although without the ever present laptop of most people. But when he gave his talk he definitely had a bigger presence and charisma. Unlike every other speaker, he did not have any powerpoint slides - he just talked. If you've read his book and blog it was nothing new. Despite being a wing nut in many ways, I still find him "inspiring" if that's the right word. The gist of his message is that you don't have to be powerlessly overwhelmingly "busy". You can re-engineer your life to do what you want to do.
Wednesday, March 05, 2008
Day 3 at ETech 2008
When I saw the first speaker of the day was some old guy instead of the scheduled Kathy Sierra, I was a little disappointed. The "old guy" turned out to be John McCarthy, the inventor of Lisp and "artificial intelligence". To me it was a pleasant surprise to see someone like this along with all the "tech kiddies". Unfortunately, his talk moved really slowly and they had to cut him off. I felt bad for him.
Kathy Sierra did speak later. She talked about how we all want to be "good" at something and what it takes to do that. Surprisingly, natural ability counts less than sheer focus and concentration. Of course, that requires motivation and it's less clear how to get that. Lately I've been thinking about when I developed Suneido and my focus (perhaps fanaticism would be more accurate). I seldom achieve that kind of focus anymore and I miss it. I just haven't quite figured out how to get it back. Then again, do I want to get it back? Spending the majority of your waking moments working on something isn't exactly a balanced life, no matter how challenging/rewarding/addictive it is.
Looking at today's sessions it seemed like there wasn't that much that was attractive to me. But sometimes it's a good thing to be be forced to go to talks that I might otherwise have skipped. After all, the whole point of the conference (for me) is to get exposed to new ideas.
Brain Imaging and I Sing the Body Electric surprisingly turned out to have a lot of common ground. There's a definite feeling around that hacking humans is the next frontier. I've never been keen on the idea of "hacking" myself, maybe because I know how easy it is to screw up complex systems.
Hackers Built My Motorcycle started out by saying his talk had nothing to do with the title, and sure enough he never mentioned motorcycles. Maybe that was an example of "hacking" the conference schedule. Nevertheless, it was a fun talk. The problem with talks about security, like talks about the environment, is that they are somewhat depressing! He proceeded to explain how easy it is to hack into cell phones, web sites, house locks, RFID credit cards ... scary stuff.
The next talk, about technology in Cuba, had the potential to be quite interesting but the two speakers read a pre-written speech and that seldom works well. I've really found that a good presenter is worth going to regardless of the topic, and no matter how interesting the topic, a bad presenter will kill it.
My final session was on OpenCV, an open source computer vision library. I don't know much about this area but it was pretty amazing what's possible these days.
Kathy Sierra did speak later. She talked about how we all want to be "good" at something and what it takes to do that. Surprisingly, natural ability counts less than sheer focus and concentration. Of course, that requires motivation and it's less clear how to get that. Lately I've been thinking about when I developed Suneido and my focus (perhaps fanaticism would be more accurate). I seldom achieve that kind of focus anymore and I miss it. I just haven't quite figured out how to get it back. Then again, do I want to get it back? Spending the majority of your waking moments working on something isn't exactly a balanced life, no matter how challenging/rewarding/addictive it is.
Looking at today's sessions it seemed like there wasn't that much that was attractive to me. But sometimes it's a good thing to be be forced to go to talks that I might otherwise have skipped. After all, the whole point of the conference (for me) is to get exposed to new ideas.
Brain Imaging and I Sing the Body Electric surprisingly turned out to have a lot of common ground. There's a definite feeling around that hacking humans is the next frontier. I've never been keen on the idea of "hacking" myself, maybe because I know how easy it is to screw up complex systems.
Hackers Built My Motorcycle started out by saying his talk had nothing to do with the title, and sure enough he never mentioned motorcycles. Maybe that was an example of "hacking" the conference schedule. Nevertheless, it was a fun talk. The problem with talks about security, like talks about the environment, is that they are somewhat depressing! He proceeded to explain how easy it is to hack into cell phones, web sites, house locks, RFID credit cards ... scary stuff.
The next talk, about technology in Cuba, had the potential to be quite interesting but the two speakers read a pre-written speech and that seldom works well. I've really found that a good presenter is worth going to regardless of the topic, and no matter how interesting the topic, a bad presenter will kill it.
My final session was on OpenCV, an open source computer vision library. I don't know much about this area but it was pretty amazing what's possible these days.
Tuesday, March 04, 2008
Day 2 at ETech 2008
Saul Griffith's talk on Energy Literacy was well done but pretty depressing. I can't dredge up much optimism that a) people will massively reduce their energy use, and b) we'll shut down much of our fossil fuel energy production and replace it via a massive construction of new green energy sources. I can't see it happening. People won't take such drastic action until there's a crisis. So it was further depressing to hear that the time lag on carbon reduction affects can be hundreds of years. i.e. By the time there's a crisis it'll be way too late to do anything. Of course, it's a contentious subject. Someone at my lunch table argued that technology would save us. Hmmm ... maybe it'll save us humans (we're good at that) but what about the rest of our ecosystem? We don't have near as good a track record there.
The presentation by MegaPhone on Collaborative Gaming in Public Spaces was entertaining. The two presenters were the founders and looked to be still in their teens! They develop ways for people to interact with public video displays, primarily by cell phone. They ended up in the debugger on the big screen a few times, but it seemed to work in the end. I'm not sure how they got to be keynote speakers but it was refreshingly "innocent".
The session on the Future of Mind Hacks was pretty interesting. You always wonder if you've picked the right session, but when Timothy Ferriss is a couple of rows ahead and Tim O'Reilly is a couple of rows behind I figure I've picked a good one.
Two Microsoft employees were sitting beside me, one had a MacBook, the other an iPhone (and worked in the mobile division). Hmmm ... is that getting to know the competition?
After the keynotes I went to Tap is the New Click on gestural interfaces (e.g. the iPhone). It was ok, but not great, mostly just a dry overview.
At lunch I counted 10 tables with a single person at them (one of them was mine). Good to know I'm not the only anti-social geek. But most geeks must be more social electronically than me. One of the big topics is social applications. Frankly, I don't have enough friends to need an application to keep track of them. Then again, I don't even have a cell phone so I'm obviously abnormal.
After lunch I went to Green Nano by an HP researcher. I've been excited by nanotech since Eric Drexler's Engines of Creation so it was nice to hear about progress. But the talk seemed a little dry.
For some variety I next went to a talk on Digital Activism by Ethan Zuckerman. It turned out to be pretty interesting. It's good to hear about some "positive" uses of technology.
Then DIY Drones by Chris Anderson (of Long Tail fame) - fun.
And finally, Personal Productivity by Gina Trapani of LifeHacker, another good talk. (Of course, Timothy Ferriss was at this one as well.)
All in all, a pretty good day.
The presentation by MegaPhone on Collaborative Gaming in Public Spaces was entertaining. The two presenters were the founders and looked to be still in their teens! They develop ways for people to interact with public video displays, primarily by cell phone. They ended up in the debugger on the big screen a few times, but it seemed to work in the end. I'm not sure how they got to be keynote speakers but it was refreshingly "innocent".
The session on the Future of Mind Hacks was pretty interesting. You always wonder if you've picked the right session, but when Timothy Ferriss is a couple of rows ahead and Tim O'Reilly is a couple of rows behind I figure I've picked a good one.
Two Microsoft employees were sitting beside me, one had a MacBook, the other an iPhone (and worked in the mobile division). Hmmm ... is that getting to know the competition?
After the keynotes I went to Tap is the New Click on gestural interfaces (e.g. the iPhone). It was ok, but not great, mostly just a dry overview.
At lunch I counted 10 tables with a single person at them (one of them was mine). Good to know I'm not the only anti-social geek. But most geeks must be more social electronically than me. One of the big topics is social applications. Frankly, I don't have enough friends to need an application to keep track of them. Then again, I don't even have a cell phone so I'm obviously abnormal.
After lunch I went to Green Nano by an HP researcher. I've been excited by nanotech since Eric Drexler's Engines of Creation so it was nice to hear about progress. But the talk seemed a little dry.
For some variety I next went to a talk on Digital Activism by Ethan Zuckerman. It turned out to be pretty interesting. It's good to hear about some "positive" uses of technology.
Then DIY Drones by Chris Anderson (of Long Tail fame) - fun.
And finally, Personal Productivity by Gina Trapani of LifeHacker, another good talk. (Of course, Timothy Ferriss was at this one as well.)
All in all, a pretty good day.
Monday, March 03, 2008
Day 1 at ETech 2008
The first day at ETech is tutorials.
First up was Live, Vast and Deep: Web-native Information Visualization by Tom Carden of Stamen Design. Although it was pretty high level and didn't get into too many details, there were lots of thought provoking examples and links to things to investigate - and that's what I'm looking for.
It was a tough choice between this and Storyboarding for Nonfiction by Kathy Sierra (or Creating Passionate Users). I decided that I might get more useful ideas (for me) from visualization. But I bet Kathy's talk was good too.
In the afternoon I went to Debugging Hacks: What They Never Taught You About Solving Hard Bugs by Marc Hedlund of Wesabe. Nothing really new but a good talk on how to solve hard bugs. One point that resonated with me: "the goal is not to suppress the symptoms, it's to understand the problem". I have this discussion with my own programmers on occasion - removing an assert is not "solving" the problem! And conversely, adding the assert did not "cause" the problem.
First up was Live, Vast and Deep: Web-native Information Visualization by Tom Carden of Stamen Design. Although it was pretty high level and didn't get into too many details, there were lots of thought provoking examples and links to things to investigate - and that's what I'm looking for.
It was a tough choice between this and Storyboarding for Nonfiction by Kathy Sierra (or Creating Passionate Users). I decided that I might get more useful ideas (for me) from visualization. But I bet Kathy's talk was good too.
In the afternoon I went to Debugging Hacks: What They Never Taught You About Solving Hard Bugs by Marc Hedlund of Wesabe. Nothing really new but a good talk on how to solve hard bugs. One point that resonated with me: "the goal is not to suppress the symptoms, it's to understand the problem". I have this discussion with my own programmers on occasion - removing an assert is not "solving" the problem! And conversely, adding the assert did not "cause" the problem.
Saturday, March 01, 2008
San Diego ETech
I'll be in San Diego this week for ETech.
Drop me an email if you're in the area and interested in meeting up.
Drop me an email if you're in the area and interested in meeting up.
Wednesday, February 27, 2008
Keep it Simple
37signals response to an article about them in Wired.
I don't necessarily agree with all of 37signal's philosophy but it's nice to see someone fighting against complexity and for simplicity.
I don't necessarily agree with all of 37signal's philosophy but it's nice to see someone fighting against complexity and for simplicity.
Tuesday, February 26, 2008
The Future is Free?
An interesting article by Chris Anderson (also know for the "long tail") in Wired on how and why more and more things are "free".
Sunday, February 24, 2008
Skype, Headsets, and Bluetooth
I'm going to be spending some time away from home without Shelley so I thought I should set up Skype so we could talk. It was no problem to download and install the software on my MacBook and on Shelley's Windows XP PC.
I bought a USB headset for the PC but I wanted something smaller for the MacBook and since it has Bluetooth I figured I could get a Bluetooth headset. I couldn't find a computer specific bluetooth headset, only cell phone ones. But would they work with the Mac? I did some research on the web and read about lots of problems, but mostly old issues that were supposedly fixed with Leopard. I didn't even bother trying to ask a clerk at Staples which bluetooth headsets worked with Skype on OS X on a MacBook. Although I guess if you were lucky you might get some kid who was an expert on the issue.
I chose a Bluetrek Tattoo headset, more or less at random. I charged it up, and managed to pair it with the MacBook, but I couldn't get any sound in or out of it. I couldn't even tell if it was "on" or not. It was supposed to have a green light when it was switched on but I got no lights. I played around for a while but it just seemed to be dead. I took it back and picked a Motorola H800, again for no particular reason. This time it worked fine, no problems. I still don't know if the Tattoo is incompatible, or if I just got a dud.
So now I have Skype working. The sound quality of the "echo" test call wasn't great but it says it's going to the UK so that might be part of the reason.
Of course, what I paid for the two headsets would have paid for more than enough regular long distance phone calls, but what would be the fun in that!
I bought a USB headset for the PC but I wanted something smaller for the MacBook and since it has Bluetooth I figured I could get a Bluetooth headset. I couldn't find a computer specific bluetooth headset, only cell phone ones. But would they work with the Mac? I did some research on the web and read about lots of problems, but mostly old issues that were supposedly fixed with Leopard. I didn't even bother trying to ask a clerk at Staples which bluetooth headsets worked with Skype on OS X on a MacBook. Although I guess if you were lucky you might get some kid who was an expert on the issue.
I chose a Bluetrek Tattoo headset, more or less at random. I charged it up, and managed to pair it with the MacBook, but I couldn't get any sound in or out of it. I couldn't even tell if it was "on" or not. It was supposed to have a green light when it was switched on but I got no lights. I played around for a while but it just seemed to be dead. I took it back and picked a Motorola H800, again for no particular reason. This time it worked fine, no problems. I still don't know if the Tattoo is incompatible, or if I just got a dud.
So now I have Skype working. The sound quality of the "echo" test call wasn't great but it says it's going to the UK so that might be part of the reason.
Of course, what I paid for the two headsets would have paid for more than enough regular long distance phone calls, but what would be the fun in that!
Tuesday, February 19, 2008
Suneido Build Frustrations
I made a minor improvement to the Suneido source code this morning, ran make, no problems, built-in tests ran successfully.
But when I tried to use the new executable I got an obscure database error. What's going on?
I had built with MinGW so I switched to Visual C++. Exact same error!?
This was at home and my last builds had been at work but I can't see why that would matter.
Remove all the object files and build from scratch. No good, same problem.
Check version control to see what I'd changed lately - very little, and nothing that seemed related to the error.
The error is from the database btree code. Maybe the database is corrupted. But all the exe's, old and new, say the database is fine.
Try creating a new database with just the standard library. Now I get a different error related to the Scintilla source code editing component.
Build a MinGW debug version and run it under GDB. That gives me a clue to what query is leading to the error. It had appeared to be outputting to the database, which seemed odd for start up, but it was actually building a temporary index for a query that it was reading. Although that query in the old working executable doesn't require a temporary index.
Turn on the query tracing at the start of the standard library Init to see where the query is coming from. It's loading the plugins.
Aha, that's why the database with only stdlib gets a different error - because it only has to look for plugins in a single library and therefore no temporary index. Yeah, if I disable the plugin loading then I get the other error.
Two unresolved questions
- why the temporary index in the new builds but not the older build?
- why the later UI error?
And how are these two questions related? (assuming a single cause) It seems like it would have to be something low level, like the garbage collector, to affect such unrelated areas.
Of course, it could be something like an uninitialized variable that happens to get a different value on my home system. But it seems too consistent for that. And something like would likely have been encountered before now.
What is different between my work and home machines? I call the office and have them install LogMeIn on my work computer so I can access it. I try building with MinGW and it works fine. The exe is a different size though. Something is different.
Transfer the home exe to work to see if it's the environment. Same error message, so at least it's not because of my Vista on Parallels on Mac setup at home.
md5sum the files at work and at home and compare. The only real difference is the change I made this morning.
But ... that couldn't be the problem could it? Revert that file and build.
Oh no, this is really embarrassing! It works. The problem was the most obvious first place I should have looked - the change I just made.
I'm really tempted not to post this - it just makes me look stupid.
Why did I go off on a wild goose chase? I guess because the error seemed to be so totally unrelated to my change, and I hadn't built for a while so it seemed likely that there could be a problem. And the change I did seemed trivial so I didn't suspect it. And because it seemed trivial I didn't write any tests. (The bug was also obvious, once I looked for it.)
Ouch. There goes a few hours down the drain. Maybe I learned a lesson, but sadly it's one I should have learned a long time ago.
But when I tried to use the new executable I got an obscure database error. What's going on?
I had built with MinGW so I switched to Visual C++. Exact same error!?
This was at home and my last builds had been at work but I can't see why that would matter.
Remove all the object files and build from scratch. No good, same problem.
Check version control to see what I'd changed lately - very little, and nothing that seemed related to the error.
The error is from the database btree code. Maybe the database is corrupted. But all the exe's, old and new, say the database is fine.
Try creating a new database with just the standard library. Now I get a different error related to the Scintilla source code editing component.
Build a MinGW debug version and run it under GDB. That gives me a clue to what query is leading to the error. It had appeared to be outputting to the database, which seemed odd for start up, but it was actually building a temporary index for a query that it was reading. Although that query in the old working executable doesn't require a temporary index.
Turn on the query tracing at the start of the standard library Init to see where the query is coming from. It's loading the plugins.
Aha, that's why the database with only stdlib gets a different error - because it only has to look for plugins in a single library and therefore no temporary index. Yeah, if I disable the plugin loading then I get the other error.
Two unresolved questions
- why the temporary index in the new builds but not the older build?
- why the later UI error?
And how are these two questions related? (assuming a single cause) It seems like it would have to be something low level, like the garbage collector, to affect such unrelated areas.
Of course, it could be something like an uninitialized variable that happens to get a different value on my home system. But it seems too consistent for that. And something like would likely have been encountered before now.
What is different between my work and home machines? I call the office and have them install LogMeIn on my work computer so I can access it. I try building with MinGW and it works fine. The exe is a different size though. Something is different.
Transfer the home exe to work to see if it's the environment. Same error message, so at least it's not because of my Vista on Parallels on Mac setup at home.
md5sum the files at work and at home and compare. The only real difference is the change I made this morning.
But ... that couldn't be the problem could it? Revert that file and build.
Oh no, this is really embarrassing! It works. The problem was the most obvious first place I should have looked - the change I just made.
I'm really tempted not to post this - it just makes me look stupid.
Why did I go off on a wild goose chase? I guess because the error seemed to be so totally unrelated to my change, and I hadn't built for a while so it seemed likely that there could be a problem. And the change I did seemed trivial so I didn't suspect it. And because it seemed trivial I didn't write any tests. (The bug was also obvious, once I looked for it.)
Ouch. There goes a few hours down the drain. Maybe I learned a lesson, but sadly it's one I should have learned a long time ago.
Monday, February 18, 2008
ZENN and the art of slow progress
One of my recurring complaints is gadgets that are only available in the US. But at least it makes a certain amount of sense when they're made in the US. But here's an example where it's made in Canada and it's still only available in the US!
Saturday, February 16, 2008
Creating Passionate Users Video
From the O'Reilly Tools of Change for Publishing conference. I like Kathy Sierra's Head First computer books and was a fan of her blog.
Wednesday, February 13, 2008
VirtualBox
In Tim Bray's ongoing blog he mentions Sun's acquisition of Innotek, the developers of VirtualBox - virtualization software for Windows, Linux, and OS X. I haven't tried it, but it looks interesting - and free for personal use.
Saturday, February 09, 2008
Catching Up
I've been traveling in Ecuador for the last month and although I managed to check my email and update my personal blog periodically I haven't been keeping up on news in the computer world.
One big acquisition was Sun buying MySQL. Sun seems to have done ok with Open Office, hopefully things will work out ok with MySQL.
And Microsoft is trying to buy Yahoo. I find this a little scary. I wonder what would happen to things like Flickr. It seems unlikely that Microsoft could manage to stay "hands off".
I knew Apple would be announcing new stuff while I was gone and I kept meaning to check it out. It didn't turn out to be anything too exciting. A cool new smaller laptop, but since I bought a MacBook not that long ago, that makes me annoyed more than anything! They also announced movie rentals which is interesting. It makes an Apple TV more attractive.
Of course, then I found out movie rentals are US only. Argh! I guess I shouldn't complain. I have no desire to move to the US and Canada has better access to new technology than many/most countries. But this "US only" thing seems to be getting more common. I just heard that Bug Labs new product (that I've been waiting for) is also initially US only.
Sometimes I enjoy the relentless rush of new hi-tech products, other times it's annoying. I buy a MacBook, they announce the thinner, lighter MacBook Air. I buy a Pentax K10, they announce the K20. In most cases I don't need the new product, the old one doesn't suddenly stop working. But it's hard to stop yourself from thinking "if I'd just waited a few months". But you know that doesn't work, the treadmill doesn't stop. Of course, when you're waiting for a new product or feature then it seems to take forever to arrive.
My One Laptop Per Child OLPC XO arrived while I was gone.
It took me a while just to figure out how to open it. I was trying to use the latches on the bottom but they're for the battery. You have to open the antennas to unlatch it. It powered up ok but the user interface was pretty cryptic at first. I kept trying to click by tapping on the touch pad (like I'm used to on other laptops) but that doesn't work. My next problem was that the left "mouse" button is labeled with an "X" which I automatically associate with "close". So I kept trying to use the right button which is labeled with "O". I should have ignored the labels - the left and right buttons work like other systems. Once I got past these roadblocks it got easier.
The keyboard is too small to touch type on with adult hands but I was expecting that. The screen is nice and is quite readable in direct light with the backlight turned off - this saves power and also lets you use it in bright sunlight. I managed to connect to my home wireless network without too much trouble (had to pick hex and shared key) and was able to browse the internet and check my email. The built in camera seems to work fine.
The only real problem I've run into is that I get two CRC errors when I boot up. I'm guessing they're from the hard drive. So far they don't seem to be causing any trouble.
I don't have a particular use in mind for the XO, it's just interesting to see what they've come up with after hearing about the project for so long.
Also waiting for me were the new books by Christopher Alexander (of design patterns fame). Now I just have to find time to read four hefty volumes. Lots of pictures at least :-)
One big acquisition was Sun buying MySQL. Sun seems to have done ok with Open Office, hopefully things will work out ok with MySQL.
And Microsoft is trying to buy Yahoo. I find this a little scary. I wonder what would happen to things like Flickr. It seems unlikely that Microsoft could manage to stay "hands off".
I knew Apple would be announcing new stuff while I was gone and I kept meaning to check it out. It didn't turn out to be anything too exciting. A cool new smaller laptop, but since I bought a MacBook not that long ago, that makes me annoyed more than anything! They also announced movie rentals which is interesting. It makes an Apple TV more attractive.
Of course, then I found out movie rentals are US only. Argh! I guess I shouldn't complain. I have no desire to move to the US and Canada has better access to new technology than many/most countries. But this "US only" thing seems to be getting more common. I just heard that Bug Labs new product (that I've been waiting for) is also initially US only.
Sometimes I enjoy the relentless rush of new hi-tech products, other times it's annoying. I buy a MacBook, they announce the thinner, lighter MacBook Air. I buy a Pentax K10, they announce the K20. In most cases I don't need the new product, the old one doesn't suddenly stop working. But it's hard to stop yourself from thinking "if I'd just waited a few months". But you know that doesn't work, the treadmill doesn't stop. Of course, when you're waiting for a new product or feature then it seems to take forever to arrive.
My One Laptop Per Child OLPC XO arrived while I was gone.
It took me a while just to figure out how to open it. I was trying to use the latches on the bottom but they're for the battery. You have to open the antennas to unlatch it. It powered up ok but the user interface was pretty cryptic at first. I kept trying to click by tapping on the touch pad (like I'm used to on other laptops) but that doesn't work. My next problem was that the left "mouse" button is labeled with an "X" which I automatically associate with "close". So I kept trying to use the right button which is labeled with "O". I should have ignored the labels - the left and right buttons work like other systems. Once I got past these roadblocks it got easier.The keyboard is too small to touch type on with adult hands but I was expecting that. The screen is nice and is quite readable in direct light with the backlight turned off - this saves power and also lets you use it in bright sunlight. I managed to connect to my home wireless network without too much trouble (had to pick hex and shared key) and was able to browse the internet and check my email. The built in camera seems to work fine.
The only real problem I've run into is that I get two CRC errors when I boot up. I'm guessing they're from the hard drive. So far they don't seem to be causing any trouble.
I don't have a particular use in mind for the XO, it's just interesting to see what they've come up with after hearing about the project for so long.
Also waiting for me were the new books by Christopher Alexander (of design patterns fame). Now I just have to find time to read four hefty volumes. Lots of pictures at least :-)
Wednesday, January 02, 2008
A Programming Hierarchy of Needs
After I wrote my last post - There's More to Software Design I was thinking about why so many programmers don't concern themselves with issues like style, grace, and elegance.
It occurred to me that part of the explanation might be something like Maslow's hierarchy of needs. Maslow theorized that people have a hierarchy of needs and that higher levels only come into play after lower levels are satisfied. The lower levels are things like food and shelter; the higher levels are things like morality and creativity. e.g. oversimplifying, starving people don't worry as much about morality
The analogy is probably obvious. The lower level of programming needs would be to get the code to [appear to] work. The inserted "appear to" is important. Without it, "getting the code to work" could be everything up to and including formal verification. Whereas I'm talking about most novice programmer's concept of "getting it working" which is to pass a few random tests.
Things like automated tests, test driven development, and refactoring would be higher up the hierarchy. (Or the equivalent from your methodology of choice.) And things like style, grace, and elegance would be near the top.
And the reason most programmers don't get to the top levels is that they're still struggling with the lower levels. That's not really their fault, either. Programming is unquestionably hard. Just to get something to work in the weak sense can be more than enough challenge.
Note: Just because I think about the higher levels doesn't mean I'm more "advanced". I'm still struggling along at the lower levels with my own code. I just like to dream :-)
It occurred to me that part of the explanation might be something like Maslow's hierarchy of needs. Maslow theorized that people have a hierarchy of needs and that higher levels only come into play after lower levels are satisfied. The lower levels are things like food and shelter; the higher levels are things like morality and creativity. e.g. oversimplifying, starving people don't worry as much about morality
The analogy is probably obvious. The lower level of programming needs would be to get the code to [appear to] work. The inserted "appear to" is important. Without it, "getting the code to work" could be everything up to and including formal verification. Whereas I'm talking about most novice programmer's concept of "getting it working" which is to pass a few random tests.
Things like automated tests, test driven development, and refactoring would be higher up the hierarchy. (Or the equivalent from your methodology of choice.) And things like style, grace, and elegance would be near the top.
And the reason most programmers don't get to the top levels is that they're still struggling with the lower levels. That's not really their fault, either. Programming is unquestionably hard. Just to get something to work in the weak sense can be more than enough challenge.
Note: Just because I think about the higher levels doesn't mean I'm more "advanced". I'm still struggling along at the lower levels with my own code. I just like to dream :-)
Monday, December 31, 2007
There's More to Software Design
There's more to software design than just the "mechanical" aspects.
This article by Mark Hamburg about Lightroom's Goals should give you an idea of what I'm talking about. (And I think they've been fairly successful with this in Lightroom.)
I struggle with this with my company's vertical application, partly because it's hard to get people to see that issues like style, grace, and elegance are relevant to a business application. I think they are. I don't mean it has to be "pretty" or "artsy". But it should look good, flow well, be smooth not awkward. Part of this is definitely the mechanical aspects but part of it is more subtle things. I'm reminded of "quality" in Zen and the Art of Motorcycle Maintenance.
This article by Mark Hamburg about Lightroom's Goals should give you an idea of what I'm talking about. (And I think they've been fairly successful with this in Lightroom.)
I struggle with this with my company's vertical application, partly because it's hard to get people to see that issues like style, grace, and elegance are relevant to a business application. I think they are. I don't mean it has to be "pretty" or "artsy". But it should look good, flow well, be smooth not awkward. Part of this is definitely the mechanical aspects but part of it is more subtle things. I'm reminded of "quality" in Zen and the Art of Motorcycle Maintenance.
Saturday, December 29, 2007
Still Learning
Even though I created Suneido and have used it pretty heavily for years, I still find myself learning better ways to apply it. (I think that's part of the reason I like programming.)
Suneido uses the open source Scintilla editor. ScintillaControl is the "wrapper" that interfaces Scintilla to Suneido's user interface framework. I needed to add a new method to it today:
I noticed I had a lot of these methods and I started wondering whether there wasn't a better alternative to adding so many simple repetitious methods. Ruby, and especially Rails, which I've been involved with on another project, make heavy use of "catching" calls to missing methods and implementing them.
I was able to replace all these simple methods with:
"Default" is Suneido's way to "catch" calls to missing methods.
Then I realized I could generalize it to handle methods with arguments:
"@args" is used to capture all the arguments and then pass them again. args[0] will be the method name.
This allowed me to remove a bunch of methods and I won't have to add any more in the future.
In addition, I noticed I had a lot of calls like .SendMessage(SCI.GETLINECOUNT) within the wrapper code. These could now be simplified to be like: .GetLineCount()
This would all fall into the category of "refactoring" since I'm improving the code without changing its behavior. (Strictly speaking, the behavior has changed slightly, but not in a way that should affect existing code unless someone is doing something unusual.)
I guess you'd call this refactoring something like: "Replace explicit methods with catching missing method calls."
Suneido uses the open source Scintilla editor. ScintillaControl is the "wrapper" that interfaces Scintilla to Suneido's user interface framework. I needed to add a new method to it today:
LineEndExtend()
{ .SendMessage(SCI.LINEENDEXTEND) }
I noticed I had a lot of these methods and I started wondering whether there wasn't a better alternative to adding so many simple repetitious methods. Ruby, and especially Rails, which I've been involved with on another project, make heavy use of "catching" calls to missing methods and implementing them.
I was able to replace all these simple methods with:
Default(method)
{
f = method.Upper()
if not SCI.Member?(f)
throw "method not found: " $ method
return .SendMessage(SCI[f])
}
"Default" is Suneido's way to "catch" calls to missing methods.
Then I realized I could generalize it to handle methods with arguments:
Default(@args)
{
f = args[0].Upper()
if not SCI.Member?(f)
throw "method not found: " $ args[0]
args[0] = SCI[f]
return .SendMessage(@args)
}
"@args" is used to capture all the arguments and then pass them again. args[0] will be the method name.
This allowed me to remove a bunch of methods and I won't have to add any more in the future.
In addition, I noticed I had a lot of calls like .SendMessage(SCI.GETLINECOUNT) within the wrapper code. These could now be simplified to be like: .GetLineCount()
This would all fall into the category of "refactoring" since I'm improving the code without changing its behavior. (Strictly speaking, the behavior has changed slightly, but not in a way that should affect existing code unless someone is doing something unusual.)
I guess you'd call this refactoring something like: "Replace explicit methods with catching missing method calls."
Labels:
refactoring,
suneido
Tuesday, December 25, 2007
More on Scratch
A few comments on Scratch:
I'd really like to be able to browse the code for the projects on the web site. (Unless there's some way I missed.) You can download the projects and presumably see the code that way but that's a bunch more steps and not very good for exploring. Since there isn't much documentation, it would be helpful to quickly look at other people's code. It doesn't seem like this would be hard to add.
Apart from the convenience, I think this is important for deeper reasons. Programming, and thinking "like a programmer" are as much or more about reading code as writing it. Seeing other people's results can give you ideas and inspire, but seeing how they did it is going to be a huge benefit too.
A suggestion for Scratch itself is to get rid of the traditional open/save file management. Alan Cooper in About Face 3 makes a good case for why open/save sucks. I never have to "open" or "save" in Lightroom. Gmail and Blogger save automatically. I don't have to pick/navigate to a directory in Google Docs. In a product for kids especially, you could avoid a bunch of issues by saving automatically to a standard location.
Finally, it's too bad Scratch is so rigid with respect to screen/window sizes. I can understand why they did it that way - it's a lot simpler than trying to use vector or higher resolution images and make things resizable. And for educational purposes maybe it doesn't matter. (Although I notice a number of people wanting to run it 800x600.) Nonetheless, it was a bit disappointing when I ran my program full screen for the first time and got a jagged grainy image (as a result of simple resizing of the low resolution stage). Maybe I'm just spoiled by things like Mac OS X's resizable icons.
I'd really like to be able to browse the code for the projects on the web site. (Unless there's some way I missed.) You can download the projects and presumably see the code that way but that's a bunch more steps and not very good for exploring. Since there isn't much documentation, it would be helpful to quickly look at other people's code. It doesn't seem like this would be hard to add.
Apart from the convenience, I think this is important for deeper reasons. Programming, and thinking "like a programmer" are as much or more about reading code as writing it. Seeing other people's results can give you ideas and inspire, but seeing how they did it is going to be a huge benefit too.
A suggestion for Scratch itself is to get rid of the traditional open/save file management. Alan Cooper in About Face 3 makes a good case for why open/save sucks. I never have to "open" or "save" in Lightroom. Gmail and Blogger save automatically. I don't have to pick/navigate to a directory in Google Docs. In a product for kids especially, you could avoid a bunch of issues by saving automatically to a standard location.
Finally, it's too bad Scratch is so rigid with respect to screen/window sizes. I can understand why they did it that way - it's a lot simpler than trying to use vector or higher resolution images and make things resizable. And for educational purposes maybe it doesn't matter. (Although I notice a number of people wanting to run it 800x600.) Nonetheless, it was a bit disappointing when I ran my program full screen for the first time and got a jagged grainy image (as a result of simple resizing of the low resolution stage). Maybe I'm just spoiled by things like Mac OS X's resizable icons.
Monday, December 24, 2007
Something Fun for Christmas
I recently discovered Scratch a programming system for kids, something like Logo.
I decided since it was Christmas I should do something fun and try it out. Here is my first "program":
I decided since it was Christmas I should do something fun and try it out. Here is my first "program":
Sunday, December 23, 2007
Ubuntu Networking Resolved
This really shouldn't have taken so long. It wasn't even that difficult. But when you only spend a few minutes on something and only every few days or weeks, what do you expect! And the issues with Parallels and Leopard didn't help.
As Larry suggested, the "expert" solution was to edit /etc/network/interfaces and change:
As he also suggested, there is a way to do this from the GUI. When I went to System > Administration > Network I saw this:
[Notice the title bar says "Network Settings" although the menu option was just "Network". I always give my programmers heck for that kind of inconsistency.]
"Roaming mode" ??? I selected Wired Connection, clicked on properties, and changed it to:
[Yet more inconsistencies - I selected "Wired Connection" but I got "eth0".]
i.e. un-checked roaming mode and picked DHCP.
This has a similar effect to the "expert" method, adding a line to /etc/network/interface:
Now when I reboot I still have a network connection. The tooltip on the network icon now says "Manual network configuration" which doesn't seem quite right to me - DHCP is pretty automatic. But I guess it's more "manual" than "roaming mode" (whatever that is).
I feel a little stupid at not having sorted this out myself right from the start but you can't win 'em all, I guess. Thanks Larry!
As Larry suggested, the "expert" solution was to edit /etc/network/interfaces and change:
#iface eth0 inet dhcpto:
iface eth0 inet dhcpi.e. uncomment it.
As he also suggested, there is a way to do this from the GUI. When I went to System > Administration > Network I saw this:
[Notice the title bar says "Network Settings" although the menu option was just "Network". I always give my programmers heck for that kind of inconsistency.]
"Roaming mode" ??? I selected Wired Connection, clicked on properties, and changed it to:
[Yet more inconsistencies - I selected "Wired Connection" but I got "eth0".]
i.e. un-checked roaming mode and picked DHCP.
This has a similar effect to the "expert" method, adding a line to /etc/network/interface:
iface eth0 inet dhcpI can see roaming mode might be a good choice for laptops, but it seems odd that it installed this way. Maybe something in Parallels makes Ubuntu think it doesn't have a regular wired connection. It would be nice if the network icon options at the top of the screen included an option to "save" your choice of wired networking (or just did it automatically).
Now when I reboot I still have a network connection. The tooltip on the network icon now says "Manual network configuration" which doesn't seem quite right to me - DHCP is pretty automatic. But I guess it's more "manual" than "roaming mode" (whatever that is).
I feel a little stupid at not having sorted this out myself right from the start but you can't win 'em all, I guess. Thanks Larry!
Leopard Falters
I spoke too soon about no problems with Leopard. I forgot one major part of my setup - my Epson R1800 wide format photo printer.
I went to print a photo for a Christmas present and found ... no printer. Installing Leopard had silently removed my Epson printer driver. (The CUPS + Gutenprint driver was still there, but I only use it to handle printing from Windows under Parallels.)
I can see a driver not being compatible with a new version of an operating system, but to just silently remove it seems pretty lame. Ideally it would warn you at the start of the install so you had a chance to abort the upgrade if you wanted. At the least it could notify you that it had removed your printer!
Luckily, I had waited long enough to upgrade that Epson had released new drivers. (They were released on Dec. 18 - if I had upgraded a week earlier I'd have been screwed.)
All's well that ends well - my printer is working again and I got my Christmas present done :-)
I went to print a photo for a Christmas present and found ... no printer. Installing Leopard had silently removed my Epson printer driver. (The CUPS + Gutenprint driver was still there, but I only use it to handle printing from Windows under Parallels.)
I can see a driver not being compatible with a new version of an operating system, but to just silently remove it seems pretty lame. Ideally it would warn you at the start of the install so you had a chance to abort the upgrade if you wanted. At the least it could notify you that it had removed your printer!
Luckily, I had waited long enough to upgrade that Epson had released new drivers. (They were released on Dec. 18 - if I had upgraded a week earlier I'd have been screwed.)
All's well that ends well - my printer is working again and I got my Christmas present done :-)
Saturday, December 22, 2007
OpenOffice.org2GoogleDocs
An Open Office extension to transfer documents to and from Google Docs.
I haven't tried it yet, but it sounds useful.
I haven't tried it yet, but it sounds useful.
Thursday, December 20, 2007
A Successful Leap for Leopard
I upgraded my MacBook to Leopard a while ago but I waited to upgrade my main MacMini.
With Leopard updates for my main apps (Parallels and Lightroom) I decided it was time to take the plunge. The upgrade went smoothly, although it seemed pretty slow - several hours. I'm not sure why it takes that long.
As a safety precaution I used SuperDuper to backup each machine before upgrading.
So far I haven't had any major problems. The first time I started Parallels I got the following error:
I found a blog post which said the MacFUSE included in Parallels is old and suggested installing the latest MacFuse. This seemed to do the trick, but:
Now that Spotlight with Leopard lets you run the top application match by hitting enter, I was able to uninstall Google Desktop. (Nothing against Google Desktop, I just prefer to keep things simple if I can) (see my previous post)
Leopard also seems to have solved the issue of automatically mounting network drives. (see my previous post) so I was able to remove the login script I had created to do this, which was nice because it took a long time (why?) and slowed down logging in.
I do have a new complaint about OS X. The Finder doesn't have an option to show hidden files. I can understand hiding them by default, so does Windows. But at least Windows gives you a way to show them. This came up when I went to copy the .svn folder from a backup. It is possible to change Finder via a command line, but on top of not being user friendly, this also requires restarting Finder. This seems like an obvious weak point. Is there someone in Apple who refuses to recognize that you might occasionally want to see these files?
I'm still having problems with accessing my 4gb USB thumb drive from Parallels. At first I blamed this on the U3 software that came installed on it, but I removed this and reformatted and I'm still having problems. It works fine on my Windows machine at work. My current guess is that Parallels doesn't quite handle 4gb USB drives. The strange part is that it works fine, but after a short time it will hang during copying from it, and Windows Explorer can no longer access it. My 1gb USB thumb drive continues to work fine.
Ok, back to Ubuntu on Parallels. I copied the virtual machine that I had created on my MacBook over to my MacMini and started it up. No display problem, but the same problem with the Parallels Tool cd image showing garbled file names. Strangely I can't find anybody else with this particular problem. While flailing a bit more I rebooted the VM and lo and behold the Parallels Tools cd image had the right file names. I installed them and restarted X windows. It appears to work! One of the most noticeable features is being able to move the mouse seamlessly between OS X and the VM. I followed the same process on the MacBook and it also worked (although I could have sworn I tried rebooting before). So I appear to be back in business with Ubuntu (albeit starting from scratch with a new VM).
All in all, a successful day!
With Leopard updates for my main apps (Parallels and Lightroom) I decided it was time to take the plunge. The upgrade went smoothly, although it seemed pretty slow - several hours. I'm not sure why it takes that long.
As a safety precaution I used SuperDuper to backup each machine before upgrading.
So far I haven't had any major problems. The first time I started Parallels I got the following error:
I found a blog post which said the MacFUSE included in Parallels is old and suggested installing the latest MacFuse. This seemed to do the trick, but:
- the error message seems backwards - the operating system was new, MacFUSE was old
- why didn't the Parallels update for Leopard include the required new version of MacFUSE?
- why did I have to get the solution from some user instead of from Parallels? even if the user community discovered the solution, wouldn't it make sense for Parallels to post it? (in fairness, maybe they have, but I didn't find it if they did)
Now that Spotlight with Leopard lets you run the top application match by hitting enter, I was able to uninstall Google Desktop. (Nothing against Google Desktop, I just prefer to keep things simple if I can) (see my previous post)
Leopard also seems to have solved the issue of automatically mounting network drives. (see my previous post) so I was able to remove the login script I had created to do this, which was nice because it took a long time (why?) and slowed down logging in.
I do have a new complaint about OS X. The Finder doesn't have an option to show hidden files. I can understand hiding them by default, so does Windows. But at least Windows gives you a way to show them. This came up when I went to copy the .svn folder from a backup. It is possible to change Finder via a command line, but on top of not being user friendly, this also requires restarting Finder. This seems like an obvious weak point. Is there someone in Apple who refuses to recognize that you might occasionally want to see these files?
I'm still having problems with accessing my 4gb USB thumb drive from Parallels. At first I blamed this on the U3 software that came installed on it, but I removed this and reformatted and I'm still having problems. It works fine on my Windows machine at work. My current guess is that Parallels doesn't quite handle 4gb USB drives. The strange part is that it works fine, but after a short time it will hang during copying from it, and Windows Explorer can no longer access it. My 1gb USB thumb drive continues to work fine.
Ok, back to Ubuntu on Parallels. I copied the virtual machine that I had created on my MacBook over to my MacMini and started it up. No display problem, but the same problem with the Parallels Tool cd image showing garbled file names. Strangely I can't find anybody else with this particular problem. While flailing a bit more I rebooted the VM and lo and behold the Parallels Tools cd image had the right file names. I installed them and restarted X windows. It appears to work! One of the most noticeable features is being able to move the mouse seamlessly between OS X and the VM. I followed the same process on the MacBook and it also worked (although I could have sworn I tried rebooting before). So I appear to be back in business with Ubuntu (albeit starting from scratch with a new VM).
All in all, a successful day!
Saturday, December 15, 2007
CouchDB
I found CouchDB referenced from one of the posts about Amazon's SimpleDB since they are both apparently written in Erlang.
It's interesting that people are exploring some alternatives to relational databases.
It's also interesting that people are implementing "real" products in alternative languages like Erlang.
I recently picked up Programming Erlang but I haven't read it yet.
One thing that caught my eye looking through the CouchDB web site was a brief note that they compact the database while it's running, by copying to a new database. Currently Suneido requires you to occasionally shut down the database in order to compact it. I had always thought about "on line" compaction in terms of doing it "in place", but that gets tricky due to updating indexes to point to new locations. But if you build a new copy you don't have that problem. You could copy the bulk of the database in a single read-only transaction (like the current on-line backup does) and then pause activity briefly to get any updates done during the copy, and then switch over to the new database. Hmmm... actually doesn't sound too bad. (famous last words!)
It's interesting that people are exploring some alternatives to relational databases.
It's also interesting that people are implementing "real" products in alternative languages like Erlang.
I recently picked up Programming Erlang but I haven't read it yet.
One thing that caught my eye looking through the CouchDB web site was a brief note that they compact the database while it's running, by copying to a new database. Currently Suneido requires you to occasionally shut down the database in order to compact it. I had always thought about "on line" compaction in terms of doing it "in place", but that gets tricky due to updating indexes to point to new locations. But if you build a new copy you don't have that problem. You could copy the bulk of the database in a single read-only transaction (like the current on-line backup does) and then pause activity briefly to get any updates done during the copy, and then switch over to the new database. Hmmm... actually doesn't sound too bad. (famous last words!)
Friday, December 14, 2007
Amazon SimpleDB
Amazon has announced a new service - SimpleDB
We are pretty happy with our use of Amazon's S3 (Simple Storage Service)
I've been curious to try Amazon EC2 (Elastic Compute Cloud) but I haven't found a good application yet.
One of the big limitations with EC2 is that it's not well suited to running database servers. I've been waiting for them to improve support for this, but instead (or at least, first) we get SimpleDB.
I wonder if someone will make Rails work with SimpleDB as the database? How would the performance compare?
We are pretty happy with our use of Amazon's S3 (Simple Storage Service)
I've been curious to try Amazon EC2 (Elastic Compute Cloud) but I haven't found a good application yet.
One of the big limitations with EC2 is that it's not well suited to running database servers. I've been waiting for them to improve support for this, but instead (or at least, first) we get SimpleDB.
I wonder if someone will make Rails work with SimpleDB as the database? How would the performance compare?
Thursday, December 13, 2007
Finally!
Finally some good progress on the ACE version of the Suneido server. It's actually working well enough to run a client IDE from it, a major milestone. The last couple of problems were minor mistakes of mine. ACE and the Boehm GC seem to be working together.
Of course, this is just the start, now comes the "fun" part - actually making my code thread safe.
Of course, this is just the start, now comes the "fun" part - actually making my code thread safe.
Saturday, December 01, 2007
ACE + GC Progress
I spent a few more frustrating hours thrashing around trying to build and link with ACE statically.
Finally, I decided to start from scratch. Strangely "make clean" didn't clean up (as I discovered when a make after make clean didn't recompile!). (Note: Don't run make clean from the top level ACE_wrappers directory - it takes forever recursing into all the examples and tests.)
I rebuilt and ... it worked! Somehow I had still been getting left over shared library stuff. Another requirement is to #define ACE_AS_STATIC_LIBS before include the ACE headers.
Boy, that shouldn't have been so hard! But I can't really blame anyone but myself :-)
But ... now Suneido crashes right away on startup, which seemed more like a step backwards not forwards!
I created a small test program that used ACE and GC. It crashed the same way. Eventually, after another few hours of flailing I hit on the right combination. The key seems to be to initialize GC first, then ACE. But to achieve that, you have to prevent ACE from redefining "main" to do their startup. Here's my successful test program:
#define ACE_AS_STATIC_LIBS 1
#include "ace/Thread_Manager.h"
static ACE_THR_FUNC_RETURN thread_func(void* arg)
{
for (int i = 0; i <>
operator new(10000);
return 0;
}
extern "C" { void GC_init(); }
#undef main
int main(int argc, char**argv)
{
GC_init();
ACE::init();
ACE_Thread_Manager::instance()->spawn_n(2, thread_func);
ACE_Thread_Manager::instance()->wait();
}
At this point I'm quitting for the day. It should be easy to incorporate what I've learned into Suneido, but then I'll just run into the next problem. I'd rather end the day on a positive note!
Finally, I decided to start from scratch. Strangely "make clean" didn't clean up (as I discovered when a make after make clean didn't recompile!). (Note: Don't run make clean from the top level ACE_wrappers directory - it takes forever recursing into all the examples and tests.)
I rebuilt and ... it worked! Somehow I had still been getting left over shared library stuff. Another requirement is to #define ACE_AS_STATIC_LIBS before include the ACE headers.
Boy, that shouldn't have been so hard! But I can't really blame anyone but myself :-)
But ... now Suneido crashes right away on startup, which seemed more like a step backwards not forwards!
I created a small test program that used ACE and GC. It crashed the same way. Eventually, after another few hours of flailing I hit on the right combination. The key seems to be to initialize GC first, then ACE. But to achieve that, you have to prevent ACE from redefining "main" to do their startup. Here's my successful test program:
#define ACE_AS_STATIC_LIBS 1
#include "ace/Thread_Manager.h"
static ACE_THR_FUNC_RETURN thread_func(void* arg)
{
for (int i = 0; i <>
operator new(10000);
return 0;
}
extern "C" { void GC_init(); }
#undef main
int main(int argc, char**argv)
{
GC_init();
ACE::init();
ACE_Thread_Manager::instance()->spawn_n(2, thread_func);
ACE_Thread_Manager::instance()->wait();
}
At this point I'm quitting for the day. It should be easy to incorporate what I've learned into Suneido, but then I'll just run into the next problem. I'd rather end the day on a positive note!
Subscribe to:
Posts (Atom)