Git + Windows + Parallels + make + msys sh + Go

Git recently made a security fix. Unfortunately, it broke things for a lot of people, including me. Just in case anyone else has a similar obscure setup (and for my own notes), here's how I solved my issue.

My configuration is a bit off the beaten path.

• I am working on a Mac
• I have my git repos in Dropbox
• I use Parallels to build and test on Windows
• I use make to run my go build
• make is using msys sh as its shell

As of 1.18 Go includes vcs info in builds. So when I ran a Go build on Windows, it would fail with:

# cd x:\gsuneido; git status --porcelain
fatal: unsafe repository ('//Mac/Dropbox/gsuneido' is owned by someone else)
To add an exception for this directory, call:
        git config --global --add safe.directory '%(prefix)///Mac/Dropbox/gsuneido'
error obtaining VCS status: exit status 128
        Use -buildvcs=false to disable VCS stamping.

Adding -buildvcs=false did get the builds working, but it seemed ugly to disable a Go feature to work around a Git issue. It also didn't help if I wanted to do other Git commands.

I struggled with adding the safe.directory. I wasn't sure if %(prefix) was literal or was a variable. I also wasn't sure about whether it should be forward slashes or back slashes and how many (a perpetual issue on Windows). And I wasn't sure about the quoting. Eventually, I just edited my .gitconfig with a text editor.

Here's what worked:

[safe]
	directory = %(prefix)///Mac/Dropbox/gsuneido

Now I could do git commands.

But my build still failed with the same error!?

make is using msys sh as its shell. And sure enough, from within sh, I was back to the same error. git config --global --list didn't show my safe directory. That turned out to be because my home directory from inside sh was different. If I ran git config --global --add safe.directory from within sh, then it created another .gitconfig in /home/andrew. Now I could run git commands from sh, and now my build works.

I'm a little nervous about having multiple .gitconfig files, one on Mac, one for Windows, and one for sh on Windows but I don't have anything critical in there, so hopefully it'll be ok.

I'm all for security fixes, and I try to keep up to date, but it's definitely frustrating when it breaks stuff.

gSuneido Roller Coaster

One of the questions with the Go version of Suneido (gSuneido) was the Windows UI interface. In the long run I'm hoping to switch to a browser based front end, but that's a long term project and I would like to replace cSuneido sooner than that. One option is to implement cSuneido's general purpose DLL interface. But this involved some ugly low level code (even some assembler) that I didn't really want to port. Another option was to use Go's syscall facilities to write the specific Win32 functions that we needed. I did a quick check on how many functions we needed to open the IDE. It was about 100. I knew the final number would be higher, but it seemed reasonable to hand write that many. One advantage of doing this is that it moved the "unsafe" code from Suneido to Go, which seemed safer.

Of course, it took much longer than I expected to write (and debug) the interface functions. There were about 300 in the end. (It's not just the functions, it's converting to and from all the structs involved as well.) I definitely had a few doubts whether this was the right approach. In retrospect, I'm pretty sure I could have ported the general purpose interface in less time. It was also a bit depressing knowing that if/when we got the browser interface finished, then this code would all be thrown out. If I was to do it again, I might write a tool that would take the cSuneido definitions and generate Go language definitions.

But finally I got near to complete and I could actually run the Suneido GUI. That felt pretty good.

Except there were intermittent crashes. I didn't worry about it too much at first. This kind of low level unsafe code can easily crash if you have bugs. But even after I cleaned up the majority of the bugs, it was still crashing. It worked most of the time, so I didn't think it was blatant errors in the code. I spent days trying to track it down. Of course, like many elusive bugs, it almost never crashed in the debugger.

My first thought was something related to garbage collection. But disabling garbage collection didn't help.

Another possibility was stack movement. Because Go supports high numbers of goroutines, they start out with relatively small stacks, which grow as necessary. This is a tricky process because you have local variables on the stack, and pointers between them, all of which have to be adjusted when the stack is moved. I couldn't figure out how to debug this. There was no way to turn it off, and no way to trace when it happened. I tried to detect it by storing the address of a stack value in an integer where it wouldn't get adjusted. But this didn't seem to work and I never did figure out why. (Part of the problem is that unlike C(++), in Go you don't control whether things are on the stack or the heap. That's all handled automatically by the compiler. It's quite possible that my value was ending on the heap and therefore useless for detecting stack movement.)

Needless to say, this was another low point. For starters, all that work for nothing - it was useless if it crashed randomly. And secondly, what now? If I couldn't reliably interface with the Windows GUI from Go, then the only other option I could see was to write the GUI interface in a separate C(++) program and use some kind of interprocess communication. Presumably that would work, but it would be ugly.

It might seem odd that something so objective and abstract would involve so much emotion like elation and depression. But software development (at least so far) is a human activity, and human are nothing if not subjective and emotional.

One thing that puzzled me was that I knew there were several other Go projects that interfaced with the Windows GUI. (Like https://github.com/lxn/win and walk) Why didn't they have this problem? I started searching and soon found this issue:

don't use Go heap for MSG
It's not entirely clear to me what's going on here, and I'm only
half-certain that this fixes the issue, due to it being somewhat hard to
reproduce ... While this might point to deeper Go runtime issues, we
just work around this here by allocating the variable using GlobalAlloc.
At the very least, I haven't been able to reproduce the bug after
applying this patch.

Following the links and looking at the related issues it sounded very much like what I was seeing. I applied their fix/workaround and sure enough the crashes went away, accompanied by feelings of relief and joy.

Part of the problem/complexity is that the way the Windows GUI works, there are nested callbacks and DLL/sycalls. i.e. Windows calls application code which calls Windows code which calls application code, and so on. On top of that, Go syscalls and callbacks do a complicated dance to switch between Go stacks and syscall stacks, and attempt to protect non-Go code from garbage collection and stack movement. It's not at all surprising that there would be ugly corner cases.

Well, my happiness lasted all of a day before I had another crash. Hmmm... was that the old problem still there, or something else? I crossed my fingers that it was something unrelated. There's a fine line between wishful thinking and having a positive attitude. Sadly, the crashes kept happening and it became obvious the problem wasn't solved.

But the lxn/walk issue had seemed so similar. And the fix had certainly reduced the number of crashes. If the issue was passing pointers to stack values, then I was doing this in a lot more places than just the message loop MSG. Whereas the lxn win/walk code always took the pointers as arguments rather than declaring them locally. (Of course, I didn't figure this out quite as step by step as it sounds. There was a lot of frowning and frustration first. )

I tried changing a few of the common crash sites to avoid anything Go might put on the stack. That reduced crashes even further. Of course, once the crashes get rare, it becomes very difficult to know whether you've fixed anything or not. It always seems to work at first. Just when you're ready to declare success, then it crashes again.

It seemed like I was on the right track, so I took the plunge and changed all the functions to allocate temporary values on my own "heap/stack". This was a bit of a gamble since it was a lot of changes (several days work) with no guarantee it would help.

But the gamble seemed to pay off. No more crashes! I was a happy camper.

I soon had more or less the whole IDE working. There were numerous little glitches but they were mostly easy to fix. Until one glitch that was more problematic. I happened to try dragging some text within an editor window, and it didn't work. At first I thought the drop was failing since the text didn't appear at the destination. But I soon realized it was getting inserted at the very beginning or end of the text, instead of where I was trying to drop it. I couldn't remember how drag and drop worked so I had to do a little digging. At this point I was still assuming it was just some minor error in one of the functions. But it wasn't anything obvious. I narrowed it down to a couple of notification callbacks (EN_CHANGE or SCEN_MODIFIED). If those callbacks did anything significant then the drop would go to the wrong place. It didn't seem to matter what they did. I never did narrow it down to whether it was call stack or heap usage.

The problem was, this was a bit of a dead end. Inserting seemingly harmless, irrelevant code in between Windows and Scintilla (the editor control) caused drag and drop to fail. I looked at the Scintilla code but it didn't seem to be doing anything unusual or suspicious.

I spent another couple of days on this issue until I admitted defeat. Again, there seemed to be something going wrong in the depths of Go's complex syscall / concurrent garbage collection / stack movement. In theory I should submit a Go bug report. But, understandably, they'd want a small example that consistently reproduced the problem. And I don't have anything close to that. I have a large system that crashes rarely.

Another low point. Now what? Such a small symptom, but there's no such thing as minor corruption, it's a bit like a little bit pregnant.

Back to my previous idea of a separate GUI process written in C. Probably the fastest interprocess communication would be shared memory. But you'd still need some kind of synchronization. It still seemed ugly.

Did I really need a separate process? I want some isolation, but maybe I can do that within a single process. Go has a facility for incorporating C code in your Go program (Cgo). What if the C GUI code ran in a separate OS thread, but still within the same process? That would separate the Windows GUI code from all the Go issues of concurrent garbage collection and stack movement. But you wouldn't have the overhead and complexity of a separate process and interprocess communications. And you could, of course, share memory. (carefully!)

But you still need synchronization. When the Go side made a call to the C side, it would need to wait for the result. Similarly when the C side made a callback to the Go side, it would need to wait for the result. One option was to use a Windows synchronization facility. That would mean a syscall on the Go side. The other option was to use a Go synchronization facility, which would mean a callback from the C side. I decided it made more sense to use Go synchronization because I suspected that would play better with the Go scheduler and concurrency.

One way to handle the synchronization was to use Go channels. If it was pure Go code I might have gone that way. But with C involved it looked like sync.Cond would be simpler. When I searched for how to use sync.Cond, what I found was an issue where someone suggested the documentation needed to be improved. I wouldn't have thought that would be at all contentious. But  it seems some of the Go team doesn't like sync.Cond and don't think people should use it, and therefore don't want the documentation improved. Personally,  I question the logic of that. It seems like poor documentation just increases the chances of people misusing it. I appreciate the Go team and I'm grateful for all they've done, but sometimes there seems like a bit of a "we know better" attitude. (Of course, most of the time, I have no doubt they do indeed know better.) Another example is the refusal to supply a goroutine ID or any kind of goroutine local storage, again because "it might be abused". And yet internally, they use goroutine ID's. I'm just an ignorant bystander, but it seems like a bit of a double standard to me.

Despite the poor documentation (and with the help of Stack Overflow)  I figured out how to use sync.Cond. Whether I used it appropriately is a question someone else can debate. No doubt some people would say I should be using channels instead.

Using Cgo turned out to be fairly straightforward. It requires GCC so I installed Mingw64. I decided to stick to C rather than use C++. For the small amount of code I had to write, it seemed simpler and therefore safer. It took a few tries to get the synchronization working properly. The code wasn't complex, but as with all concurrency, the devil is in the details.

Thankfully, it didn't require a lot of changes to my existing Go code. I basically just swapped out the syscall and callback facilities.

Of course, it was another gamble to implement a whole approach on the basis of not much more than a gut feeling it should work. And gut feelings are notoriously unreliable in software development. But it only took a day or two to implement, and I was back to running the the Suneido IDE. And the moment of truth - would drag and drop work now? It did! By this point my reaction was more sigh of relief than elation.

Are there other bugs lurking? Almost certainly. But I'm cautiously optimistic that I've solved the crashing and corruption. The only thing talking to Windows is now a single C thread (with a regular dedicated fixed stack and no garbage collection or even heap allocation) which is a very tried and true setup. You can't get much more basic than that. The only interaction is C calling a Go function to signal and wait. No stack or heap pointers are shared between the two sides. Go's concurrent garbage collection and stack movement can work totally independently of the C thread. And as a bonus, it's more efficient to run the message loop in C with no Go syscall overhead.

I was all ready to post this, but I was a little nervous because I really hadn't tested much. I didn't want to declare victory and then find another problem. Of course, there were always going to be "other problems".

So I did some more testing and it seemed a little sluggish. At first I didn't pay much attention - there's so much background stuff going on in our computers these days that can affect speed. But it didn't go away. I did a quick benchmark of a simple Windows call and got 70 us per call. That seemed high but I wasn't sure. I checked cSuneido and it was 2 us. Ouch. Maybe it was just Go syscall overhead? (i.e. can I blame someone else?) No, directly from go it was .3 us.  My guess is that there is some kind of bad interaction with sync.Cond, but I don't really know.

Another downer, just when I thought I'd won. C'est la vie. The obvious alternative (that I had considered earlier) was to use Windows synchronization. Luckily Windows condition variables and critical sections were similar enough to Go sync.Cond that it didn't take long to switch.

I didn't actually hold my breath when I ran the benchmark, but I mentally had my fingers crossed. And it worked - now it was 1 us per call, better than cSuneido. And the IDE no longer felt sluggish.

I've been running this version for a couple of days, and it seems that this chapter of the saga is over, so I will post this and move on to other bugs.

Windows Overlapped Socket IO with Completion Routines

Up till now, cSuneido has used WSAAsyncSelect to do non-blocking socket IO. But WSAAsyncSelect is deprecated and it's not the nicest approach anyway. cSuneido needs non-blocking socket IO for background fibers, the main fiber uses synchronous blocking IO. (Although that means the main fiber will block background fibers.) Note: Windows fibers are single threaded, cooperative multi-tasking, coroutines. The advantage of fibers is that because they are single threaded and you control the context switches, you don't have the concurrency issues you would with "real" preemptive threads.

I thought that the WSAAsyncSelect code was the source of some failures we were seeing so I decided to rewrite it. My first rewrite used a polling approach. I know that's not scalable, but cSuneido doesn't do a lot of background processing so I figured it would be ok. Basically, I put the sockets in non-blocking mode, and whenever an operation returned WSAWOULDBLOCK the fiber would give up the rest of its time slice (e.g. 50ms) This was quite simple to implement and seemed to work fine.

However, I soon found it was too slow for more than a few requests. For example, one background task was doing roughly 400 requests. 400 * 50 ms is 20 seconds - ouch!

Back to the drawing board. One option was to use WSAEventSelect, but it looked complicated and I wasn't quite sure how to fit it in with the GUI event message loop.

Then I saw that WSARecv and WSASend allowed completion routines, a bit like XMLHttpRequest or Node's non-blocking IO. This seemed like a simpler approach. The fiber could block (yielding to other fibers) and the completion routine could unblock it.

At first I thought I had to use WSASocket and specify overlapped, but it turned out that the regular socket function sets overlapped mode by default. That's ok because it has no effect unless you use WSARecv or WSASend in overlapped mode.

Sending was the easy part since there was no need to block the sending fiber. It could just "fire and forget". One question was whether it would always do the full transfer or whether it might just do a partial transfer and require calling WSASend again (from the completion routine) to finish the transfer. I couldn't find a definitive answer for this. I found several people saying that in practice, unless there is a major issue (like running out of memory), it will always do the full transfer. Currently I just have an assert to confirm this.

Receiving is trickier. You may need to block until the data is available. And the completion routine may get called for partial data in which case you need to call WSARecv again for the remainder. (Although this complicates the code, it's actually a good thing since it allows you to request larger amounts of data and receive it as it arrives.)

WSASend and WSARecv can succeed immediately. However, the completion routine will still be called later. And for WSARecv at least, "success" may only be a partial transfer, in which case you still need to block waiting for the rest.

One complication to this style of overlapped IO is that completion routines are only called when you're in an "alertable" state. There are only a handful of functions that are alertable. I used MsgWaitForMultipleObjectsEx in the message loop, and SleepEx with a zero delay in a few other places. Note: although the MSDN documentation is unclear, you must specify MWMO_AWAITABLE for MsgWaitForMultipleObjectsEx to be alertable. (And it has to be the Ex version.)

Each overlapped WSASend or WSARecv is given an WSAOVERLAPPED structure and this structure must stay valid until the completion routine is called. I ran into problems because in some cases the completion routine wasn't getting called until after the socket had been closed, at which point I'd free'd the WSAOVERLAPPED structure. I got around this be calling SleepEx with a zero delay so the completion routines would run.

When I looked at some debugging tracing I noticed that it seldom blocked for very long. So I added a 1ms SleepEx before blocking to see if the data would arrive, in which case it wouldn't need to block and incur a context switch. This eliminated some blocking, but sometimes it didn't seem to work. I realized it was probably because the sleep was getting ended by an unrelated completion routine (e.g. from the preceding write). So I added a loop to ensure it was waiting at least a millisecond and that fixed it. Of course, I'm testing with the client and server on the same machine so the latency is very low. Across the network it will be slower and will still need to block sometimes.

Although the code wasn't that complicated, it took me a while to get it working properly (i.e. fast). As always, the devil is in the details. But the end result looks good. Background socket IO now runs about 30% faster than the old WSAAsyncSelect version, and almost as fast as the foreground synchronous blocking IO.

Mac Remote Desktop Resolution

At home I have a Retina iMac with a 27" 5120 x 2880 display. At work I have a Windows machine with a 27" 2560 x 1440 display set at 125% DPI. I use Microsoft Remote Desktop (available through the Apple app store) to access my work machine from home.

I'm not sure how it was working before, but after I upgraded my work machine to Windows 10, everything got smaller. It comes up looking like 100% (although that's actually 200% on the Mac).

Annoyingly, when you are connected through RDP you aren't allowed to change the DPI.

I looked at the settings on my RDP connection but the highest resolution I could choose (other than "native") was 1920 x 1080 which was close, but a little too big.

Poking around, I found that in the Preferences of Microsoft Remote Desktop you can add your own resolutions (by clicking on the '+' at the bottom left). I added 2048 x 1152 (2560 x 1440 / 1.25)

Then changed the settings on the connection to use that and it's now back to my usual size.

The screen quality with RDP doing the scaling does not seem as good as when Windows is doing the scaling, but at least the size is the same.

I'm guessing from what I saw with searching the web that there might be a way to adjust this on the Terminal Server on my Windows machine, but I didn't find any simple instructions.

If anyone knows a better way to handle this, let me know.

TortoiseSVN + TortoiseHg Problem

I use Subversion (SVN) for cSuneido (for historical reasons) and Mercurial (Hg) for jSuneido, both on SourceForge (again for historical reasons).

On Windows I use TortoiseSVN (1.8.5) and TortoiseHg (2.11.2) with Pageant (part of PuTTY, but supplied with TortoiseHg) so I don't have to type a password all the time. This combination has worked well for a long time.

I came into work this morning and TortoiseSVN kept popping up a Plink dialog asking for my password. That's what Pageant is supposed to avoid, especially since SourceForge needs an SSH key, not a password.

TortoiseHg was working fine, which meant Pageant was ok.

I used TortoiseSVN a few days ago. As far as I can recall I didn't change anything since then. But possibly I updated it. There are so many updates going by these days that it's hard to remember.

I searched the web but didn't find anything that seemed to be related.

I tried rebooting. I tried changing my path to put TortoiseHg and TortoiseSVN in the opposite order. Didn't help.

After some digging I found TortoiseHg was using older versions of TortoisePlink and pageant (both from 2012) whereas TortoiseSVN had a new TortoisePlink (from 2014). I wasn't sure it was a good idea, but I tried replacing the new TortoisePlink with the old one, thinking that maybe it needed to match the version of pageant.

That worked! Or at least appears to work. (I even rebooted to make sure the problem wouldn't come back.) It's probably going to break next time I update TortoiseSVN, and I'll probably forget the fix, but at least I'll have this blog post to jog my memory :-) And hopefully in the long run this will get sorted out. I can't be the only person running both. I'm not sure why TortoiseHg has such old versions. There seem to have been similar version issues a few years ago.

No auto-update for 64 bit Java on Windows

It's hard to believe after all these years, and all the security issues, that there's still no auto-update for 64 bit Java on Windows.

I have known that Java on my Windows machine wasn't updating properly, but I just assumed it was because I had multiple copies and versions etc. But it's a hassle having to remember to manually download and install updates so finally I decided to try to fix it, only to discover there is no fix.

This was entered as a bug in 2006, and it's currently scheduled to be fixed in Java 8 (!)

Sometimes you really have to wonder about how these things get prioritized. Granted, our customers say the same thing about us, but considering what a security issue this is, I would have thought it would get addressed. Back in 2006 I can see thinking "no one" was running 64 bit, but nowadays that's not a good assumption.

The Joys of High DPI

I recently got a new monitor at work (a Samsung SA850). It's a nice monitor and it matches the resolution of my home iMac (2560 x 1440).

The first hurdle was that I was using the on-board video and it didn't handle that high resolution. I needed a new video card. It still didn't work. Finally figured out it needed a dual-link DVI cable. Thankfully our company hardware guy dealt with all this. Just hearing about it reminded me of how much easier it is to just go buy an iMac. (And this is before all the software fun described below!)

I was hoping the new video card would raise my Windows Experience Index, but strangely, it made it worse! Before my desktop graphics was at 5.5, now it's 4.6. The gaming video index increased, but I don't do any gaming! I guess these cards are optimized for gamers. Anyone have a recommendation for a good video card for programming? (preferably low power)

Once it was working, I found the everything a little small, especially the fonts. (My eyes aren't getting any younger unfortunately.) So I played with the Windows 7 display settings. 125% was too big, so I tried a couple of custom sizes and settled on 115%. Afterwords I thought to measure the actual DPI - it was roughly 109 dpi (not exactly a "retina" display - my MacBook Pro is 220 dpi) Windows standard is 96 dpi. 109 / 96 is about 114% so my 115% setting was very close.

Note: You can adjust the size of Windows icons and their labels separately so don't go by that when choosing the scaling. (Select one, hold down the CTRL key, and then use the mouse scroll wheel)

So far so good, except now Chrome (and other programs) look "fuzzy"! It turns out, this is due to Windows DPI scaling. Taking the bitmap for a small font and just enlarging it by 115% gives crappy results. You can turn off the scaling by setting "Disable display scaling on high DPI settings" in the Compatibility section of a program's Properties. (for 32 bit but not 64 bit programs, and for local but not network drives, sigh) Of course, this may mean it goes back to smaller fonts, but you can choose font sizes within Chrome. To their credit, Firefox and Thunderbird were not fuzzy. It's hard to tell if they are actually scaling since 15% is a fairly subtle difference.

To avoid this issue, programs need to declare themselves DPI aware, either in their manifest or by calling SetProcessDPIAware. (See the MSDN article Writing High-DPI Win32 Applications)

I'm amazed that so many programs don't handle display scaling! It's not like this is something new. Sure, most people don't have high DPI monitors, but of anyone, I would expect programmers to have them, and therefore care about such things. I guess not. Even some Windows dialogs come up fuzzy!

I feel a little guilty about this. For years I've been telling people NOT to set their LCD monitor resolution lower just to get larger fonts. Leave it at its native resolution and use the Windows scaling, I tell them. But now I can see why they do it. Just using the Windows scaling gives even worse results than non-native resolutions!

Even setting scaling is an ugly user experience. (This is Windows 7) You start with Control Panel > Appearance and Personalization > Display. (searching for DPI will get you there). Notice that the title "Display" doesn't even mention anything to do with scaling, even though that's all that's on this screen. To get a custom size, you have to pick "Set custom text size (DPI)". This is the first mention of DPI. Notice that it says "text size" even though the main screen says "size of text and other items". This gives you an image of a ruler. And if you figure it out (I didn't at first) you can hold a physical ruler up to the screen and drag until they match. Unfortunately, the only way to really see what it will look like is to click OK, at which point it makes you log out (shutting down all your programs) and log back in again.

It gets worse. At the bottom of the Custom DPI Setting dialog is a check box labeled "Use Windows XP style DPI scaling". In the words of Princess Bride, "I do not think it means what you think it means". From what I could find out, this actually means something more like "Use ONLY Windows XP style DPI scaling". Or in other words, disable Windows Vista (and later) DPI virtualization. (more info in High DPI Settings in Windows) If this box is unchecked, then virtualization is actually enabled.

Then I discover that Suneido is fuzzy. Argh! I confirmed that calling SetProcessDPIAware fixed the fuzziness. But I ended up updating the manifest since that seemed like a safer approach. Mostly, Suneido was actually already "DPI aware" since it used GetDeviceCaps(hdc, GDC.LOGPIXELSY) to scale fonts. But there were a few spots that weren't scaling (e.g. tab controls). I also realized that Suneido wasn't using the best fonts. We were using "MS Sans Serif" for the default font and this is a bitmap font which scales poorly. Which is why we overrode it to use Arial for larger headings. The current standard for Windows is Segoe UI, and prior to that, Tahoma. We were using Courier New for mono-spaced code, whereas Consolas is much nicer (IMO).

So I spent a bunch of time cleaning up and modernizing the fonts in Suneido. I think the end result is a lot nicer. However, things have changed, which will mean complaints. (Yes, some of our customers complain about any kind of change, no matter how small, regardless of whether it's an improvement or not. You can't win.)

I'll end with a wish that someday we might have GUI's that are truly scalable. I realize one of the problems is bitmaps, but we could use vector graphics for icons (which is often how they're designed in the first place). Photographs and video aren't the problem - we scale them all the time. Apple is no better in this respect. They might hide the issues better than Windows, but they are still tied to certain DPI and on their mobile devices, specific screen sizes.

Java + Guava + Windows = Glitches

Some of my jSuneido tests started failing, some of them intermittently, but only on Windows. There were two problems, both related to deleting files.

The first was that deleting a directory in the tear down was failing every time. The test created the directory so I figured it probably wasn't permissions. I could delete the directory from Windows without any problems. The test ran fine in cSuneido.

I copied the Guava methods I was calling into my own class and added debugging. I tracked the problem down to Guava's Files.deleteDirectoryContents which is called by Files.deleteRecursively. It has the following:

// Symbolic links will have different canonical and absolute paths
if (!directory.getCanonicalPath().equals(directory.getAbsolutePath())) {
    return;
}

The problem was that getCanonicalPath and getAbsolutePath were returning slightly different values, even though there was no symbolic link involved - one had "jsuneido" and the other had "jSuneido". So the directory contents wasn't deleted so the directory delete failed. From the Windows Explorer and from the command line it was only "jsuneido". I even renamed the directed and renamed it back. I don't know where the upper case version was coming from. It could have been named that way sometime in the past. I suspect the glitch may come from remnants of the old short and long filename handling in Windows, perhaps in combination with the way Java implements these methods on Windows.

I ended up leaving the code copied into my own class with the problem lines removed. Not an ideal solution but I'm not sure what else to do.

My other thought at looking at this Guava code was that if that test was extracted into a separate method called something like isSymbolicLink, then the code would be clearer and they wouldn't need the comment. And that might make it slightly more likely that someone would try to come up with a better implementation.

The other problem was that new RandomAccessFile was failing intermittently when it followed file.delete. My guess is that Windows does some of the file deletion asynchronously and it doesn't always finish in time so the file creation fails because the file exists. The workaround was to do file.createNewFile before new RandomAccessFile. I'm not sure why this solves the problem, you'd think file.createNewFile would have the same problem. Maybe it calls some Windows API function that waits for pending deletes to finish. Again, not an ideal fix, but the best I could come up with.

Neither of these problems showed up on OS X. For the most part Java's write-once-run-anywhere has held true but there are always leaky abstractions.

Chromium Embedded continued

Previously I got the Chromium Embedded (CEF) sample application to compile (and run). That was only a tiny step towards what I needed to do.

Suneido has a DLL interface, but it can only handle straightforward interfaces. Although CEF has a DLL with a C interface, it's fairly complicated. So I decided I needed to write a small DLL that Suneido could call that would talk to CEF.

But I wasn't sure what the minimum functionality I'd need. The sample application is actually quite large and complex because it is demonstrating a bunch of features like extensions and plugins. So I started by whittling down the sample to the bare minimum. That process helped me identify the core functions I'd need.

It took me a certain amount of struggling to recall / figure out how to build a Windows DLL. (It's not something I do very often.) Eventually I got to the point where Suneido could see the DLL functions. But calling them didn't work (it gave some weird stack errors). Now what?

Then I remembered the issues with building single-threaded versus multi-threaded. The main version of Suneido we use is built single-threaded (/ML), but CEF is built multi-threaded (/MT). I tried it with a multi-threaded build of Suneido and it worked! (Suneido is still single-threaded, it's just linked with the multi-thread runtime libraries.) Here is Chromium Embedded running in a Suneido window:

Coincidentally, the single/multi-thread version issue has also come up in another area. I'm looking at upgrading my work computer to Windows 7. But before I take the plunge I want to make sure the software I need is going to run under Windows 7. One of the "oldest" packages I use is Visual C++ 2003 (VC7). I would have moved to a more recent version long ago, but VC7 still generates the smallest, fastest executable. I think the main reason for this, is that VC7 was the last version that supported building with the single-threaded libraries. (/ML) I'd prefer to use GCC but it doesn't handle the interface with the Internet Explorer browser component. (It's also a bit slower, but not too bad.) So using CEF instead of IE might also allow switching to GCC.

Chromium Embedded

Our Suneido applications use the Internet Explorer browser component included in Windows.

This works pretty well, but we are running into a few problems:
- people have a wide variety of IE versions
- some people have Javascript turned off
- security settings are getting more and more restrictive

And in the long run I'd like to run on Mac and Linux which don't have IE built in for some reason :-)

So I've been thinking about using another browser component that we can have better control over. Initially I was looking at Gecko (the Mozilla Firefox engine). There's even a Mozilla ActiveX control with the same interface as IE, but it requires registry changes which isn't always feasible for our clients. I tried to find a workaround for the registry issue but couldn't get it working (and no one on Stack Overflow had an answer).

The recent trend seems to be to use WebKit - like in Safari, Chrome, Google Earth, etc. I found the Chromium Embedded project that seems like it would do the trick.

As a first step, I thought I should try to build the example application.Four hours later I think I have achieved this. Since it wasn't easy, I thought I'd recount the saga in case it might help someone else.

First, the standard build environment seems to be Visual Studio C++ 2005. I had various other version installed but not that one. Unfortunately, it's been replaced by VS 2008 which I tried building with, but it gave a lot of errors. (Although maybe the fixes below would have worked?)

A Stack Overflow post led me to someone's page with links to old versions. When I installed, I got compatibility warnings which suggested some updates. I downloaded the updates but when I tried to install it failed, saying the software wasn't installed. (Even though I could run it.) Instead, I was able to get the updates by running Windows Update.

When I tried to build it couldn't find atlbase.h  That's because Visual Studio Express (the free version) doesn't come with the Windows SDK. I already had a Windows Server 2003 SDK installed so I updated Tools - Options to point to it.

But I still got errors about shlwapi.lib missing. It turns out this is part of "Microsoft Web Workshop (IE) SDK", an optional part of the SDK. So I tracked down the right SDK installer and installed it.

Next, I had to patch atlbase.h and atlwin.h as explained in a Code Project post. This cleaned up a bunch more errors. But I was still getting errors about _Module which I fixed based on yet another post.

Then I got errors that I was missing opengl32.lib which stumped me for a while till I found someone else with a similar problem, and ended up at instructions for adding the SDK libraries to Visual Studio. Since you don't actually need opengl32.lib, I'm not sure why the project specifies it.

Initially the Debug version built. It ran, but it had a bad tendency to crash. The Release version seems to be fine.

Another issue is that it's big, about 20mb for the required dll's. I guess that's not that big these days, but it's still a fair size to deploy over the internet.

This all seems way harder than it should be. I realize it's Windows and Visual Studio and C++ and open source, but still! Maybe I've been spoiled by working with Java lately. Java does have benefits.

How Can This Be Acceptable?

I recently downloaded the latest version of the Scite programming editor. And subsequently, every time I ran it I got Windows security warnings. There's a check box that implies it will let you stop these warnings, but as far as I can tell it has no effect. I have no idea why the previous version ran without any warnings.

I eventually got these instructions to work:

1.. Right-click the file and select Properties.
2.. Click on the Security tab.
3.. Click Advanced in the lower right.
4.. In the Advanced Security Settings window that pops up, click on the Owner tab.
5.. Click Edit.
6.. Click Other users or groups.
7.. Click Advanced in the lower left corner.
8.. Click Find Now.
9.. Scroll through the results and double-click on your current user account.
10.. Click OK to all of the remaining windows except the first Properties window.
11.. Select your user account from the list up top and click Edit.
12.. Select your user account from the list up top again and then in the pane below, check Full control under Allow, or as much control as you need.
13.. You'll get a security warning, click Yes.
14.. On some files that are essential to Windows, you'll get a "Unable to save permission changes. access is denied" warning and there's nothing that you can do about it to the best of my knowledge.
15.. Reconsider why you're using Windows.

At my count, that's 7 levels of nested dialogs. And my name didn't show up in the list for step 12 so I had to Add "APM\andrew" (obviously, users would know to type that). Who designs this stuff? Who reviews it? Microsoft is supposed to hire all these really smart people, but they still seem to produce a lot of stupid stuff.

Windows 7 on Parallels on Mac

I just installed Windows 7 (Professional 32 bit) on Parallels (4.0.3846) on my iMac (OS X 10.6.1)

The install went smoothly and relatively quickly.

You still can't use Aero (fancy effects like transparency) under Parallels but that doesn't bother me too much.

My first challenge was that I couldn't see the task bar in Coherence mode (mixed Mac and Windows). A quick Google search found other people with the same problem. Eventually I discovered that there is a menu option to show it (Applications > Show Windows Task Bar). I wonder if it wouldn't be better to show it by default and let the people who want to hide it go hunting for the menu option.

My network icon in the taskbar shows a yellow exclamation mark warning, with a tooltip of "No internet access", but I seem to be able to access the internet fine through Internet Explorer and Windows Update worked fine. When I run the troubleshooter it tells me everything is fine and asks what my problem is. I guess I just ignore the warning. Hopefully Parallels will fix this at some point.

Windows 7 has dropped the Quick Launch bar. Instead you can "pin" things to the task bar. Except that I couldn't. I tried dragging and using the right click context menu. No errors or anything, it just didn't work. More Google searching showed other people with the same problem but no clear solution. Someone suggested setting up a new user account. This didn't really make sense, but I tried it and it worked.

I guess the default initial Administrator account doesn't let you pin anything to the task bar. It would be nice if it gave you some kind of message.

If the default initial Administrator account is not a regular account, why doesn't the Windows install process create a regular account for you? Maybe because of installing via Parallels "unattended" method? I'll have to ask someone who has installed Windows 7 on a PC.

One of the few features that the Windows task bar had that the Mac OS X dock didn't was the ability to toggle between showing and hiding windows by clicking on the task bar icon. (A Windows Feature I'd Like on the Mac) Sadly, this feature seems to be gone in Windows 7. (Unless there is a way to enable it somewhere.)

I've been using Vista on Parallels, but I probably would have been better off with XP because it needs less resources. (That's why Netbooks come with XP.) I'm hoping Windows 7 will be less demanding than Vista. (It's supposed to be.)

A Windows Feature I'd Like on the Mac

Both Windows and Mac OS X let you "minimize" windows to the task bar / dock.

Both let you bring a window back by clicking on the task bar / dock.

But on Windows you can click on the task bar icon a second time to minimize the window again. I've got in the habit of using this to take a quick look at a window and then hide it again. I keep trying to do that on the Mac but it doesn't work.

I can see one argument against this feature would be that people often get confused and double-click instead of single-clicking. If implemented naively, a double-click would show and then hide the window immediately, frustrating the user. But Windows solves this problem by treated a double-click the same as a single click.

If anyone knows a way to make this work on the Mac, leave me a comment and I'll owe you one.

One part of this that is nicer on the Mac is that "Hide" minimizes all of an application's windows, and clicking on the dock brings them all back, whereas on Windows it's one window at a time. I have a vague memory that Windows 7 might improve this.

Eclipse Tip for Windows and Mac Users

Switching back and forth between Windows and Mac, my fingers find it hard to remember whether copy/cut/paste/undo/redo are with the control key or the command key.

In Eclipse you can configure the keyboard so you can add the control versions of these (as well as the command versions).

I also added home and end (Line Start/End).

One nice feature of Windows under Parallels on the Mac is that it lets you use either command or control.

New Computer

As I mentioned in Double Speed, I ordered a new Acer Veriton L460-ED8400 for at work. As you can see, it's certainly tiny. (about the same size as the book included for comparison) I ordered from Frontier PC who were very helpful figuring out what memory and hard drive I needed to go with it.

The only minor complaint about the Acer is that the connector between the optical drive and the motherboard is very flimsy. When we opened it up to put in the extra memory and the bigger hard drive a tiny fragment of plastic broke off the clip that holds the cable. We ended up having to replace it with a binder clip!

Otherwise, the Acer works great - it's small, fast, quiet. Vista seems to work well, including all the Aero effects.

I've gotten quite fond of the current Mac keyboard that I use at home. Compared to it, PC keyboards are incredibly clunky. They all seem to think the more bells and whistles the better. Personally, I want a minimum of bells and whistles. I wondered if a Mac keyboard would work on a Windows PC. I took my Mac keyboard to work and it functioned perfectly. So I went and bought another Mac keyboard. I use the wired version since I find the wireless one too small.


On the other hand, I'm not really fond of the Mac mouse. I'm not sure if it's the hardware or the driver, but right clicking is painfully unreliable. It's almost as if they finally gave in to having a right click, but they made it really crappy because they still didn't think you should be using it. It's especially painful running Parallels because Windows relies on right-clicking so much. So I bought a Logitech LX8 mouse. I liked the cordless receiver, especially since I could plug it into the front of the Acer. (It also would have been harder to use a Mac wireless mouse since they are Bluetooth and most PC's, including the Acer, don't have Bluetooth built in. (Macs do)

I kept the same monitor I've had for a while - a 24" Samsung SyncMaster 245BW that I'm quite happy with. I haven't gone to dual monitors, prefering a single large monitor. Partly maybe because I prefer to focus on one thing at a time. The same reason I don't keep my email running all the time. If I was buying a new monitor I think I'd be tempted by the new Apple 24-inch LED Cinema Display.

I'd say it was the easiest switch-over I've had. Despite moving more and more to on-line apps, I still install and use quite a lot of software. I wish there was a way to migrate Windows apps to a new computer. On the other hand, it's also nice to start fresh with clean installs.

Here's the main stuff:

- Kaspersky (we have been using AVG but we're thinking of switching)
- Firefox
- Thunderbird (for our separate internal email system)
- Open Office
- Scite
- Winmerge
- Gmail Notifier (so email links go to gmail, but not running all the time)
- Snagit
- PDFCreator
- KeePass
- LogMeIn
- TortoiseSVN
- MinGW G++
- Visual C++ 2003 (vc7)
- Visual C++ 2008 express (vc9)
- 7Zip
- GnuWin32 - grep, findutils, diffutils, fileutils/coreutils
- Google Earth
- Google Picasa
- Canvas

Plus all my Firefox add-ons:

- Adblock Plus
- Delicious Bookmarks (classic mode)
- Distrust
- dragdropupload
- Firebug
- FireFTP
- Foxmarks
- Google Gears
- Google Notebook
- IE View
- S3 Firefox Organizer
- Web Developer
- Yslow

I used Foxmarks to sync my bookmarks and passwords (more for the passwords since most of my bookmarks are in Delicious). Too bad you can't sync the add-ons themselves.

Wireless Security

Sometimes it seems like I go out of my way to make things hard for myself!

I got home from work to find a note from Shelley that her laptop wireless wasn't working. Oh yeah, I hadn't got it working with the new Time Capsule.

I run the Airport Utility, change the Wireless Security to WEP, and try to set the password to what I was using previously (so I didn't have to mess with the laptops or other devices).

Nope, it will only take an exactly 13 character password. Okay, I make up a new password and go to connect Shelley's Windows laptop. But it won't work. I mess around with the settings, stupidly failing to record the original settings. No luck.

I search on the internet. Some people say that on Windows you have to use the hex version of the password (you get that from the AirPort Utility from Equivalent Network Password on the Base Station menu). I type in a 26 character hex password multiple times. No luck.

More digging reveals that this is 128 bit WEP. (More or less. It's also Transitional Network Security that is also WPA compatible. Whatever that means.) The older 40 bit WEP that I was using previously is actually available - you just have to turn off the new 802.11n :-( and then hold down the option key while you pull down the Radio Mode list. (how could I have missed that!)

Now I can enter the old 10 digit hex password (with a dollar sign). And now the Windows laptop connects.

I pull out the Chumby to make sure it can still connect. It can, but in the process I find that it does handle WPA! So I don't need to use WEP after all. Argh! (It must have been my old Squeezebox audio player that was WEP only, but I'm not using it any more - too much hassle to run its proprietary server.)

Back to the Airport Utility, re-enable 802.11n, set it back to the default WPA. Back to the Windows laptop, enter the new password, it connects fine. Back to the Chumby, enter the new password, it works fine. Same with my MacBook.

So I think I'm all good. I'm glad my Mac mini is hard wired to the Time Capsule. Trying to configure wireless through a wireless connection would have added even more fun to the process!

Google Desktop versus Mac Spotlight versus Vista

One thing that frustrated me when I started using Google Desktop Search was that I would type my search, it would correctly identify and highlight the top result, but when I hit Enter it would open my browser and show me the search results there (instead of just running/opening the top result). Eventually I discovered you could change this in the Preferences "Launch Programs by Default" (instead of "Search by Default").

I guess I should mention that I use Desktop Search as much or more to launch programs rather than find files. My Windows Start menu has so many programs on it that it is a hassle to use.

Now I have the same hassle on my Mac. Spotlight finds the right result, but when I hit Enter it brings up a search window instead of running the top result. Unfortunately, so far I have not found a setting to change this. It hasn't been too annoying yet because I don't have as much software installed on my Mac so I don't need to use it as much.

As much as I like to dislike Microsoft, they appear to have got this right in Vista. The new search box defaults to running the top result when you hit Enter.

I wonder how Beagle on Linux works in this respect?

PS. I normally have my Google Desktop set to show on my Windows task bar, but today it was missing. I thought it must have crashed or something so I rebooted. But still no search box. I checked whether it was still set to run on startup but that looked ok. When I tried running it manually from the Start menu it displayed in "pop up" mode, but came up so fast it must have already been running. When I checked my preferences I found it was set to not show up. I'm pretty sure I never changed this, so I'm not sure what happened. Maybe some automatic update turned this off?