Building Suneido with Visual Studio 2012

Each time Microsoft releases a new version of Visual Studio I try building Suneido with it. (Using the free Express version.) Usually it's easy. Occasionally I have to tweak some code because they've tightened up the compiler.

Unfortunately, up till now the results have been slower than the old version (version 7) I've been using for release builds. I assume that's because version 7 was the last version to support single threaded libraries. But it makes me a little nervous to still be using such an old version. So I was happy to find that Suneido compiled with the latest version 11 seems as fast as version 7. (it's confusing that the version number is different from, but so close to the year e.g. VS 2012 = version 11) If my assumption about the speed being a result of the multi-threaded libraries, then maybe this means they have reduced the overhead. Or it could  be that they've just improved the compiler enough to balance out the overhead from the multi-threaded libraries. (The C++ version of Suneido is not multi-threaded so it's annoying to pay a penalty for something you don't need. It goes against the C++ philosophy of only paying for the features you use.)

Normally I have just been building using a makefile from the command line. This time I decided to try building with the Visual Studio IDE. Having got accustomed to using an IDE with Java, I find I miss it when I work on the Suneido C++ code. I've been using Eclipse CDT a bit, but I haven't been able to get it to build and it shows all kinds of spurious errors.

You'd think that once your code builds with a certain compiler, that it would be easy to build with the same compiler from an IDE. But that hasn't been my experience. I've tried to build with Eclipse CDT and with Netbeans but I've always given up in frustration. (Even though it's using the same MinGW compiler that I build with from a makefile.)

I did manage to succeed in building Suneido from the VS 2012 IDE but once again it was an extremely frustrating process. A large part of the problem is the huge number of compiler options. I know exactly what I'm specifying for the command line builds, but I don't know what the defaults are for what I'm not specifying. You can see what all the settings are in the IDE, but the defaults are different from the command line compiler. It's helpful that the IDE settings will show you the command line equivalent to your current settings, but I'm not convinced that this is 100% correct. I'm pretty sure that at times I was getting different results in the IDE from what it claimed was the equivalent command line. I could be wrong - there are so many settings that it's easy to get mixed up unless you're incredibly disciplined in your testing.

Even though I had the same warning level and the same settings (AFAIK) I was getting compiler errors in the IDE for code that compiled perfectly from the command line. Eventually I gave up and changed the code so it would compile in the IDE. That feels a bit like giving up, but I'm only willing to spend so long thrashing on this kind of accidental complexity.

One big mistake I made was that I inadvertently did all my tweaking of the settings under the Debug configuration. So when I switched to the Release configuration, I lost all my settings. Argh! I had to redo the settings under All Configurations. The problem is that when you go into the settings it goes to the current configuration. So if you forget to switch to All Configurations you end up changing only one of the configurations. Leading to frustration when you change to the other configuration and things are broken.

Once I got it to compile, it passed my C++ test suite perfectly. But when I ran the Suneido IDE I was missing most of the toolbar buttons. Oops, didn't include the resources. I added the resource file to the project but then I got errors about a duplicate manifest?! After some digging on the web I found this was because Visual Studio automatically creates its own manifest but my resource file added my own manifest. One solution was to change my resource file to not include my manifest. But the automatic manifest didn't include one of the things I needed (version 6 of the common controls). There might be a way to do this through the IDE but I didn't find it. Eventually I figured out how to turn off the automatic manifest and just use my own. In my digging on the web I'd come across people saying this was a "bad" idea, but it seems to work fine for me. I looked at the automatic manifest and it didn't seem to have anything special in it. (The free Express version of Visual Studio doesn't include the resource editor, but that's not a problem because the main suneido.rc resource file is just a text file.)

Another issue I discovered is that programs compiled with VS 2012 won't run on Windows XP (they require Vista or newer). That's not a problem for me, but I'm pretty sure we have a substantial number of customers who still have XP. It sounds like maybe there was some backlash on this issue and Update 1 for VS 2012 now includes support for building programs that will run on XP (or newer). This was fairly easy to turn on in the IDE settings, but I haven't got around to figuring out the equivalent makefile changes.

But even this setting wasn't without its frustrations. After I turned on XP support, I started getting weird errors about multiple definitions of preprocessor symbols in the standard headers. Once more I thrashed around on this for a long time. I found a couple of places on the web talking about the same (or at least similar) problem but they didn't help. Eventually, after a lot of semi-random poking at the settings, the problem went away. I think I had messed something up in my earlier tweaking of the settings, possibly related to turning on or off the "inherit from parent or project defaults" on certain settings.

I know it's fashionable to dislike Java, but I have to say it's a heck of a lot easier compiling a Java program than it is a large C++ program! Although I'm sure Visual Studio experts will shake their heads at my ignorance. It took me a while to get familiar with Eclipse for Java as well. But that's was just learning the IDE, not a zillion obscure compiler options. This isn't really the fault of the C++ language itself, but it doesn't seem to be a common feature of C++ compilers.

Once I got Suneido to build I started poking around seeing what the IDE was like. One of the things I looked for is an outline of the current file in the editor, showing the classes and their members. I assumed this must exist and that I just wasn't looking in the right place. But  when I started searching on the web, I found that the closest thing it has is a Class View, but it shows all the classes in the project - a huge list for a big program. There are some add ons to do this but no good free ones. It's not a big deal but it's such a main feature of Eclipse that I've come to expect it. Even the Suneido IDE has an outline! (I did discover there were pull down lists at the top of the editor pane that provide similar functionality, but it's not quite the same as an outline view.)

If I just want to build the code, I prefer the command line. But if want to navigate around in the code and look for things, then the IDE is nice. Refactoring support is another reason to use an IDE, but Visual Studio doesn't have any refactoring for C++ (it does for C#). There are commercial 3rd party add-ons for refactoring like Visual Assist X (which also has an outline view)

I very much doubt if I've managed to get the settings identical between the makefile and the IDE project. (I know for sure I haven't in the case of XP support.) So which one should I build with? I think I can build the IDE project from the command line if I want. That would presumably ensure I was at least getting consistent settings.

I had one mysterious crash when I first started testing this build of Suneido. But since then I  have used it quite a lot without any problems. Maybe I changed (fixed) something in between, but I'm not sure about that.

As usual, the code changes and the Visual Studio solution are in version control on SourceForge. If you try building it, let me know how it works.

The Evolution of a Feature

The application software that we sell has gotten bigger over the years. We now have roughly 750,000 lines of (Suneido) code. That's peanuts to someone like Microsoft, but it's a lot for a handful of programmers in a small company.

Mostly I work on the Suneido implementation itself i.e. the C++ and Java code. But recently, for a change, I've been doing some programming in Suneido. (My staff doesn't always appreciate this because I tend to be a little more "aggressive" than they would be, and as a result I've been know to break a few eggs in the process of making my omelettes.) I quite enjoy the chances I get to actually use the language I've designed and implemented. And unlike C++ and Java, if there's something I don't like, I have the option of fixing it!

As our code has grown navigating through it has become more of an issue. I might vaguely remember the name I'm looking for, but I have a hard time remembering if it's FooBar or Foo_Bar, or Foobar. We have naming conventions, but there's still variation.

A lot of the time you can follow references through the code using "go to definition", but sometimes you need to start from scratch.

We use a "Find" function in the library view, basically a "grep". That works, but it's not ideal. It's too awkward to write regular expressions like "Foo_?Bar" every time.

Several years ago I improved the Find function to search the names by converting both the actual names and the search to a canonical form, e.g. converting to lower case and stripping out underscores. It then "scored" the matches.

This was an improvement, but it didn't show you a list of matches, it just let you step through the matches. There were still lots of times when the first match wasn't what you wanted.

About the same time, I added auto-completion in the code editor. So you can start typing a name and it would display a list of matches. At least this way you could see a list and pick the one you wanted.

I noticed programmers go to the work space, start typing a name, use auto-completion to get the right one, and then go to the definition. This worked, but it's a little round-about!

My recent next step was to change the name search field in the "Find" dialog to do auto-completion. This avoided having to go the workspace route.

But because of the way Scintilla (the code editor component we use) works, auto-completion can't ignore underscores or do other kinds of matching.

Also, it seemed a little ugly to auto-complete the exact name, and then go searching for it, even though no searching is necessary in this case.

So my most recent improvement is to add a "locate" field (at the top-right of the library view). And instead of using the Scintilla auto-completion, I used a different auto-complete control we had which allows more flexible matching. (In the process finding and fixing a bunch of bugs in this control.)

I also got the idea to add what Eclipse calls "camel case searching", being able to search using just the capitalized letters in a camel case name. e.g. using "oic" to search for OpenImageControl. This is quite handy when names get long.

The simplest way to implement this would be to just perform a linear search of all the names. But we might have roughly 20,000 active names, and doing a complex search linearly is fairly slow. (Although perhaps usable.)

The auto-completion uses a sorted list of names and does a binary search in it. This is fast, partly because Suneido has a built-in lower bound binary search.

But how do you implement something like camel case searching with a binary search?

I realized that I could do it by putting multiple entries in the list. e.g. for OpenImageControl there would be two entries in the list, one for "openimagecontrol" and one for "oic". I also needed the original name so the entries in the list look like: "oic=OpenImageControl".

Suneido also lets you overload the same name in multiple libraries (a much abused feature!). So I added the library to the entries e.g. "oic=OpenImageControl:stdlib".

But when I sorted those entries, the library part sorted alphabetically, which isn't a useful order. I wanted to sort the library part in the same order they were being used. So I changed the entries to use a library index instead of name e.g. "oic=OpenImageControl:01"

The auto-complete does a binary search to find the matches, removes the part up to the equals sign, sorts and uniques it, then converts the library back to the name, resulting in e.g. "OpenImageControl (stdlib)"

The other question is when to build and update the list. It takes a noticeable amount of time to build the list (about 1/2 a second on a fast machine) so I didn't want to do too often. And yet I wanted the list to be up to date. One option is to update it when the GUI is "idle". But most of the time it wouldn't need updating. Ideally, I wanted changes to "invalidated" the list, and only update it when it was next required. Unfortunately, there's no easy way to be notified of library changes. I ended up using two things to determine if the cached list was still valid. First, if the libraries in use changed, then the list had to be updated. Second, if the maximum "num" id field in any of the libraries changed, that meant a record had been added and the list needed to be updated. But checking the num field meant database queries so I didn't want to do that too often so I throttled it to check at most every 15 seconds. That still leaves a slight lag occasionally when you use the locate feature (if an update is needed) but it's rare enough (and brief enough) not be be annoying.

I only just finished this feature so it's hard to say how it will work out, but so far it seems like a good (or at least better) solution.