RethinkDB

I just listened to a podcast from the MySQL conference from RethinkDB about better database storage engines. Apart from being an interesting talk, a lot of what they were talking about parallels my own ideas in Suneido.

For example, they talk about log structured append-only data storage. Suneido's database has always worked this way.

Next they talk about append-only indexes. Suneido does not have this, but it is something I've been thinking about. (see my post  A Faster, Cleaner Database Design for Suneido). They have a different idea for reducing index writes. It's an interesting solution, but more complex. It won't be as fast as delaying writes, but it would allow crash recovery without rebuilding indexes (as Suneido requires).

I mostly arrived at these design ideas from basic principles e.g. immutable structures are better for concurrency. But it sounds like the file system folks have been working on a lot of the same ideas. It's hard to keep up with everything that might possibly be relevant.

The other interesting part of this talk was the idea that there are a lot of pieces that have to work together and performance depends on the combination. This means there is a huge possibility space that is hard to explore.