30 August 2012

I started work on CakeDB many months ago when one of my many projects started overwhelming MongoDB. Now before anyone gets the wrong idea, the reason it overwhelmed Mongo is because I don’t have enough RAM for the 800GB data set and nor do I have the money, space or inclination to run the number of nodes I would need to scale it up to that level. So it’s not that Mongo can’t do it, it’s that it can’t do it on the hardware I have.

I also noticed that Mongo’s goals were different from my own. I was mostly storing event data (such as tweets or price feed data from my forex broker) and at some point I might come back and look at it. How fast it makes it to disk, doesn’t really bother me. However when you have an application that can burst pretty high, you need a database  that’s not going to tie itself in knots. So I also needed a database that could accept data very quickly, with minimal effort from the client with a non-blocking architecture.

In short, I needed a database that I could hammer with burst traffic and that would at some point get the data on to disk so that later on when and if I got around to it I could query the data. It also had to handle reasonably large datasets (i.e. 1TB or above) and be able to respond to queries in a reasonable time. I use this for reporting so instant response is not necessary, I can wait a few seconds but I don’t want to wait 20 minutes just to see how many new updates were added today.

I did some looking online and looked at a few alternatives but nothing seemed suitable and so I decided to  build my own. CakeDB is pretty much flat files on steroids. It’s very much a niche database that I will most definitely be using in conjunction with MongoDB. I will also be releasing it under GPLv3 so that other people may enjoy the humour of my attempts to write an application that’s so far out of my depth a bouy and shark fin would not be amiss!

Last but by no means least, I have set up a site for it at http://cakedb.org and have what I think is an awesome logo:

More updates to follow and I will be setting up the CakeDB site as a blog in the near future. If you’re interested in CakeDB or have any questions, please feel free to get in touch :)

blog comments powered by Disqus