In-memory databases poised for takeoff
The classic database engines – like the big relational behemoths from IBM, Microsoft and Oracle – store the data on disk. So do many of the open-source databases, like MySQL and PostgreSQL, as well as the vast array of emerging NoSQL databases. While such database engines keep all the rows and columns on the relatively slow, disks, they can boost performance by putting some element, including indices and sophisticated predicted caches, on faster solid-state storage or even faster main memory.
From a performance perspective, it would be great to store everything in main memory. It’s fast, fast, fast. It’s also expensive, expensive, expensive, and in traditional services, is not persistent. That’s why database designers and administrators leverage a hierarchy: A few key elements in the fastest, most costly main memory; more data in fast, costly solid-state storage; the bulk in super-cheap rotating disks. In some cases, of course, some of the data goes into a fourth layer in the hierarchy, off-line optical or tape storage.
In-memory databases challenge those assumptions for applications where database response time is the bottleneck to application performance. Sure, main memory is still fabulously expensive, but it’s not as costly as it used to be. New non-volatile RAM technologies can make main memory somewhat persistent without dramatically harming read/write times. (To the best of my knowledge, NVRAM remains slower than standard RAM – but not enough to matter.)
That’s not to say that your customer database, your server logs, or your music library, should be stored within an in-memory database. Nope. Not even close. But as you examine your application architecture, think about database contents that dramatically affect either raw performance, user experience or API response time. If you can isolate those elements, and store them within an in-memory database, you might realize several orders of magnitude improvement at minimal cost — and with potentially less complex code than you’d need to manage a hierarchical database system.
Not long ago, in-memory databases were a well-kept secret. The secret is out, according to research from Evans Data. Their new Global Development survey says that the developers using in-memory databases has increased 40% worldwide from 18% to 26% during the last six months. An additional 39% globally say they plan to incorporate in-memory databases into their development work within the next 12 months.
