In this special guest feature, Eric Frenkiel, Co-founder and CEO, MemSQL writes about the three myths surrounding in-memory computing and how companies that don’t take advantage of IMC risk being left behind.
Got Big Data? Need to do business in real-time? Then get in-memory computing (IMC)—or at least, that’s the takeaway from the “Cool Vendors in In-Memory Computing Technologies, 2014” report by Gartner, Inc.
IMC technology is increasingly emerging as a key enabler for the digital business by empowering the agility, Web-scale processing and fast decision making needed to respond to the business challenges of the digital era,” states the Gartner report. “It completes batch processes that would otherwise take hours in minutes or even seconds, enabling processes to be delivered … as real-time cloud services… [and helping] pinpoint emerging opportunities and threats as things happen, across millions of events, in the blink of an eye.”
IMC is a simple concept: keep data in silicon-based main memory rather than out on mechanical disk storage for faster and more predictable execution. IMC can deliver a ten-fold or twenty-fold increase in performance over disk-based models, making it a perfect match for applications that demand real-time data analytics. Starting in the 1990s, IMC has been proven in practice, especially in high-speed, high-demand applications, such as telecommunications, network equipment, and mobile advertising networks.
Those demanding applications are a hint of what’s to come in almost every industry. Across the board, companies are dealing with large and rapidly increasing data stores, and trying to meet increasing customer demands for real-time response. While IMC was once the leading-edge technology for companies in the forefront, it’s on its way to becoming the standard. But a few myths are keeping organizations from thinking of IMC as an appropriate solution to their own challenges.
Myth 1 – Memory is expensive
The core principle of IMC—keep data in main memory—is so obvious and intuitive that in a perfect world, we never would have put data anywhere else. Why would we mess with network storage and racks of disk space, when it’s faster to access data in main memory?
The answer is cost. Once upon a time, putting all your data in memory would have been an enormous extravagance—but that’s no longer true. Memory is cheap, which means IMC is affordable. New developments in solid-state drive technology and specific software platforms have made this possible—plus the fact that DRAM and NAND flash memory prices have plummeted since 1995, from approximately $30/MB to $.0073/MB.
Myth 2 – In-memory can’t scale out
Sure, when memory was expensive, it wouldn’t have been affordable to scale IMC up for enterprise systems. But now the tables have turned, and IMC is widely considered the only solution that can handle Big Data for enterprises. IMC enables organizations to capture, store and query hundreds of terabytes in real-time, without the performance drag of batch loading and ETL.
Distributed databases are key to scaling up IMC. Proven in high-velocity Big Data environments, they are an order of magnitude faster than traditional databases, don’t require batch loading, and scale out horizontally on commodity hardware.
Myth 3 – Flash is as fast as in-memory
In-memory is the fastest way to store and access data, so it’s a better choice than flash in real-time applications.
But with the need to access huge quantities of data for certain analytics processes, flash memory still has its place, for example, as part of a tiered architecture. A tiered architecture enables companies to leverage an in-memory row store for transactional and time-sensitive workloads, then seamlessly move data to a highly compressed column store on low-cost flash technology for further analysis without costly ETL.
How companies are using IMC
Finance is an obvious fit for IMC. For example, to support an asset management business, IMC can enable real-time analytics on data feeds of variable structure and support rapid scale out on commodity hardware.
Gaming is another example of a high-volume, real-time application. Some social games have millions of active users daily, which means game providers need to analyze vast amounts of data to produce fast insight from Big Data. IMC enables such companies to make decisions based on billions of data points in real-time to provide better in-game personalization and greater overall customer satisfaction.
Get over the myths and get started with IMC
Companies that don’t take advantage of IMC risk being left behind. Legacy technologies will not be able to keep up with the speed, innovation, or price savings that IMC enables. Gartner cautions, “Application architecture leaders should master IMC architectures and technologies, or run the risk of undermining their organizations’ ability to leverage digital business opportunities.”
So stop thinking about whether you’ll adopt IMC—and start thinking about when. And when you do, consider this piece of advice from Gartner: “Don’t just consider megavendors for your IMC initiatives; small, innovative pure players often provide more cost-effective and/or more technically valid alternatives to large vendors’ offerings.”
Sign up for the free insideBIGDATA newsletter.