In this special guest feature, Jamie Clark, Senior Developer at Dimensional Insight, discusses why columnar databases are the future of business intelligence as massive amounts of data continue to accumulate. As senior developer at Dimensional Insight, Jamie Clark is the lead developer of DivePort and the server components of DiveTab, and he manages and writes code for Spectre. Jamie is an expert in the inner workings of almost every Dimensional Insight product. He joined Dimensional Insight after graduating from the Massachusetts Institute of Technology in 2001.
To succeed in business intelligence, you need to get results fast. However, with the vast quantities of data that now reside in organizations, fast analysis of that data is getting to be more and more difficult. Fortunately, new technology advancements are solving that problem. Today, more and more companies are turning to columnar databases for fast delivery of data analysis.
Columnar databases are not new – they’ve been around for years. But the cost has plummeted dramatically. In 1990, 1GB of RAM cost $103,880. Today, it’s just $5.50, making it much more affordable for organizations to use columnar databases. In fact, an October 2016 survey by Gartner shows that for three consecutive years, production deployments of columnar/in-memory technology have consistently increased.
The traditional relational databases have served the business intelligence industry well over the years, but columnar databases address some of the limitations inherent in relational databases. Let’s take a look at what sets columnar databases apart and why they are the future of business intelligence.
Hardware Advantages
For one thing, columnar databases can sort and aggregate data more quickly than relational databases by taking advantage of hardware improvements. A good relational database, on which many business intelligence products are built, can get results quickly even without fast hardware. But with the latest hardware advances, such as multiple cores that are faster in speed with built-in parallel processing, larger memory, and advanced compiler technology, columnar databases are able to take advantage of these upgrades to boost performance.
Speed of Columnar Databases
Columnar databases store data in column-major order, as opposed to row-major order in a standard relational database. That difference means that columnar databases allow for a single column to be kept in a continuous piece of memory. With a columnar database, values in each column are of the same data type, allowing the data to be highly compressed. This allows the software to take advantage of CPU caches for better performance. Users get faster calculations because only the relevant columns need to be accessed. Because the columnar database allows users to access only the relevant data elements, this increases the speed of a query even in a database containing millions of records.
Simplicity of Analyses
Stop and think about that for a second – millions of records. That’s what many companies are looking at for their analysis. Enterprise-level businesses are seeing rapid changes in the business environment. There has been an increase in the quantity and diversity of data in different formats from a variety of platforms as well as an increased need to combine views. That’s perhaps the greatest advantage of a columnar database: While relational databases can serve as an authoritative source for data and allow users to easily add or remove specified rows, columnar databases allow for an easier analysis of that data overall.
Business intelligence needs to keep up with the changes that are happening, and columnar databases allow businesses to do just that. Because of how the data is organized, columnar databases get results faster and allow for more efficient data analysis. They may have been around in the past, but columnar databases are ready to take business intelligence into the future.
Sign up for the free insideBIGDATA newsletter.
Speak Your Mind