How will IoT affect Big Data in 2015?

In this special guest feature, Himanshu Bari of DataTorrent explores the rise in IoT-connected devices and how data collected by these sensors are exploding in popularity and destined to make 2015 the year of “fast big data.” Himanshu Bari is the Director of Product Management at DataTorrent, the enterprise-class real-time streaming platform built exclusively on Hadoop.

Data is being generated today in unprecedented volume, variety and velocity. Human generated data is being surpassed by automatically generated data, much of it from Internet of Things (IoT) related devices and sensors. Many consumer devices that we are familiar with are coming online – from smartphones to fitness trackers to home security systems to kitchen appliances – and data is poised to grow exponentially. Looking beyond consumer IoT, enterprises are instrumenting their factories, products and infrastructures to keep up with the increasing number of IoT devices.

For example, a door manufacturer might want to correlate temperatures, humidity levels, materials and employee schedules to increase yield. Another example is the connected sensors on trains that read the train’s performance and environment, including track condition. These sensors allow for predictive maintenance on the train and tracks for safety and cost containment. In each of these use cases, there is a clear requirement for a solution that is both easy to use and enterprise-grade in nature. Gartner estimates the number of IoT-connected devices will grow to 26 billion units within five years, a 30-fold increase from 2009. So, what will the Internet of Things mean for big data in 2015?

To start, IoT stands to make 2015 the year of “fast big data.” Fast big data is creating a need for providing analytics and the ability to take action in real-time on streams of data. Processing fast Big Data creates demanding requirements that many Big Data systems cannot meet. A fast Big Data application never “stops” running, as the data never stops being created.

With data being generated from mobile devices, wearables and sensors, critical insights into customer habits are also being generated. The value behind those insights decreases as time goes on, so organizations need to leverage technology that enables them to deliver tailored offerings to customers to generate revenue or create automated operational efficiencies to save cost.

What are the requirements for an enterprise-grade solution for IoT? Enterprises need a solution that is scalable, highly available, reliable, performant and secure (SHARPS). A SHARPS solution provides the operability that is required for an always-on, 24×7 application. Regarding ease of use, a solution should be flexible enough to allow data scientists, business managers and business analysts to set up complex analytics and take action in a visual environment. Additionally, a solution must be powerful enough to enable a developer to easily create custom applications using a well-know programming language like Java, all the while extracting out the complexity of the underlying system processing.

Organizations of all sizes are aiming to provide platforms that enable IoT applications to gather, store, analyze and distribute data. In 2015 it will be crucial for enterprises to find a way to implement IoT solutions that adapt and adjust at the speed of specific situations and businesses. If they don’t, it’s possible that they will find themselves left in the dust of all the fast big data.

Sign up for the free insideBIGDATA newsletter.