As 2014 draws to a close, here at insideBIGDATA we’re trying to cool our heels a bit to reflect on the amazing year we had for Big Data. The year saw so much progress in terms of defining the problems to be solved by the big data technology stack as well as how to go about strategic deployments. As we play the role of arbiter of news for our growing industry, another progression that was very prevalent to us was the astronomical surge in the number of new companies operating in this space. Innovation in big data is happening at a very fast clip! We thought it would be interesting to put together a 2015 Predictions article that included perspectives from many of our new (and old) industry friends. Below, is a compilation of some of the most compelling predictions we heard. What do you all think? Leave us a note!
Organizations move from Data Lakes to Processing Data Platforms.During the past year, data lakes and data hubs have represented a popular first deployment for Hadoop. A data lake or data hub is a scalable infrastructure that’s both economically attractive (reduced per-terabyte cost) and agile—it has the ability to store various forms of both structured and unstructured data. The ability to use thousands of servers and store petabytes of data at less than $1,000 per terabyte per year is a core benefit of Hadoop. In 2015, data lakes will evolve as organizations move from batch to real-time processing and integrate file-based, Hadoop, and database engines into their large-scale processing platforms. In other words, it’s not about large-scale storage in a data lake to support bigger queries and reports; the big trend in 2015 will be around the continuous access and processing of events and data in real time to gain constant awareness and take immediate action. — John Schroeder, MapR CEO and Cofounder.
Mobile will force a fundamental change in the approach to BI. When it comes to mobile BI, adoption has been shockingly poor because it doesn’t usually work well with mobile devices. You can’t read data in depth on a mobile device, but rather you need to get to the point quickly. With everything shifting to mobile, the approach to BI will change. Rather than elaborate visualizations, you will see hard numbers, simple graphs and conclusions. For instance, with wearable devices, you might look at an employee and quickly see the KPI (key performance indicator). The BI game is about to change — primed to go mobile this year. — Adi Azaria, co-founder, Sisense
Big Data in 2014 was supposed to be about moving past the hype toward real-world business gains, but for most companies, Big Data is still in the experimental phase. With 2015, we will begin to see Big Data delivering on its promise, with Hadoop serving as the engine behind more practical and profitable applications of Big Data thanks to its reduced cost compared to other platforms. — Mike Hoskins, CTO at Actian
The data scientist role will not die or be replaced by generalists. In 2014 we saw c-suite recognition of Chief Data Officers as key additions to the executive team for enterprises across industry sectors. Data scientists will continue to provide generalists with new capabilities. R, along with BI tools, enable data science teams to make their work accessible through GUI analytics tools which amplify the knowledge and skills of the generalist. — David Smith, chief community officer, Revolution Analytics
The Internet of Things will gain momentum, but will still be in the early stages throughout 2015. – While we’ve seen the number of connected devices continue to rise, most people still aren’t putting network keys into their toasters. … Expect toasters to get connected, eventually, too — as people get readings of the nutritional value of the bread they toast, their energy consumption, and their carbon footprints, among other things — but the more mundane items won’t be connected for another year or two yet. — Information Builders
Big Data becomes a “Big Target.” As the bad guys realize big data repositories are a gold mine of high value data. — Ashvin Kamaraju, VP Product Development at Vormetric
Data will finally be about driving higher margins/bringing value to the enterprise. A lot of the “confusion” around big data has been defining it. 2015 will see companies deriving value. — VoltDB
Big Data moving to the cloud – enterprises are increasingly using the cloud for Big Data analytics for a multitude of reasons: elastic infrastructure needs, faster provisioning time and time to value in the cloud, and increasing reliance on externally generated data (e.g., 3rdparty data sources, Internet of Things and device generated data, clickstream data). Spark – the most active project in the Big Data ecosystem – was optimized for cloud environments. The uptake of this trend is evident in the fact that large enterprise vendors – SAP, Oracle, IBM – and promising startups are all pushing cloud-based analytics solution. — Ali Ghodsi, Head of Product Management and Engineering Databricks
IOT drives stronger security for big data environments. With millions of high-tech wearable devices coming on line daily, more personal private data including health data, locations, searching histories, shopping habits will get stored and analyzed by big data analytics. Securing big data using encryption becomes as inevitable requirement. — Ashvin Kamaraju, VP Product Development at Vormetric
SQL will be a “must-have” to get the analytic value out of Hadoop data. We’ll see some vendor shake-out as bolt-on, legacy or immature SQL on Hadoop offerings cave to those that offer the performance, maturity and stability organizations need. — Mike Hoskins, CTO at Actian
In 2015, enterprises will move beyond data visualization to data actualization. The deployment of applications will require deployment of production-quality analytics that become integral to applications. — Bill Jacobs, Vice President of Product Marketing, Revolution Analytics
Big Data will turn to Big Documentation – Most people won’t know what to do with all of the data they have. We already see people collecting far more data than they know what to do with. We already see people “hadumping” all sorts of data into Hadoop clusters without knowing how it’s going to be used. Ultimately, data has to be used in order for it to have value. … Big Documentation is around the corner. — Information Builders
The Rise of the Chief-IoT-Officer: In the not too distant past, there was an emerging technology trend called “eBusiness”. Many CEOs wanted to accelerate the adoption of eBusiness across various corporate functions, so they appointed a change leader often known as the “VP of eBusiness,” who partnered with functional leaders to help propagate and integrate eBusiness processes and technologies within legacy operations. IoT represents a similar transformational opportunity. As CEOs start examining the implications of IoT for their business strategy, there will be a push to drive change and move forward faster. A new leader, called the Chief IoT Officer, will emerge as an internal champion to help corporate functions identify the possibilities and accelerate adoption of IoT on a wider scale. ParStream
The conversation around big data analytics is becoming less about technology and more about driving successful business use cases. In 2015, we’re going to see a continuous movement out of IT and into generating ROI-orientated deployment models. Secondly, I think we’re going to see resources move away from heavy in-house big data infrastructure to big data-as-a-service in the cloud. We’re already seeing a lot of investment in this area and I expect this to steadily grow. — Stefan Groschupf, CEO at Datameer
The cloud will increasingly become the deployment model for BI and predictive analytics – particularly with the private cloud powered by the cost advantages and of Hadoop and fast access to analytic value. — Mike Hoskins, CTO at Actian
Big Data meets Concurrency – New Big Data applications will emerge that have multiple users reading and writing data concurrently, while data streams in simultaneously from connected systems. Concurrent applications will overtake batch data science as the most interesting Hadoop use case. — Monte Zweben, co-founder and CEO of Splice Machine
The term “Business Intelligence” will morph into “Data Intelligence.” BI will finally evolve from being a reporting tool into data intelligence that every entity from governments to cities to individuals will use to prevent traffic, detect fraud, track diseases, manage personal health and even notify you when your favorite fruit has arrived at your local market. We will see the consumerization of BI where it will extend beyond the business world and become intricately woven into our everyday lives directly impacting the decisions we make. — Eldad Farkash, co-founder and CTO, Sisense, Sisense
Self-Service Big Data Goes Mainstream.In 2015, IT will embrace self-service Big Data to allow business users self service to big data. Self-service empowers developers, data scientists and data analysts to conduct data exploration directly. Previously, IT would be required to establish centralized data structures. This is a time consuming and expensive step. . Hadoop has made the enterprise comfortable with structure-on-read for some use cases. Advanced organizations will move to data bindings on execution and away from a central structure to fulfill ongoing requirements. This self service speeds organizations in their ability to leverage new data sources and respond to opportunities and threats. — John Schroeder, MapR CEO and Cofounder.
‘Data wrangling’ will be the biggest area requiring innovation, automation and simplicity. Modeling and wrangling data from disparate systems into shape for insights for decades has been lengthy, tedious and labor-intensive. Most organizations today spend 70-80% time modeling and preparing data rather than interacting with data to generate business-critical insights. Simplifying data prep and data wrangling through automation will take shape in 2015 so businesses can reach a fast-clip on real data-driven insights.— Sharmila Mulligan, CEO and founder of ClearStory Data
In the coming year, analytics will have the power to become the next killer app to legitimize the need for hybrid cloud solutions. Analytics has the ability to mine vast amounts of data from diverse sources, deliver value and build predictions without huge data landfills. In addition, the ability to apply predictions to the myriad decisions made daily – and do so within applications and systems running on-premises–is unprecedented. — Dave Rich, CEO, Revolution Analytics
Increasing Role of Open Source in Enterprise Software – Data warehousing and BI has long been the domain of proprietary software concentrated across a handful of vendors. However, the last 10 years has seen the emergence and increasing prevalence of Hadoop and subsequently Spark as lower-cost open source alternatives that deliver the scale and sophistication needed to gain insights from Big Data. The Hadoop-related ecosystem is projected to be $25B by 2020, and Spark is now distributed by 10+ vendors, including SAP, Oracle, Microsoft, and Teradata, with support for all major BI tools, including Tableau, Qlik, and Microstrategy. — Ali Ghodsi, Head of Product Management and Engineering Databricks
Personal predictive technology will come to the forefront – and fail. – As analysts see that they can use data discovery and other analytical tools, they’ll want the power of predictions to fall within their grasp, too. Unfortunately, most people don’t understand statistics well enough (see the “Monty Hall problem”) to make predictive models that really work, no matter how simple the tools make it. If anything, simple tools are more likely to get them into trouble. — Information Builders
Real-Time Big Data – Companies will act on real-time data streams with data-driven, intelligent applications, instead of acting on yesterday’s data that was batch ingested last night. — Monte Zweben, co-founder and CEO of Splice Machine
Analytics in the cloud will become pervasive. As organizations continue to rely on various cloud-based services for their mission-critical operations, analytics in the cloud will become a prevalent deployment option. Not only does the cloud offer self-service, intuitive experiences that allow organizations to implement an analytics solution in minutes, it speeds insight and enables data sharing across internal and external data sources. By reducing the over-reliance on IT, users can focus on asking new questions and find new answers at a unprecedented rate.— Sharmila Mulligan, CEO and founder of ClearStory Data
The end of the data-hoarding era – Companies will begin to realize that they’ve amassed data to the point of hoarding. In 2015, smart business leaders will start to demand more immediate returns from data to directly solve real business problems. Forward-thinking CEOs will curb spending on data infrastructure technologies and invest more heavily in tools that explain, communicate and derive meaning from data. The business metrics will shift from understanding “How many terabytes of data do we have?” to “How much do we know?” — Kris Hammond, P.h.D., cofounder and chief scientist at Narrative Science and professor of computer science at Northwestern University
Predictive Analytics will be a key differentiator for modern apps: Embedded predictive analytics, e.g. on devices, in memory or in real-time analytics systems will become more frequent. Consumers will expect their software to anticipate their needs, driving requirements for predictive technologies in all apps. Spatially aware apps will become more intelligent and more common – the “who is near me” functionality of technologies/apps like Waze and Tinder will become more intelligent and prevalent in other apps. — Simon Arkell, CEO, Predixion Software
Bye, bye vizzy: The ‘data discovery’ market will undergo a major upheaval in 2015 with the acquisition of visualization vendors. The driving forces behind market consolidation include companies’ desire to acquire new customers and product portfolios, coupled with revenue targets. However, with the freemium model now so popular, data discovery vendors must offer more value than just visualization.
Business users will also increasingly question the value of visualization tools that don’t have data quality capabilities embedded. An overload of unintelligible data will see more demand for better quality data and data preparation technologies leading to a surge of start-ups offering these services. — Charles Clark, CEO of Rosslyn Analytics.
Sign up for the free insideBIGDATA newsletter.