Report: An In-Depth Look at Big Data Trends and Challenges

Our friends over at Qubole, the data activation company, announced the results of its 2018 Big Data Trends and Challenges report — an annual, global survey of IT and data professionals on the trends, challenges, and solutions of big data and machine learning initiatives across the enterprise. The survey found that the value achieved from big data in on-premises environments has significantly lagged behind expectations, and as a result, companies are rapidly shifting to the cloud for applications centered around machine learning (ML), and analytics.

According to the survey, 73 percent of businesses are now performing their big data processing in the cloud, up from 58 percent in 2017. The shift towards cloud is necessitated in part due to the ever-growing volume and diversity of data that companies are dealing with, as 44 percent of organizations now report working with massive data lakes over 100 terabytes in size.

Facilitated by this shift to the cloud, machine learning programs are expected to expand across a wide range of use cases in the next year. A majority of respondents cited improving data security and threat protection as the top priority for their machine learning initiatives while optimizing customer experience (49 percent) and predictive maintenance (43 percent) were also high on the list of ML priorities.

Apache Spark and Presto have also shown impressive gains among big data frameworks in the last year. 31 percent of respondents now report using Spark as their framework, with a 29 percent growth from 2017. Presto, while in use by a smaller 13 percent of companies, saw its user base grow 63 percent in the last year. The survey results also indicated a shift by organizations away from homegrown approaches in favor of open source technologies.

The size, diversity, and applications of big data are accelerating at a near-exponential rate, and businesses are quickly discovering that traditional data management systems and strategies are no longer capable of supporting their demands,” said Ashish Thusoo, co-founder and CEO, Qubole. “Instead, a new generation of cloud-native, self-service platforms have become essential to the success of data programs, especially as companies look to expand their operations with new AI, machine learning and analytics initiatives.”

Additional findings from the report include:

Data teams continue to face a number of challenges when implementing both big data and machine learning projects:

• Respondents cited a lack of experience slowing project progress (44 percent), struggling to keep up with new data sources (42 percent) and issues with constantly evolving use cases (41 percent) as their top challenges.
• On the machine learning side, analyzing extremely large data sets (40 percent), ensuring adequate staffing and resources (38 percent) and integrating new data into existing pipelines (38 percent) were cited as the primary obstacles to machine learning projects.

Talent shortages are also a major issue for businesses:

• 79 percent of companies are looking to increase their data team headcount in the next year, but 83 percent also say it is difficult to find data professionals with the right skills and experience.
• 75 percent of respondents also reported that a sizeable gap exists between the potential value of the big data available to them, and dedicated tools and talent dedicated to delivering it.

A majority of businesses expressed plans for moving to a self-service analytics model, but few currently have it in place:

• 9 percent of respondents reported that their business already supports self-service analytics.
• 61 percent reported that they have plans to move to a self-service analytics model.

To view the complete survey findings, the 2018 Big Data Trends and Challenges report is available for download from the insideBIGDATA white paper library HERE.

Research Methodology
The research report, commissioned by Qubole and conducted by independent research agency Dimensional Research, polled 401 IT and data professionals with big data responsibilities globally. The survey was administered electronically and participants were offered a token compensation for their participation.

Sign up for the free insideBIGDATA newsletter.