Comprehensive End User Data Preparation Market Study Defines an Emerging Space

In this special guest feature, Adam Wilson of Trifacta talks about a recent market research study that points to the importance of end user data preparation in the big data space. Adam Wilson is CEO of Trifacta, a software company developing productivity platforms for data analysis, management and manipulation. Adam has spent eighteen years in leadership roles focused on data integration and analytics. As GM of Informatica’s Information Lifecycle Management division, Adam grew revenues more than 400% in five years. Prior to that, as SVP of Product Management & Marketing for Informatica’s flagship data integration products, Adam was instrumental in helping the company expand beyond ETL and grow product revenues to more than $200M. Adam also co-founded and was COO of Zimba, a pioneer in mobile business intelligence.

Our founders started Trifacta based on two key insights, data preparation had become a source of great pain and inefficiency, and as a result, an entirely new technological approach was necessary. We are delighted that a recent report by Dresner Advisory Services, a well-respected research firm in the Business Intelligence (BI) community, headed by former Gartner analyst, Howard Dresner, further validates our founding insights.

The first-ever “End User Data Preparation Study” conducted as part of the Wisdom of Crowds™ Market Study series shows that an overwhelming percentage of respondents recognize the need to improve end user data preparation. In the study, 85 percent of respondents indicated that end user data preparation is “important to critical” with 65 percent considering it “very important to critical.”

The study also validates the unique combination of visualization, machine learning and Predictive Interaction™ technology designed into the Trifacta platform. Trifacta ranked first among 26 vendors in the study based on five criteria, usability, integration, output, data manipulation and deployment features. This is a great endorsement for the work we’ve done to make the platform intuitive and ensure that the people most familiar with the data, and its context, are the ones doing the data preparation.

In addition to showing a rapidly expanding market for data preparation tools, the Dresner study also reveals unique characteristics of a maturing data preparation space:

For sales, marketing and finance, the data preparation pain is especially acute

While enterprises across the board recognize the need for data preparation, highest demand comes from sales, marketing and finance, where more than 70 percent of respondents rated data preparation as either “very important” or “critical.” Yet, even the category with the lowest demand, Information Technology, still had 60 percent of respondents rate data preparation as “very important” or “critical.”

Organizations small and large need data preparation tools

Demand for data preparation tools peaks at either end of the company size spectrum. “Small organizations of between 1 and 100 people, and large organizations with more than 5,000, are most likely to take advantage of end user data preparation and most likely to consider it critical,” according to the report.

Business users are already doing data preparation

Business users are already doing data preparation – a lot of it. In the report, 66 percent of respondents said they engaged in data preparation either “frequently” or “constantly.”

Current data preparation products aren’t effective enough

Whether relying on scripting, Business Intelligence tools or excel, the study found that the existing approaches to end user data preparation are not meeting market expectations. Thirty-five percent said current tools are “partially to totally ineffective,” further highlighting the need for innovation in the end user data preparation space.

Data preparation platforms need to support a variety of functions

According to more than 70 percent of those surveyed, critical data preparation functions include:

• Aggregating, pivoting and reshaping
• Deriving new data features
• Structuring raw data
• Normalizing, standardizing, cutting, merging and replacing of values
• Manipulating the order of data transformation steps

Trifacta came in first among the 26 vendors in the study due to the platform’s ability to support all of these data preparation demands.

Users need to output data in a multitude of formats

More than 90 percent of those surveyed would like to be able to output their results for personal use, either as Excel spread sheets or in CSV files. More than 70 percent want tools that will let them output to databases for sharing with colleagues and more than half want to be able to export directly to popular BI tool formats. Trifacta allows users to output data in an extensive variety of formats including nearly all of the formats in wide-use at enterprises.

Sampling and pushdown processing are critical

Scheduled execution of data transformation, sampling for interactive tests and pushdown processing of data transformations are ranked “important to critical” for 90 percent of respondents and “very important to critical” for 70 percent.

The Dresner Advisory Services’ report mirrors what we have been seeing from current and prospective clients – there is a market need for new solutions that enable a growing set of users to interact with and prepare data for analysis. We look forward to witnessing market growth and are confident that the strategy and talent guiding the development of Trifacta will solidify our leadership position.

Sign up for the free insideBIGDATA newsletter.