Why Big Data Isn’t Helping Retailers Innovate

Print Friendly, PDF & Email

jamie-rapperportIn this special guest feature, Jamie Rapperport of Eversight discusses how most major retailers today have access to massive volumes of transaction and customer data, and are starting to use more sophisticated analytics to better understand their customers and deliver a more personalized shopping experience. However, the industry is also becoming familiar with the shortcomings of this data. Jamie is the co-founder and CEO of Eversight and has 25 years of experience as a software entrepreneur. Prior to founding Eversight, Jamie was co-founder and EVP at Vendavo, the leading B2B pricing technology company. Prior to Vendavo, he served as a founder and vice president of marketing and sales at VXtreme, which was acquired by Microsoft and became the core of “Windows Media Player.” Jamie has a B.A. from Harvard University and an MBA from Stanford.

Big data, and the analysis of it, has been a key marketing tool of brick-and-mortar retailers for a very long time. Retailers often leverage the insights gleaned from big data to try to gain a crucial competitive advantage in a market where competition intensifies each year. However, these days, big data isn’t all it’s cracked up to be in the retail industry; particularly when it comes to the $300B retail trade promotions space.

Despite the fact that most retailers today use very sophisticated analytics that slices and analyzes data in all kinds of ways (loyalty, structured POS, transaction history, etc.), many are still floundering when it comes to building effective promotions. In fact, according to Deloitte over 70 percent of today’s promotional events lose money.

One of the biggest reasons why big data is failing retailers and brands when it comes to building effective promotions comes down to the tools being used. Current trade promotion optimization (TPO) solutions (which most brands use to determine how effective a promotions campaign was after the fact) simply weren’t designed to manage and extract meaningful insights from the enormous volume of data that modern retailers generate. They aren’t able to capture the level of granularity required to understand what differences in how an offer is “framed” could mean to its effectiveness. The result is that retailers and brands often aren’t aware of different offer structures, price points, call to actions, etc., which could influence the performance of a promotion by 20-50% or more.

In addition, and perhaps most critically, current TPO solutions don’t support the development of new, never-before-tried offers as they only evaluate the performance of events that have already been run. This means that possibilities for novel offer structures, cross-merchandising, discount levels, and so on can’t be found in the data. Using past transactions to determine go-forward strategy handicaps retailers from developing creative promotions that may work better than anything they’ve tried in the past. Furthermore, as retailers continue to optimize in a “backwards-looking” mode using transaction data, offers become increasingly homogeneous. This decreases both the impact on consumers and the ability to learn with each successive campaign.

As the industry continues to optimize in a “backwards-looking” mode using transaction data, the resulting set of promotions on retailer calendars will continue to converge, and the data will become increasingly homogeneous. This explains why certain brands only run one or two types of promotions; they simply have no other data points to look at.

There’s no doubt that retailers and brands are in desperate need of a new approach that doesn’t rely solely on last year’s data. Taking a page from the ecommerce playbook, recently a handful of well-known CPGs have begun adopting a digital testing approach called Offer Innovation. Taking advantage of the recent growth of omnichannel, they are testing new offer ideas by serving dozens of variations with different offer structures, discount depths and quantities, product combinations, marketing language, etc. to groups of test shoppers online (think print-at-home coupons, retailer mobile apps, Facebook, email, etc.). They’ve been able to leverage rapid online learning to (more effectively) inform new in-store promotional events such as feature, display, and TPR.

With new technologies like those described above gaining popularity with major consumer goods manufactures and retailers, there’s a renaissance in trade on the horizon – with promotions which are both more performant for retailers and brands and more relevant for consumers. As the effectiveness of digital test-based promotions becomes more clear, I’m confident that the rest of the retail industry will follow suit.


Sign up for the free insideBIGDATA newsletter.

Speak Your Mind



  1. M Connaughton says

    Where’s the data that “most retailers” use very sophisticated software to leverage big data? How are you defining “most retailers”? …of a certain size?

    I would say that is a generalisation and isn’t underpinned by data, which would go some way to undermining the article – not that retailers aren’t innovating (they aren’t), but why they aren’t.

    I would say why they aren’t is a failure to de-clutter and keep pace with change of consumer expectations and channel optimisation and taking a singular customer view – not that they’re using big data incorrectly.

    While it’s true there’s a tendency to backwards engineer legacy systems and adopt piecemeal or “bolted on” capabilities to existing processes, I would say “most retailers”, first and foremost, don’t have the right approach to customer strategy – let alone the technology.

    From experience, that comes with investment decisions; when fractional profit margin fluctuations are the difference between making investments and laying off staff, it’s clear to me where priorities are.