In this special guest feature, Chris Surdak, JD of HP discusses observational bias and how it can work to taint the result of big data analytics. Chris is Global Subject Matter Expert on Information Governance, analytics and eDiscovery for HP Autonomy. He provides guidance and expertise to C-Level executives in how Autonomy’s unique and revolutionary technology can transform how their businesses operate. Chris is also author of the popular tech industry title “Data Crush,” Benjamin Franklin Innovator of the Year, 2015 and IGI’s Information Governace Evangelist of the Year, 2015.
How Observational Bias May be Killing Your Big Data Efforts
Summary
Anyone who has spent more than 3 minutes watching any TV news programming is familiar with the concept of observational bias. When a talking-head reporter asks some politician, “Are you for murdering babies or murdering convicts?” or if they ask, “Should starving children or homeless veterans be allowed to die?” observational bias is at work (actually, these are variations of omitted variable bias, too, but I’ll not nitpick here).
Similarly, when a business person asks the question, “Would you like to make a profit, or waste our resources?” what they are really asking is, “Do you dare to have the temerity to disagree with me?” There really isn’t a question there, it’s more of a threat, or a dare.
Nearly all questions asked by mere mortals are tainted by our observational biases. Rarely are our intentions pure when we ask questions of the world around us, and our biases can significantly impact the results of our analytic efforts. In a world of Big Data such biases may not only be disingenuous, they can be disastrous. As such, to be successful at Big Data you must have the courage to be aware of your own observational biases and be willing to ask questions that challenge rather than reinforce them. If you can’t do this, if you can’t move beyond your own biases and discover the greater truth beyond, why bother asking?
MBA: Might Be (Slightly) Arrogant?
While I completed my MBA many, many years ago, my memories of that time are still very fresh. Like many MBA students I was looking forward to making a giant leap in my career after graduation. Wall Street, corporate executive fast-track, internet start-up mogul all called to me, yet I chose the path that seemed sexiest at the time: strategic management consultant.
In the mid 1990’s nothing in business was cooler than being an Ivy-League-trained analyst with one of the elite strategy consultancies, and I was eager to join the ranks of such business gurus. As MBA students we all believed that these firms were an express-train ticket to the C-Suite, and we twenty-somethings were going to take these outdated executives to school!
I was hired by such a strategy boutique, and was initially swept-up in the pomp and circumstance of the role. When a client engaged us, clearly they were in dire straits and required our stupendous intellects, our tireless energy, and our innate super-human abilities to save their poor firms from their incompetence. We would parachute in like Seal Team Six, armed with spreadsheets, data models and a relentless sense that we couldn’t possibly be wrong.
Brother, Can You Tell Me the Time?
As great as this all sounded, there was a darker, less impressive side of the whole strategy consulting business. Many people complained, then as now, that these firms never justify their incredible expense. These firms easily charge two or three times the going hourly rate of other business consultants; ostensibly because they’re just that much smarter than anyone else. But, the work they perform typically involves taking information that their clients already have in their possession and simply performing a different set of analyses against them. More cynically, some clients would say that these firms take your watch from you and then charge a few million dollars to tell you the time. Like comparing a Seiko to an Apple Watch, noon is noon, regardless of how clever the wrapping.
Cynical or not, there was no small degree of truth to this viewpoint. When we arrived at a client site to work our miracles our first step was to obtain the client’s data as rapidly as possible. Our senior consultants would perform any number of “executive interviews,” which was code for brown nosing over massively-expensive lunches, while we mere analysts would assault the client’s IT department in search of their data. We would move through IT room by room, employing our best sweep-and-clear techniques (free donuts or pizza ALWAYS yielded the best results) in search of client data that was presently being abused by the company’s merely-mortal technicians. We were on a hunt for raw material to recast as our brilliant insights and we would not fail in securing our digital Geronimo.
Dumpster Diving for Data
As an interesting aside, we’d also engage partners who were “dumpster divers;” people who would literally go through the client’s dumpsters (and naturally their competitors too), looking for any tidbits of information that might help us tell the time more convincingly. We would never do this work ourselves, we were far too ‘educated’ for that, since it was unseemly and it would ruin our Versace ties and Ferragamo loafers.
Ironically, we often got greater insights by sifting through what these companies threw away than we ever did in crunching through what they kept. This was long before the era of “discovering” unstructured data. Hence, the stuff that organizations threw away as irrelevant crud frequently was the information that gave us the best context and meaning for the stuff they decided to keep. Some things never change.
Indeed, looking at what organizations throw away, and hence don’t value, can give great clues as to their own observational biases. If you sincerely believe that something has no value you’re not likely to invest much time or energy into validating your prejudice. Indeed, this is what prejudice is all about; seeing through something’s value rather than acknowledging it.
Croissant Quandary
One of my clients from this era is a perfect demonstration of the dangers of observational bias. The client in question was a large manufacturer of consumer packaged goods (CPG). In particular, they were known as a major player in the baked goods business, with particular emphasis on the frozen foods’ aisle. Their business at the time was both declining in revenue and profitability (never a good pairing), and social trends indicated that their fortunes would continue to decline. The company was generating about $5 million per year in profits from $250 million in revenue; a return of 2%, which looked as awful in the mid 1990’s as it now looks reasonable here in 2015.
Such performance was simply unacceptable, and the company’s executives were more than a little nervous. The company was in a quandary, and they were convinced that they needed the help of our elite “Seal Team Six” to set them straight. Their goal in hiring us: use our analytic skills to find efficiencies in their business, enough to double their profitability. Basically, we were hired to figure out how to get their profits up to $10 million per year by cutting costs wherever possible. Sound familiar? It should, there’s nothing new about cutting corners to try to remain competitive.
Reefer Madness
One of the weird twists with this company is that, because of their poor performance, they had recently fired their Chief Financial Officer. Whether this was a case of his incompetence or his peers killing the messenger was never made clear. Nonetheless, there was a new CFO in place, and he was there to set things right. What was peculiar about this guy (we’ll call him Fred) was that he didn’t have a financial background. Rather, Fred was the company’s former head of logistics, and had been working in that arena nearly his entire career.
Why would this company pick Fred to be their CFO? Well, their analysis of their business up to that point had indicated that approximately 40% of their total cost structure was in logistics, that is, moving their product from factory to retail outlet. Because their products were frozen, they had to be stored in enormous refrigerated warehouses, and then transported in refrigerated trucking containers, known as reefers. All of this refrigeration cost a lot of money, especially in the South, which was one of their largest markets.
The company reasoned that this was where their $5 million of cost savings was waiting to be found, so it made rational sense to put a logistics expert in charge of finance in order to be more efficient at logistics. And so, they made a logistics expert their new CFO, hired us and another firm to look for those efficiencies, and then waited for us to start generating results which would tell them how to cut the cost of operating their reefers.
Our team invaded the company’s headquarters, and began hunting down the data that would allow us to tell them the time, using their watch. Once we had our data our analysis began in earnest. Over the course of a month we crunched through the numbers, coming up with detailed cost breakdowns for each product they sold, in each market across the country, down to the penny. Our analysis was excruciatingly-detailed, something our firm was known for, and it quite literally pushed the limits of the software and computers available to us at the time.
The results of our work supported our client’s view that logistics was a major source of cost for the business. What compounded their problem was that these costs were highly variable, and largely out of their control. If diesel fuel prices spiked, so did their logistics costs and their profits could instantly tank, so to speak. We developed a range of recommendations for how to streamline their operations, cut a corner here, eliminate an inefficiency there, and so on. We managed to identify perhaps half of the profitability they were looking for, which was good, but not great.
Fred was absolutely convinced that there was more money to be saved in their logistics operations, and signed us up for an extension to dig deeper into the problem. The savings HAD to be somewhere in their logistics. If it wasn’t, they had no idea where else the savings might come from.
Penny Wise, Pound Crazy
One of the interesting things I learned in this job was that frequently it was easier for us, as consultants, to get access to our client’s data than it was for the client’s own executives. As a result we often had insights that they didn’t have, and this provided the opportunity for some significant embarrassment.
While going through the detailed financials for this client, there were several line items that leapt out at me as, well, shocking. These line items detailed the money the client was spending on US, and the other consulting firm they had hired to find cost savings. The staggering thing was this: together our two firms had charged the client $8 million in consulting fees. This client had spent $8 million in trying to figure out how to save $5 million… Staggering, isn’t it?
Now, I could have recommended to them that in order to far exceed their goal of $5 million in additional profit all they had to do was fire us, but this would have been frowned upon by my own management, for sure. Hence, those line items were noted somewhere on page 212 of the addendum to our PowerPoint, never to be mentioned again. In the end, I think the client implemented some of our recommendations, and came up with maybe a couple million in additional profit. I heard from Fred about a year later, letting me know of his new job with another company. Not surprisingly, his new role was head of logistics, NOT CFO.
Different Results Through Discomfort
My experience with Fred and his company was an extreme, but not unique, example of observational bias at work. Fred believed that logistics would be the source of savings, and so all of his efforts and focus was on proving himself right. This is a very human trait, as all of us tend to believe that our own thinking is normally correct, and it’s others who are a little bit nuts.
The problem is our understanding of our world drives our existing behaviors, and if we want to create different results than we’ve already achieved, we need to change our thinking. As Einstein put it, “We can’t solve today’s problems with the same thinking that created them.” Breakthrough results require break through thinking and observations. To come up with new insights that lead to new results, we need to get out of our comfort zones, question our assumptions, experiences and biases, and embrace new thinking.
This ties directly to the current craze around Big Data. If your analytics efforts are merely reinforcing your existing biases and beliefs, why bother? I talk with hundreds of business people about their Big Data efforts, and consistently, most organizations aren’t seeing the awesome results that they have been told to expect. I think that much of this can be explained by their observational biases. They keep asking the same old questions of the same old data (just more of it), and they’re hoping for some kind of breakout results. Is it any surprise that this doesn’t work?
To succeed at Big Data you must seek to disprove your beliefs, or at least challenge them; your competition is. You must recognize your own observational bias, and you need to call yourself out on it. To get this right you have to think and act differently than you did before. If your analytic results lead to some surprising and maybe discomforting insights, you’re on the right track. And you might find that you don’t need to bring in someone from the outside to take your watch and tell you the time after all.
Sign up for the free insideBIGDATA newsletter.
Speak Your Mind