Don’t Solve the Wrong Big Data Problem

In this special guest feature, Todd Hinton, VP Engineering and Data Management at RedPoint Global, discusses how big data has conquered the world, but as with any technology, you need to proceed with caution when looking to deploy a big data solution. If you’re not careful, you could end up with a solution that costs a lot of money and doesn’t do what you need. Todd brings over two decades of product management and executive leadership to the company, providing strategic direction for RedPoint’s data management products, including RedPoint Data Management for Hadoop. Prior to joining RedPoint, Todd served as executive vice president for Bernard Data Solutions, where he was responsible for the overall technology direction of the company’s CRM SaaS application serving the nonprofit industry. Todd specializes in data quality and the creation of building high-performance database applications capable of querying vast amounts of data in high-volume environments. Todd has served in a number of executive management roles during his career, including general manager and CTO in a division at MarketModels, as well as director of postal products for Qualitative Marketing Software/Sagent Technology.

Big data has conquered the world. Customers increasingly use multiple channels and devices to engage with brands, generating a veritable flood of data points that grows exponentially every year. To say that big data has moved beyond a buzzword and into a fundamental fact of your business life is understating the reality that brands now face: big data isn’t going away.

Consider that recent research from Northeastern University found that the amount of data in the world will grow to 44 zettabytes by 2020, with 2.5 exabytes of new data created every single day. This equates to 2.5 quintillion bytes of new data from social media, web traffic, and other sources created every hour of every day. The explosion of data translates into a massive expansion in the amount of data that companies collect, store, and manage, which results in a greater interest in big data technologies.

The interest is well-founded because of the huge volume, velocity, and variety of data now being created. Big data technologies can make a powerful impact on your business, especially from the perspective of managing increasing volumes of data and creating business value. But as with any technology, you need to proceed with caution when looking to deploy a big data solution. If you’re not careful, you could end up with a solution that costs a lot of money and doesn’t do what you need.

Big Data Technology Can’t Solve All Your Problems

Modern big data technologies are built to solve specific problems related to the very large datasets that are considered “big data” in the marketplace. The related solutions are extremely powerful, and can streamline operations dramatically where you have those kinds of data volumes. That said, big data technologies are nevertheless software built specifically to manage and derive value from datasets that are too large or too complex for traditional data processing applications. These are technical solutions to technical problems.

One thing that big data solutions cannot do is solve problems, like organizational issues, for which they were not designed. Seek out “big data technology,” and there is a good chance that you will find a host of companies willing to accept your business. But beware – if you begin the purchase process for a big data solution without an idea of what you want to achieve, you risk losing both time and money in the long run and finding yourself stuck with an expensive component of your technology stack that doesn’t provide business value.

Start with a Business Problem

It may be an old piece of advice, but if you start by defining the problem you want to solve, you’ll end up with a better result. A clear definition of your business problem focuses any technology search onto solutions that will solve your specific use-case first and foremost. You’ll likely gain other efficiencies along the way, but the core of your search should ideally focus on whatever specific objective you want to achieve.

If you have a lot of unstructured data, for example, you might seek out a data lake. For massive volumes of structured data, perhaps a purpose-built data warehouse can help. If you need to blend high volumes of structured and unstructured data that travels at batch and streaming cadences, might I suggest a data hub instead?

Starting with a business problem also includes enlisting the aid of any other potential stakeholders in the organization. If your company is like most, then your team isn’t the only one struggling with data volumes. If you’re marketing, consider bringing sales and service to the table for a discussion about their data travails. Engaging the IT team is imperative, if for no other reason than they can tell you whether that flashy solution will play nice with the rest of your systems.

If you can gain alignment with the various stakeholders who stand to benefit from a big data solution, then you also could have a stronger case to seek out a new technology in the first place. Multiple departments willing to share a joint solution tends to be an easier sell than marketing or sales asking for a new department-specific technology.

By defining your business requirements upfront – i.e., what the actual problem you’re trying to solve is – you may also find that the issue is less one of technology and more one of process. In that case, you could end up finding efficiencies that otherwise would have gone unrealized because of a too-heavy focus on deploying a new and shiny technology. If it is a problem in need of a technology solution, then you also can more clearly define the true drop-dead requirements of the solution.

You shouldn’t be led astray by vendors who claim that their solution will solve all your problems. The truth is that if you believe the hype, you’re more likely than not going to deploy a new technology without fully understanding the problem you need to solve. That leads nowhere except to wasted time and money, which is something no one wants.

Sign up for the free insideBIGDATA newsletter.