Big Data Project Failure Pain Points and their Solution

Print Friendly, PDF & Email

Sponsored Post

Big data projects don’t typically fail for a single reason, and certainly not for technology alone. A combination of factors serve to derail big data deployments. Problems and failures occur due to factors including business strategy, people, culture, inattention to analytics details or the nuances of implemented tools, all intensified by the rapid advancement of digital transformation.

Common Pain Points

A critical success factor to realizing big data’s benefits and avoiding failure is how you operationalize big data implementations. Some considerations are:

  • Predicting IT infrastructure requirements – One pain point is that exponential data growth poses a challenge to business infrastructure. Businesses will need to know how much capacity it has, how much capacity it needs, and most importantly how it can grow the infrastructure in line with the business needs. The right tools also must be used to optimize an organization’s existing infrastructure. Enterprise class capacity optimization and capacity visualization are two values that the right IT operations management tool can bring. Predictive modelling enables an enterprise to forecast future usage and predict infrastructure growth.
  • Achieving appropriate scale – Another pain point involves getting data sets from multiple sources (such as ERP, CRM, e-commerce, social media, sensor networks, etc.) into a big data platform manually would be overly time and labor-intensive, and would involve integration of multiple tools and technologies. The right tool enables data to flow in seamlessly, essential to repeat the success of a proof-of-concept big data use case across additional use cases. However, a tool requiring a high degree of custom development can be restrictive in achieving appropriate scale and must be able to integrate seamlessly into a company’s existing technology infrastructure.
  • Ensuring “right time” data – Enterprises need to garner the correct insights to impact the bottom line from increasingly large big data repositories. To successfully accelerate insights, the big data technology environment must interface with other enterprise applications and data sources. In addition to managing the data workflows from one end of the network to the other, it’s also necessary to manage the schedules for those workflows to ensure analytics teams can view the data on a timely basis. Lack of “right timing” data can be a significant pain point.
  • Securing data – If enterprises show pause in investing in big data, it’s often due to security concerns. No company wants to fall victim to the next data breach. You can only secure what you know about. In the case of big data, the enterprises that get security right have holistic visibility in terms of how the big data infrastructure is connected to enterprise applications. This is vital to overcoming an important pain point by taking measures such as managing access to sensitive data that’s being stored or is on the move.

Elements of a data-driven culture

In addition to the operational considerations, a big reason why big data projects fail is – not quantity of data, and not the technology – it’s the people. Research from PricewaterhouseCoopers cited culture — or rather, the wrong culture — as a biggest roadblock holding enterprises back from exploiting data and succeeding in big data implementations. The study found that three out of four enterprises are extracting little to no advantage of any kind out of their data. There seems to be a problem with capitalizing on the value of enterprise data assets.

Unless you have a culture that’s data oriented and you build the results of your insights into business processes, a big data platform is not really going to take hold. But how does a company move toward a data-centric culture, which Gartner defines as businesses that use data “to organize activities, make decisions and resolve conflict”?

In a research note titled, “How to establish a data-driven culture in the digital workplace,” Gartner analysts Alan Duncan and Frank Buytendijk offer a short checklist on how enterprise thought leaders can build a data-driven culture that will in turn help lead toward the path of big data success:

  • Lead by example. Make a conscious effort to communicate to employees how they’re using data to make decisions. “In meetings, in presentations, in all daily interactions, executives need to show they are looking for the right data to base decisions on,” the report states.
  • Hire data-driven people. Send a signal that data is playing a prominent role by hiring candidates who think in data terms. One of the most common big data failures involves setting overly optimistic expectations when a skilled team is not in place to deliver. A key success factor for implementing big data is the enterprise’s ability to build, grow and sustain a multi-disciplinary team with the expertise needed to address identified business problems.
  • Create more transparency. Make it easier to access data and make information governance policies more transparent.
  • Conduct data-driven performance reviews. In a data-driven culture, data should be used in every aspect of the business — from hiring to goal setting, according to Gartner.

To start building a data-driven culture, it’s important to find use cases that are compelling to the business. Particularly for companies that are new to big data – identifying and prioritizing the right projects and use cases to show value builds credibility within the organization. That’s why experts suggest introducing the business to big data by solving a small, discreet problem. You want to find one example that can be a standard. There, you have a very concrete problem, and you’re going to have an applied solution for that problem. As the business sees quick wins, as trust in the methods grows — as the culture becomes more data driven, IT’s strategy will have to change.

Becoming data-driven can involve navigating uncharted territory to make the significant technology and cultural changes required. But a strategic approach to managing, sustaining and growing a big data environment, involving the right technologies, means organizations can overcome operational roadblocks to focus on insights that help their businesses thrive in the digital era.

About the Author

This article was written by the staff of Clarity Insights, a big data and data science consultancy. It is the largest 100% onshore big data and data science consultancy in the United States. Clarity Insights helps companies unleash their insights by creating data strategy, building data platforms, and finding actionable insights that build processes & culture.


Speak Your Mind