Why Technology Should Take a Backseat for Data Projects to Succeed

Print Friendly, PDF & Email

In this special guest feature, Andrew Stevenson, CTO and Co-founder at Lenses.io, believes that without tooling, governance and visibility around open source technologies, applying them successfully at an enterprise level is very hard. In 2017 Andrew co-founded Lenses.io to create a platform that makes data accessible to everyone. His vision is for anyone to be able to drive business with data rather than a few people with deep technical experience. With more than 20 years of experience with real time data, Andrew has led and architected big data projects for banking, retail and energy sector companies and is a leading Open Source contributor.

Data driven is a nice buzzword. We run around our organizations shouting that we need to be data driven and try to wade through all our data to find the nuggets of gold we’ve been promised. We convince ourselves, as technologists, that we have big data, massive streams of data on par with Uber and we need the latest open source projects to handle this. 

Businesses have empowered engineering teams to drive data projects. At the same time I, like many in the industry, had been guilty of focusing on technology in order to further my career, worried we would fall behind the rest of the market unless we adopted the latest open source.

But having spent a lot of time on using both open source and proprietary software, I have recognized one point time and time again:  Without proven tooling, successfully applying governance and visibility around open source technologies at an enterprise level is very hard.

In the land of the blind

Gartner stated that 60% of big data projects fail, then quickly revised this up to 85%. That’s really bad but no surprise to me. Projects which start with a business idea become consumed by technologists using technology, just for the sake of technology, who lose focus of the business objective.
To stay focused on business outcomes and to increase project success, visibility, safety and security are key.

I recently bought a new car. One of the criteria I used to select my car was visibility. I want to be able to see all the dashboard information available and have it easily accessible.  I also want good safety and security.  Why should working with data be different?  In my time working with fast data I would never have put a solution together than didn’t also have these features. Yet many happily build around projects that have these massive gaps.

For data projects, visibility comes in the form of identifying and giving the right users access to explore the payload and metadata, as well as how and where that data is being used and processed with appropriate control. Visibility combined with control will make or break the success of a project. 

Another important consideration is to recognize the value of both technical and business expertise.  I have seen extremely knowledgeable business domain experts sidelined in favor of highly paid “big data experts” (me included) across many industries. They were seen as more valuable as the technologies deployed were so complex. With this approach, they forgot that without the domain expertise, the project would never be able to stay focused on the business objective.

We all stand together

Where I have seen success is when powerful open source technologies have been used while giving business users with domain knowledge the ability to self-serve their data access.   A ubiquitous language, such as SQL, makes it possible for a wider array of users to serve themselves and get visibility into the data platform and data applications. Business experts who were able to discover, explore, visualize and build using data in an accessible way advanced the organization’s goals and optimized data in ways I couldn’t because they were the domain knowledge experts.  The best role for a technologist is to be a technology partner to the business and enabler of the business goals.

Without building integrated data teams of business analysts and technologists, we will continue to see this high project failure rate — which isn’t acceptable in any other industry.  Imagine 85% of, say, construction projects being abandoned? 

Time for Data Intensity 

The surest way to find our gold nuggets in a data driven world is to make data actionable by business.  Use a data mesh architecture that is composed of commoditized best-of-breed technologies that eliminates the need to create complicated solutions from the ground up. Decentralized technology coupled with a DataOps approach, that provides governed and secure access to data, gives organizations what they need to deliver faster and be more successful with their data projects.

Sign up for the free insideBIGDATA newsletter.

Speak Your Mind



  1. In order for the data projects to succeed they must have the visibility, safety & security. Technology overshadows various data projects as people give technology the first priority. The article will be very much helpful in knowing it better. Thank you for listing it and sharing it with us.