In an era where everything can be measured, and often is, data accumulates faster than most businesses can keep up, filling vast volumes of storage and requiring costly resources to maintain. In fact, the average enterprise environment now includes more than eight data lakes.
These disparate data sources and data lakes often create more chaos and disruption than they add value. Think about the steps involved enforcing data quality in this kind of dynamic situation. Not only must IT managers profile, catalog, and store the data, but they must also adhere to a variety of regulatory requirements governing its use. That makes for a manual, slow and error-prone process.
- Data silos get in the way of sharing useful information that might prove valuable to other departments.
- Enterprises lack a culture of data and collaboration that’s critical for decision making.
- When initiatives overlap or start in parallel, organizations often end up with duplicative data and data whose provenance is unknown. And if you cannot establish data quality, it’s “garbage in, garbage out,” rendering any kind of analytics or insights essentially meaningless. That’s a big problem because the quality of the data should be table stakes. To put it another way, the enemy of what we do is the uncertainty about the provenance of the information that we have.
All this results in wasted, unmonetized data. Meanwhile, the compliance risks are high, even as businesses continue to spend crazy amounts of money to try and manage it all. Here’s where the deployment of DataOps can help organizations get back on track as they navigate past the shoals.
The Future is DataOps
Over the last few years ‘data operations,’ or, DataOps, has made steady inroads into the mainstream to the point where a lot of organizations are adopting or seriously considering agile principles for managing data.
Think about DataOps as a methodology, as opposed to a set of tools or offerings. The concept, which derives inspiration from the practices of Lean Manufacturing, Agile and DevOps, helps organizations overcome bureaucratic hurdles and complexities to deliver analytics with speed and agility, without compromising on quality or data governance.
The business advantages pay off in the form of greater innovation, faster data analytics cycle time, and revenue growth. More specifically, it translates into the following:
- A common business understanding of your data. This involves a combination of a data catalog and collaboration with business stakeholders to curate the data.
- Automating the data quality, governance, and compliance tasks to make sure that data is ready to be consumed.
- An organizational emphasis on the agile and automated management of data where people, processes, and technologies are focused on managing data geared towards business outcomes and goals while keeping governance risks low and costs under control.
- DataOps allows you to break down silos, by far, its biggest value proposition as it allows your business to run a lot faster and to deliver new products to the market at an accelerated cadence since that valuable information is no longer locked up in a single department and can be shared throughout the company.
AI, ML & Automation
Metadata is at the root of how to deliver any good DataOps approach. When creating metadata automatically at ingestion using Artificial Intelligence and Machine Learning algorithms, a business can significantly reduce the manual effort. When that’s achieved, you’ll speed the development of your data pipelines and accelerate adoption and effective analysis by your teams.
By automating that end-to-end management of that metadata, the technology can then start to deliver against the promise. To make this happen, you need data integration technologies to onboard the data as well as an effective mechanism to catalog the data and rules need to be applied to it in order to establish lineage.
What can derail such operations, is a piecemeal approach, where data is handed from one tool to another. That’s where AI and machine learning (ML) techniques can help organizations better understand the data and semantically enrich it, as required. Further, the integration of AI and ML can help identify any quality issues while enabling governance rules to be added to the data.
The Transformation
Defining outcomes and being very clear about the ultimate objectives is key to successful data transformations. It’s a matter of figuring out where the data is coming from, selecting the best methodology to store and manage it, and figuring out how to ensure that you can harness it to quickly deliver value to your end customer.
At the same time, a big part of this evolution is cultural. Indeed, the successful adoption of DataOps will hinge on things such as ending silos once and for all while encouraging more collaboration between data and IT teams, end-to-end design thinking, and treating data as a shared asset
These challenges are not trivial. But as you review your company’s data posture, I encourage you to think seriously about treating data truly as an asset that can enhance outcomes for your company. If you do, DataOps will put you on the right path.
About the Author
Radhika Krishnan, Chief Product Officer, Hitachi Vantara. As the global head of products, Radhika is responsible for the vision, strategy, delivery, and business performance for all Hitachi Vantara products and solutions across data storage, data operations and analytics, as well as Hitachi’s industry leading Lumada Internet of Things and industrial software. She has incubated, transformed, and scaled several businesses focused on software and cloud infrastructure. Before joining Hitachi Vantara, Radhika was executive vice president and general manager of software at 3D Systems and has previously held leadership roles at HP, Cisco, NetApp, Lenovo, and Nimble Storage.
Sign up for the free insideBIGDATA newsletter.
Join us on Twitter: @InsideBigData1 – https://twitter.com/InsideBigData1
Speak Your Mind