In the age of digital transformation, data is the new currency of business. Used in modern analytics and to feed ML/AI models, it drives the planning and decision-making that keeps an organization competitive.
But as data sources have proliferated and the cloud has accelerated, the data feeding analytics has become far more complicated and dynamic – even chaotic. To tame this data chaos, organizations are turning to operationalization in the form of XOps, one of Gartner’s Top 10 Data and Analytics Trends of 2021.
What is XOps?
Born out of the principles of DevOps, XOps is the operationalization of other disciplines, starting with DataOps and moving on to MLOps, ModelOps, PlatformOps and so on. At its core, XOps refers to the strategy of building out agile solutions that easily adapt as data landscapes evolve over time. ‘Reliability, reusability and repeatability through process and automation’ is the constant guiding mantra. And ultimately, as Gartner says, “XOps will enable organizations to operationalize data and analytics to drive business value” [1] — through the productionalization of AI in particular.
Why XOps Now?
In short, it all comes down to digital transformation, AI, and – of course – data. 72% of data and analytics leaders are spearheading or heavily involved in their organization’s digital transformation efforts. A recent 451 Research survey found that 95% of enterprises consider AI to be important to their organization’s digital transformation efforts. [2]
But as AI and ML initiatives mature, enterprises are facing significant scalability and operational challenges, leaving many at a loss about how to systematically productionalize their ML pipelines. IBM found just 21% of over 5000 companies surveyed had deployed AI, Gartner reports that just 53% of AI PoCs ever scale to production, and up to 70% of companies report no value from AI investments.
By relying on XOps, Gartner says companies will be able to “facilitate the performance, scalability, interpretability and reliability of AI models while delivering the full value of AI investments.”
DataOps: The Foundation of XOps
The more data an algorithm must work with, the more accurate the results. But AI value, ML value and analytics value are meaningful only if the data it operates on is valid. Noise in the data disrupts learning and leads to unreliable outcomes. Traditional data integration methods invested heavily in data quality procedures to ensure that only the cleanest data made it into the analysis — but those procedures were brittle. And the scale and complexity of today’s dynamic data architectures make this approach very risky. So as companies operationalize ML, they increasingly depend on DataOps, where resiliency and checks are built into the operational pipelines themselves.
Adding DataOps is the force multiplier for the effectiveness of your machine learning and MLOps. And it’s the same with all the Ops disciplines because they all need data pipelines. Beyond building these pipelines, they must operate continuously. Every Ops discipline needs continuous data and delivering that continuous data requires DataOps. The three key principles that allow the continuous delivery of data are continuous design, continuous operations and continuous data observability.
The People Behind XOps
As XOps practices continue to mature, data engineers have an opportunity to step into the spotlight. According to the Dice 2020 Tech Job Report, data engineering was the fastest growing job of 2019, growing by 50% year-over-year. Fast forward to today, and data engineering opportunities are continuing to outpace data scientist roles. Data engineering is at the frontier of the data revolution.
Data engineers play a critical role in the control of data as it relates to DataOps and XOps as a whole. Their job is to enable fast, confident decision making to drive positive business outcomes. Data engineers are in the unique position to enable data scientists and business partners with real-time data.
Furthermore, the development of the XOps disciplines is fundamentally changing the roles of data engineers within the organization. Ops channels, specifically relating to data infrastructure, are living, breathing platforms that continue to change, grow, and evolve. Because of this automated environment, data engineers are now shifting their focus to creating stable data environments while the rest of the enterprise becomes increasingly data literate to support growing Ops channels. In short, data engineers are leading the XOps march forward.
Key Takeaways
XOps has become a growing data analytics trend in 2021 for good reason. Evolved from the DevOps movement to better support and enable AI and ML automation workflows, XOps enables organizations to operationalize data and analytics to drive greater business value and establish a solid yet flexible foundation for future technology development. And the people behind the revolution? Data engineers. The data ecosystem of the enterprise is quickly changing, and it will be critical to keep a finger on the pulse of the next iterations of widespread digital transformation.
[1] https://www.gartner.com/smarterwithgartner/gartner-top-10-data-and-analytics-trends-for-2021
[2] Voice of the Enterprise: AI & Machine Learning, Infrastructure 2021 – Advisory Report; 451 Research, Oct 8, 2021, Nick Patience, Rachel Dunning
About the Author
Arvind Prabhakar is CTO of StreamSets. Arvind is a seasoned technology leader, who has worked on data integration challenges for over 10 years. Before co-founding StreamSets, Arvind was an early employee of Cloudera, and led teams working on integration technologies such as Flume and Sqoop. A member of the Apache Software Foundation, he is heavily involved in the open-source community as the PMC Chair for Apache Flume, the first PMC Chair of Apache Sqoop, and member of various other Apache projects. Prior to Cloudera, Arvind was a software architect at Informatica, where he was responsible for architecting, designing, and implementing several core systems.
Join us on Twitter: @InsideBigData1 – https://twitter.com/InsideBigData1
Sign up for the free insideBIGDATA newsletter.
Speak Your Mind