The Truth Behind Why Most ML Projects Still Fail and What to Do About It

In this special guest feature, Gideon Mendels, CEO and co-founder of Comet ML, dives into why so many ML projects are failing and what ML practitioners and leaders can do to course correct, protect their investments and ensure success. Gideon is a computer scientist, ML researcher and entrepreneur at his core. Before Comet, he co-founded GroupWize where they trained and deployed NLP models processing billions of chats. His journey with NLP and Speech Recognition models began at Columbia University and Google where he worked on hate speech and deception detection.

While it is true that ML adoption has significantly increased, there is still the hard reality that most ML projects fail to see the light of day. Maybe not to the level Gartner predicted back in 2018, but the numbers are not in ML’s favor. Depending on who you ask, failure rates range from 50% to above Gartner’s 85%.

If we take an honest look at what’s actually happening, it is not an ML maturity issue, as many would suggest. The technology should not be considered a scapegoat. Rather, several different factors are at play. Factors, which once identified, can be easily addressed so that organizations can reap the exceptional benefits of ML once and for all. 

Why ML Projects are Failing 

There are 3 high level reasons why ML Projects are failing

1. Lack of Clear Business Goals and Objectives
2. Lack of Collaboration with Business Leaders and SMEs 
3. Not investing in the right tools and processes

Lack of clear business goals and objectives

Often, there is a lack of understanding of the business problem that teams are attempting to solve and how success will be measured. Without clear objectives it is difficult for ML teams to determine the right approach, algorithms and metrics to use. For example, if the objective is to create a model that can accurately predict customer churn, the ML team needs to understand how customer churn is defined for that problem and what factors contribute to it. This would require insights from domain or subject matter experts. The ML team will then have to determine what metrics will be used to measure the model’s performance. These metrics should be defined early in the project and used to guide the team’s effort throughout the project. And even when understanding exists, it is not uncommon to continue to see a misalignment of business needs with the capabilities of the ML model being developed. Is the model designed to reveal the answers teams seek or are teams trying to force a square peg into a round hole? 

Lack of collaboration with business leaders and SMEs 

Another very common issue is that ML teams may not have sufficient collaboration with key business leaders, domain experts, and stakeholders who could add necessary insights. Without open communication, there is a lack of understanding on both sides that results in models failing to meet expectations. This is true not only in the early development process but also once the model makes it into production and the business is actively using it

Not investing in the right tools and processes

As more companies invest in ML, there is a growing need to move from proof-of-concept (POC) to production. This requires investing in the right tools and processes to ensure success. ML projects can be hindered if the wrong tools are selected or if processes are not efficient. For example, teams may have initially relied on manual experiment tracking or ad-hoc model deployment, which may work for a small number of models in a research-focused setting. However, when it comes to building production-ready models, these ad-hoc methods may not meet business expectations in terms of collaboration, reproducibility, explainability, and governance. The same can be said for companies that have built their own in-house MLOps tools. As they scale their efforts, they may find that these tools are not able to meet their scaling requirements especially if they are built on top of open source solutions.

The Importance of Understanding

Using ML to make better decisions that positively impact your bottom line, whether it be through more efficient ad targeting or improved customer retention, is a goal for many companies. However, assembling a team of top data scientists and tasking them to find this information alone may not be enough.

For ML projects to be successful, it’s crucial for the team to not only understand what information is needed, but also the reasoning behind why it’s needed. Without a deep understanding of how different business lines function and what information is important to different teams, ML models may not provide the insights and results that are needed.

To overcome this, team leads and decision-makers must work closely with ML teams throughout the entire process, from identifying the correct questions to ask to ensuring that the data team understands the reasoning behind what the business wants to know. This ongoing collaboration is essential for success and cannot be circumvented by talent alone.

Standardize on the Best Tools and Practices for Your Needs

Machine learning projects are complex and multi-faceted, as they are not just reliant on code but also on data and models. There are several steps that ML teams can take to standardize on best tools and practices to make their ML projects successful:

1. Research and evaluate different ML tools and frameworks: There are many ML tools and frameworks available, and it’s important to research and evaluate them to determine which ones are best suited for the specific project and use case. This includes evaluating the ease of use, scalability, and performance of the tools.
2. Establish best practices: Once the team has selected the appropriate tools, they should establish best practices for using them. This includes guidelines for data preprocessing, model training, and evaluation. This helps to ensure consistency and reproducibility across projects.
3. Implement a version control system: To ensure that changes to the code, data, and models can be tracked and easily undone if necessary, it’s important to implement a version control system such as Git. This also helps to facilitate collaboration among team members.
4. Automate the ML pipeline: To increase efficiency and reduce the risk of errors, it’s important to automate as much of the ML pipeline as possible. This includes automating tasks such as data preprocessing, model training, and deployment.
5. Invest in MLOps tools: MLOps tools can help to streamline the process of experimentation, deploying, monitoring, and maintaining ML models both during training and production. These tools can automate the process of tracking experiments, managing models, and deploying updates.
6. Encourage open communication and collaboration: To ensure that the team is aligned and working towards the same goals, it’s important to encourage open communication and collaboration. This includes regular meetings and check-ins to share progress, discuss challenges, and get feedback.
7. Continuously evaluate and improve: Finally, it’s important to continuously evaluate the tools and processes used by the team and make improvements as necessary. This includes gathering feedback from team members and using it to identify areas for improvement.

Carry On

In today’s world, it is very easy to brush off ML as simply “not being there yet” but to get “there”, we have to acknowledge that it’s not just about the technology and its capabilities. Teams don’t always do everything right– and that’s ok. It’s part of learning. The bottom line is, the technology works, it’s valid, and some businesses are growing significantly as a direct result of project outcomes. 

To get there, organizations must follow five clear steps:

1. Ensure that ML teams have a deep understanding of the business problem they are attempting to solve, including the specific factors that should be considered and how they will play a role in the outcome. Alternatively assign someone from the business side to the ML project as a domain expert.
2. Align business needs with ML capabilities by clearly defining the goals and objectives of the project and selecting the appropriate tools and processes to support those goals. What does success looks like to the business and how can we find the right “offline” metric to optimize for that
3. Foster communication and collaboration between ML teams and business leaders or other stakeholders to ensure that all parties have a clear understanding of the problem being solved and the needs of the organization.
4. Implement efficient tools and processes, such as automated experiment tracking and standardized model deployment, to streamline the ML development process and increase the chances of success.
5. Engage in perpetual learning. ML tools are developing quite rapidly. Build in time to experiment so that your team stays fresh and forward-thinking. And look for integrations between tools that will make life easier and more productive.

By following these steps, you can dramatically increase the likelihood of ML success. By following these steps, teams can make magic happen.

Sign up for the free insideBIGDATA newsletter.

Join us on Twitter: https://twitter.com/InsideBigData1

Join us on LinkedIn: https://www.linkedin.com/company/insidebigdata/

Join us on Facebook: https://www.facebook.com/insideBIGDATANOW