AutoML: How Do You Measure Return On Investment?

So your company has decided to invest in an Automated Machine Learning (AutoML) platform. Excellent – AutoML promises that it can help accelerate and automate much of your data science process. At first blush, the return on investment (ROI) for your technology purchase would seem simple: Measure how many data science projects your team could produce on average before your platform purchase, and then measure again afterward. If your results are anything like what our clients have seen, you will likely measure ROI in terms of time: many of our clients are finding that they can deliver data science projects 10X to as much as 32X faster than they could manually. While those numbers are high, however, there are other even more powerful means of measuring ROI that will be even more meaningful and valuable to your business. Leaders should think beyond cost savings and look at developing sustainable competitive advantages enabled by AI.

AutoML ROI Starts with Value to the Business

One of the most revealing stats recently published by VentureBeat shows that nearly 87% of data science projects never make it into production. While there are multitudes of technical reasons why these failures are common, there are also underlying business reasons. Chief among them is that there is often a disconnect between your data science team and your line of business users as to the purpose of the data science project. Creating a tighter alignment between your business units and your data science and BI teams is fundamental to drive ROI from data science effectively.

Investing in AutoML in and of itself will not remove this potential roadblock. An automated data science practice is still flying blind if the people driving the practice don’t know the business challenges they are trying to solve. In understanding the value of the project, the ability to measure ROI increases. If, for example, your team is working on applying machine learning to reduce churn rates, it will be valuable to empower the team with knowledge about the value of each customer that churns out. What is the financial impact of lowering your churn rate from 3% to 2.5%? What will the gains be on the business? Tying your data science projects to concrete financial implications will not only help motivate your teams but will also give you direct means of measuring ROI.

Measuring ROI: Lower Development Costs

One of the first, and most obvious steps, in measuring the ROI of your AutoML investment is to measure the changes in capacity for your team. However, how exactly do you achieve that? Most organizations have a relatively good measure of how long it takes (on average) to complete a data science project. For most businesses, the most complicated – and time-consuming – part of the process is known as feature engineering. During feature engineering, your team must carefully assemble fields from different tables, often sourced from multiple systems – and use statistical means to create “feature tables.” Tables of data that will be iteratively processed by machine learning algorithms to evaluate which features work best with appropriate models.

With the right data science automation platform, your team might be able to reduce its average development life-cycle from three months to as little as three days. When calculating on a “man-day” basis, a three-month project lifespan means sixty person-days worth of effort (20 working days per month x 3 months). Measuring the productivity gains achieved through automation, the team in this example has decreased development life-cycles by 20X (60 days / 3 days). To monetize this return, measure the “average cost” of your data science team in terms of salary, and you’ll quickly see how fast your investment will be paid back. Let’s assume in our example that we have 5 team members with an average annual salary of $150,000 per team member – that means a daily rate of $3,125. An AutoML platform, in this example, you would be saving seventeen days’ worth of development, a savings of $53,125 just on the first project!

Going Beyond Cost For Proper AutoML ROI

Having arrived at a basic calculation for return on investment that is based on productivity, we are ready for the next phase: Measuring ROI based on value to the business. After all, your data science team is not working in a vacuum. A bank might be leveraging machine learning to identify fraud, a retailer might be analyzing product inventory optimization, and an insurance provider might be attempting to find ways of lowering customer churn. Each of these scenarios has very high dollar-values associated with them. In the world of Software as a Service (SaaS), for example, customer churn is a critical metric.

Let’s assume your company has 3,000 customers, each paying on average $5,000 per year in software licenses. At an annual churn rate of 8%, that means a net loss of $1.2M in annual revenues EACH YEAR. Decreasing that churn rate to just 7% – nets an additional $150,000 per year in revenue. Reaching a more palatable 5% churn rate, nearly halves the annual revenue loss and nets the business an additional $450,000 in annual revenue. This example, of course, is a bit simplistic – but you get the point.

New Possibilities: Additional Revenue Streams

For many organizations, the reality is that hiring data scientists is a financial and practical impossibility. These businesses, however, typically already have BI and analytics teams in place that are responsible for creating reports and dashboards. Empowering these BI teams with tools that allow them to build predictive analytics reports and dashboards, can provide a significant benefit to the business. For example, ML can predict additional insurance products to sell for existing customers, create better efficiencies in supply chain and operations at pharmaceutical companies, improve processing times between lenders and consumers with smarter underwriting decisions and can be leveraged to predict customer churn and more efficiently manage cashflow.

Measuring the ROI of your AutoML investment must move beyond the simple view of data science productivity and must be measured against the business outcome including generating new revenue streams – incredible gains that can be had by minimizing fraud, lowering customer acquisition costs, reducing churns, optimizing marketing campaigns and the myriad of other use-cases that are possible with Machine Learning and Artificial Intelligence powered by AutoML.