We all know that machine learning (ML) and AI gets the analytics and data science community excited. Every self-respecting forecasting department is developing ML algorithms to predict who will click, buy, lie, or die (to borrow the title of Eric Siegel’s seminal work on the subject). All analytics conferences and publications are filled with AI buzz words.

But when it comes to real-life implementation, the majority of demand forecasters are somewhat cautious about implementing machine learning. Why is that? Isn’t machine learning all about predicting, which is literally a Forecasters job? Let’s explore the opportunities and pitfalls of applying machine learning in forecasting.

Demand Forecasters & Data Scientists Define ‘Prediction’ Differently

There is a subtle difference in the way forecasting and ML define ‘prediction’. When Forecasters say ‘prediction’ we mean a prediction about the future. Traditional forecasting prediction methods include Time Series modelling, algebraic equations and qualitative judgement calls. As a result traditional forecasting is somewhat manual and time consuming, and may be swayed by human judgement. However, the outputs are easily interpreted and it is an agile process; the Forecaster knows where the numbers are coming from and may easily make corrections as needed. Further, traditional forecasting may be done with limited data.

Machine learning or statistical model ‘prediction’ refers to predicting the past. This sounds a bit counterintuitive, but the idea is to compare the model ‘prediction’ with reality and measure the difference or error. These errors are used to finetune the model to predict the future. Consequently, model predictions are heavily driven by past performance and are almost impossible to finetune. Also, the interpretability of models is very limited. Another factor to consider is that by design ML requires a lot of data. On the upside, machine learning is quick and automated as well as objective, being free from human judgement.

Machine Learning was Built for the Digital World; Forecasters Work in the Real World

Machine learning and AI algorithms were created for a digital world with almost unlimited data on customer clicks, purchases and browsing data. As we know, these algorithms do an excellent job in luring us to make repeat purchases, buy complimentary items, and sign up for loyalty programs. The sunk cost of prediction error (lost sales) is relatively low. In addition, every error is an opportunity for the machine learning algorithm to improve itself.

The real world marketplace is quite different from the digital, marketplace, however. The data here might be limited to cash register sales, loyalty program data, or shipment data. The sunk cost of prediction error can be quite high as restaurants and retailers make procurements in bulk. Also, predictions cannot improve themselves as there is no automatic feedback loop. For these reasons, many brick-and-mortar retailers and their suppliers still rely on traditional forecasting methods. This does not mean that Machine Learning cannot offer opportunities in improving forecasting but there are a few considerations that need to be addressed before venturing into machine learning.

Machine Learning Requires Much More Data Than Time Series 

Any machine learning algorithm requires a lot of data. By a lot of data, I do not mean dates or variables. Machine learning models run on defined observation levels—this can be customer, store etc. You need at least a thousand of those (if not thousands) for machine learning to work. If the sample is limited to only 10 stores, it is probably better to refrain from machine learning and use Time Series techniques instead. Another factor to consider is the cost of maintaining the data. Is it readily available or does it need to be inputted manually? Does the data need to be engineered? Would that be a one-time effort or an ongoing process requiring human and computing resources? What would be the cost of storing data over the years?

Machine Learning is far Less Interpretable than Time Series 

By design machine learning is a black box. For example, predictions may be generated by a vote of thousands of decision trees. You can use colorful histograms to depict the weight of each factor in the model. These charts look very smart on presentation slides but are very far from intuitive. If the cost of a wrong prediction is millions of dollars, companies might be more comfortable with Time Series and arithmetic they can understand rather than a slick black box algorithm. This especially applies to new products with no sales data or limited test data.

There are a few workarounds for understanding machine learning. Playing with parameters might be a good indicator of the robustness of results. If one slight change to model inputs or specifications results in significant changes to predictions, this might be a red flag.

At the end of the day model trustworthiness may be proved by testing on new data. We don’t necessarily need to understand the ins and outs of algorithms if we are confident in the end result. Robustness of this argument may depend on your audience. Typically, analytics professionals are comfortable using machine learning predictions as long as they are tested. Supply chain leaders might be more cautious in making business decisions based on black box. A good sanity check is to run traditional forecasting methods in parallel to machine learning. If there is feasible difference between the results there might be either an issue with the model or an important consideration was left out when creating a traditional forecast.

The Cost Benefit of Machine Learning is not Always Clear

It goes without saying that when machine learning is set up right it is wonderfully efficient. All one needs to do is provide inputs and press the button. The ‘setting up right’ piece might be relatively straightforward or extremely difficult depending on prediction goal and data available. Repeat products with abundant history may be easily predicted using even out-of-the box ML packages such as SAS or Azure as long as the data is readily available. New product predictions may require intricate proxy algorithms to solve for limited data. This may require development of ML algorithms from scratch. In addition, there may also be a need to engineer data from different sources to feed the algorithm. This might require significant investment to either hire contractors, expand analytics team or put pressure on existing resources. Before ramping up a data science crew, companies would be well advised to consider how often the algorithm will be used, the efficiency gains, and the computing resources required for the project.

Impacts on Overall Business Planning 

Forecasting is the cornerstone of business planning. Any changes to the forecasting process may have an impact on other areas of the business such as Finance and Supply Chain. Typically, traditional forecasting methods rely on a top-down approach. A forecast is created in aggregate and then broken down by store/time period, etc. These breakdowns may be later used for financial targets or demand planning at store level. By design, ML Forecast utilizes a bottom-up approach. A prediction is created at store/time period level and later aggregated. When switching from traditional forecasting to ML, companies must ensure smooth transition at all stages of business planning. If not done right, this transition may result in discrepancies between the ML prediction vs the financial targets and supply plans.

To summarize, ML is a great instrument to streamline forecasting. As with any tool, it has its applications, benefits, cost, and risks. When utilizing ML for forecasting, companies should consider their data, business need, decision making culture, and planning workflow. A great place to start might be trying out ML on your data using online, off-the-shelf solutions such as Azure and SAS. Most of these solutions have step-by-step training videos that will help fit an ML algorithm to your data. Experimenting with these solutions may help decide whether ML is a good tool for your company’s forecasting, and whether an off-the-shelf solution is sufficient or there is a need for in-house development. Even if it turns out that for whatever reason ML is not a good fit for your company, there is no investment lost and some analytical knowledge will gained.

This article first appeared in the summer 2023 issue of the Journal of Business Forecasting. To access the Journal, become an IBF member and get it delivered to your door every quarter, along with a host of memberships benefits including discounted conferences and training, exclusive workshops, and access to the entire IBF knowledge library. 

The world of forecasting today is filled with machine-related buzzwords like AI, predictive analytics and machine learning. Will all of forecasting be done by machines a few years from now? Do humans still have any competitive advantage over software? In the realm of new product forecasting – the human wins.

Human trumps machine forecasting in the following areas:

• Measuring promo impact
• Gauging life cycle trends
• Disentangling overlapping marketing tactics
• Incorporating qualitative analysis

Machines have great potential to improve forecasting processes. Automating forecasts for existing products that have a lot of data can free human time to focus on new product forecasting. However, engaging software in forecasting all products, both old and new, can pose serious risks for forecast accuracy.

How Much Value Does AI Really Add?

Hollywood has been questioning whether the machines will take over the world for almost half a century so it must be true. How will it start? Machines flipping burgers, executing court rulings or maybe doing demand planning? Demand planners are naturally intrigued about the latter. Industry trend-setters prefer to talk about AI fueled by machine learning algorithms over Excel books fueled by ridiculous amounts of coffee. Buzzwords sell, but do they really add value?

To answer that question we need to ask what AI and machine learning are. In the context of forecasting, these disciplines are essentially a series of algorithms that create baseline models and measure promotional impacts. The ‘machine learning’ component is a fancy term for the trivial process of feeding the algorithm with more data. These tools are very useful for forecasting products with lots of history and homogeneous promotions. These types of products are usually the easiest to forecast. Forecast error for products with lots of history is typically low, regardless of the level of automation. The tools save a lot of time and brain cells from doing tedious repetitive tasks, but they hardly improve forecast accuracy.

Can Algorithms Work With Little To No Sales Data?

But what about new products that have no or little sales data? Ironically these are also the ones that are subject to all kinds of marketing extremities. Can any machine algorithm forecast those? The short answer is no. And this is why:

1. Measuring promo impact as % vs Baseline may be misleading

How does software measure promo impact? Most of the time it would just compare actual sales to the baseline. % difference would be the promo impact that the machine will use to forecast future promotions. However, reality might be a bit more complex than that. Imagine you are forecasting a winter ice cream promotion based on a previous summer promotion. Your summer baseline was 100 units and you sold 200 units as the result of the promotion. The software calculates a 100% lift. Impressive right? Let’s apply that to your winter baseline of 15 units. Not that impressive anymore. A human can create a promo lift that might not be calculated by a particular formula but nevertheless is reasonable.

2. Product life cycle trends can throw machines off

Most products have a sustain period after the launch. This is the period of the product life cycle right after the initial launch when sales gradually decline until they level off and start exhibiting seasonality. Software is not very efficient at forecasting this trend. It usually either underestimates future sales by applying the negative trend indefinitely or overestimates by applying seasonality to sales that are still elevated after the launch. A human can apply common sense to determine when the sustain period ends and normal seasonality starts.

3. Disentangling components of promo impact via algorithms may result in lower accuracy

Every new product promotion is unique with its own mix of marketing support, pricing and product strategy. Imagine you are forecasting a promotion based on two historical promotions. One historical promotion had a 10% lift and the other a 100% lift. An algorithm would use the average 60% lift for the forecast. In most cases this would be inaccurate. A human can make a judgement call on what particular factor differentiated the two historical promotions and choose the most similar one as a proxy for the forecast. I know what you’re going to say, “But we can teach the machine to recognize different kinds of promotions!” This brings me to my next point…

4. Teaching machines may be more work than doing it yourself

When a human analyses historical promotions he/she can look at different files, process different formats and even ask for other human opinions. A machine, however, demands to be spoon-fed data in a consistent format from one source. Creating a variable for every marketing strategy for every historical promotion can be very time-consuming.

5. Are you working on a forecast or on a model?

A statistician will say that all of the above issues may be resolved within a model by overriding product life cycle trend, flagging different kinds of promotions, using proxy seasonality, etc. This is true. The caveat is that these processes require a lot of human hours. Human hours dedicated to perfecting the machine, rather than perfecting the forecast. Anyone who has worked with models knows that statistical significance does not always go hand in hand with common sense. Overrides and interventions increase the risk of over-specification of the model. This means that model results stop making sense because there is not enough data to support all the additional variables.

6. Humans can think outside the box while machines are the box

Forecasting new products that have no data is solely reliable on proxy selection. Imagine you are forecasting a new LTO flavor of ice cream – dark chocolate. You have history on two LTOs – coffee and caramel chocolate ice cream. Which flavor will you choose as the proxy? A human will consider a plethora of factors. Which of the 2 historical flavors was more similar in taste? Is the bitterness or chocolatey-ness driving the sales? Is there any consumer research on the two flavors? Which of the promotions had similar marketing support? Was the creative more appealing for one of the historical flavours? All these questions belong to the qualitative, not quantitative realm. Hence, the software will not be helpful.

So what is the right balance between human and machine in forecasting? The answer is obvious – leave the quantitative tasks to the machines and qualitative to humans. The problem is that there is a fine line between the two in the realm of forecasting. The best term for a forecast in my mind is ‘a scientific guess’. We don’t want to leave the guessing part to the machines as it relies on intuition – an inherently human feature. It is worthwhile investing time in the exercise of classifying forecasts into ones that are generic enough to be done by machines and the ones that require a human touch. Machine and human can even efficiently cooperate within one forecast. For line extensions, software can predict the baseline level and even the promotional impact for the total line. Human can decide what proxy to use for the new flavor.

New product Forecasting Software Not Worth The Price

The good news is that software can effectively and efficiently forecast products with lots of history and homogeneous promotions. And they’re not expensive and are very easy to operate. There are even add-ons in Excel that can create baselines for hundreds of SKUs at different levels – provided you have enough data. It is the software that claims to forecast new products that can get pricey while providing dubious value add.

Demand planners needn’t fear being replaced by machines – only humans can provide valuable qualitative judgement.

Reconciling Quantitative & Qualitative Is Key In new Product Forecasting

Automation can release human time that can be spent on increasing forecast accuracy on the products that are the most difficult to forecast. There is a lot of room for improvement in the realm of forecasting. Average error for new food is around 50%. This equates to lost profits and excess inventory. One third of global food production is wasted, a problem that software alone will not resolve. Perfecting qualitative methods of forecasting is as important as quantitative, and even more important, is developing methods for reconciling the two. There are numerous consumer research specialists in the retail industry selling their qualitative expertise on new product appeal. Developing a system that converts such consumer research into real numbers that can be used for forecasting can significantly improve forecast error.