Even before the pandemic, foward-thinking retailers leveraged AI and machine learning technology (ML) for demand forecasting. In the post-Covid world, however, those ML models have failed to provide accurate predictions because they don’t know that the data they use is now obsolete due to changing demand patterns. How can we upgrade them to the new reality?

There are six possible ways to get a more accurate forecast:

1 – Gathering data on new market behavior: As dynamics within the new market stabilize, use that data set to create a new model for forecasting demand.

2- Use a feature engineering approach: Track external data sources like price indices, market states, latest news developments, exchange rates, and related financial/economic factors. Using these, models can generate more accurate predictive outputs.

3 – Factor in up-to-date POS data: Analyzing recent POS data can allow us to observe and react to real-time shifts in patterns of demand, improving forecast reliability. Depending on a given product’s classification, the appropriate range for a POS data set might be between a month to two months.

4 – Use the transfer learning approach: If we possess any data sets relating to historical pandemics or behavior based on similar principles, we’re able to use that data within the context of this present-day pandemic.

5 – Utilize a model for information cascades: Merge the cascade modeling with current POS data sets to create a demand forecasting model that is able to recognize aggregated consumer behavior patterns and predict herd patterns for future sales.

6 – Leverage NLP (natural language processing) technology: NLP analyzes actual consumer comments and posts from an array of social media sources, from media platforms to popular social media sites. NLP can use sentiment analysis algorithms to collect and analyze conversations and discussions from real customers. This gives an unfiltered look at consumers’ behavioral patterns, preferences and attitudes.

If you are looking for a way to improve your current ML models and thinking of building a demand forecasting feature from scratch, this will help you to choose the best approach depending on your business type.

A data scientist generally works with historical data, and it’s impossible to predict such drastic changes as a worldwide pandemic. But as a general rule, you should prioritize flexibility in retraining your models, add more external factors as predictors, and account for a short-term perspective as long-term models become less relevant.

We tackled this problem with pre-Covid models for a restaurant business. Here is an example from our dashboard:

Forecasting dashboard showing normal revenue before the pandemic.

Forecasting dashboard revealing revenue post-lockdown.

In this case, we rebuild our models from scratch, losing a degree of accuracy. At this point, the historical data are not relevant anymore and we wait for new statistics and patterns inside the data.

Not every demand forecasting article on the internet fits the particular needs of your business and industry. Effective approaches vary dramatically depending on business types – here are some of the distinctions and varying ways to address demand forecasting:

Small vs Large Businesses

Small and large businesses should be approaching forecasting in completely different manners. First of all, the acceptance criteria for huge businesses is significantly smoother than for small businesses. We can have a higher error for prediction in quantity, as high sales volume allows for greater tolerance for error. When it comes to historical data, large businesses have a higher volume of collected data, making it easier to identify patterns in customer behavior. With small businesses, it’s often necessary to test your hypothesis to prove the correlations between sales volume and predictors.

Demand for a product at a large business, showing clear demand patterns.

Demand for a small business showing erratic sales patterns.

Online vs Offline

Online sales allow for a greater range of predictors and external factors in your modeling. It’s not necessary to have a POS (point of sale) system, as you’re able to collect all relevant data from the website. You know more information about customers and collect their historical purchases. With different predictors, you can apply a wider range of machine learning models: Gradient Boosting, Random Forest, SVR, Multiple Regression, KNN, etc. With offline sales, you are often limited to historical sales only. The best approach here is to use Time Series Analysis.

The USA vs Europe

Regional differences play a huge factor in predictive modeling, as varying locations will have different behaviors both in specific sales and general cultural factors. It’s worth noting that certain regions’ consumers are influenced to different degrees by marketing campaigns. Additionally, holidays vary by region, and you’ll need to decide whether to add this feature to the model or not. Take into account different legal constraints (limit for product amount) etc.

Perishable vs Non-Perishable Products

Finally, it’s crucial to factor product type within your demand modeling. With perishable products, you need to set up the right metrics to penalize the model when the prediction is much higher than the real value, as the consequences for excess inventory are significant. You need to be careful with data preparation and work with outlier detection because this prediction should be extremely sensitive to any changes.

Conclusion

The pandemic has made us all hyper-aware of the limitations and constraints of forecasting models, but the best practices in forecasting are the same whether we’re in a pandemic or not. You should always be striving to fit your particular business model and business needs to your forecasting tools, to attract new customers, increase your revenue, and expand your market share.

To find about more about practical applications of machine learning models, pick up a copy of Eric Wilson’s new book, Predictive Analytics For Business Forecasting & Planning. Written in easy-to-understand language, it breaks down how machine learning and predictive analytics can be applied in your organization to improve forecast accuracy and gain unprecedented insight. Get your copy. 

“Change is the only constant in life”, said the Greek philosopher, Heraclitus. Uncertainty is a derivative of change and has been a constant ingredient of the demand planning process. Thus, uncertainty in business is not new but the pace at which uncertainty is building is unprecedented.

Uncertainty in business is increasing due to the increase in options available to consumers in terms of new products and different channels, and occasionally worsens dramatically when events like trade regulation overhauls, economic recessions, social unrest, and pandemics like we are experiencing now occur. So, whether by choice or by necessity, our product and channel mixes can become disrupted presenting major challenges for Demand Planners.

Understanding Uncertainty In Your Product Mix

The good news is that there are ways to reinsert certainty back into business planning. First we must  comprehend the uncertainty we’re dealing with and draw a perimeter around it. This can be achieved in the following ways:

1. Segment Your Products

First, understand the scope of uncertainty and drive segmentation. The Covid-19 pandemic has been the biggest driver of uncertainty in recent times. However, it has not had a uniform impact on all products and services across organizations. On one side, where products like gym equipment and frozen foods have seen sky-rocketing demand, the hospitality and tourism sectors have drowned. There have been products and categories like fresh foods, home and personal care, consumer electronics that have stabilized after an initial knee-jerk reaction.

The first thing Demand Planners should perform is segmentation of products, services and customers based on the value they contribute, i.e. revenue, profitability, stability of the segments, and by value they seek e.g., quality, innovation, cost, and agility. Segmentation will allow planners to improve focus, reduce noise from the demand signals, and drive technical improvements to predictions and drive a collaborative response in unpredictable segments.

2. Understand the Assumptions Behind Your Demand Signals

Second, drive clarity on the assumptions associated with the demand signals. Demand plans and actual sales are both outcomes of multiple factors, assumptions, and decisions. In times of uncertainty, it becomes doubly important for planners to tag the changes in their demand plan with factors like internal and external events; product, customer and supply chain related assumptions; and, strategic and operational decisions made by the organization.

Clarity on these factors allow advanced technology to apply AI/ML algorithms to generate improved forecast accuracy and allow planners to identify the impact of various factors on the demand plan.

3. Assess the Agility Of The Supply Chain

And third, assess the agility of the supply chain. While the uncertainty on consumer demand directly and adversely impacts demand planning performance, the uncertainty on the supply side – plants, suppliers, warehouses, transportation services – directly impacts the responsiveness of supply chains to react to changing demand. Like the demand side, there are segments in supply chain as well, which operate at different levels of upside and downside adaptability.

Some of your plants, production lines, and suppliers may be suitable for a continuous manufacturing operation while others might be more flexible and accommodate different products with minimal changeover cost. It is essential for Demand Planners to understand these segments in the supply chain and shape the demand plan based both on constraints and dependencies on internal and external partners.

The aforementioned activities help us comprehend the uncertainty we’re facing and allow organizations to create a value-for-the-organization vs. value-for-the-customer matrix, which will sharpen our focus on the relevant segments.

Injecting Certainty back Into The Business

Following the assessment of uncertainty, the following 4 steps can be taken to drive certainty in the demand planning process and outcomes.

1. Update Demand Plans Regularly

As the first step, Demand Planners must establish a frequent cycle of plan refinement in the short-term. This can be a combination of demand sensing based on daily order positions or POS information, and consensus planning for key segments on on a weekly basis. The key to success is understanding of the internal and external factors driving sales and the ability to incorporate the impact of external factors in the demand plan.

The goal of short-term plan refinement is to get the deployment of products in the network right. As a by-product, it may improve forecast accuracy as well. Another important thing to note is that the periodic demand planning cycle should not lose sight of the big picture of achieving the annual operating plan.

2. Document Risks & Opportunities In The Demand Plans

The second step is that the demand planning process must improve the governance around recording of risks and opportunities and their probabilities. Risks and opportunities are the most ‘certain’ aspects of uncertainty. In current times, we are seeing customer and channel stability risks, product vitality risks, fulfillment lead time variability risks, and so on. On the other hand, we are seeing innovation-driven opportunities in product channels; the eCommerce channel is enabling leaders in the CPG industry to reach consumers directly, efficiently, and profitably.

With such risks and opportunities, Demand Planners must establish risk and opportunity governance and their inclusion in the normal, aggressive, and conservative demand plans in collaboration with customer facing roles. The organization must develop a response plan in collaboration with supply chain roles for each of the three IBP-generated demand plans to ensure agility when the business situation changes from one plan to another.

3. Foster Certainty By Doubling Down On Your Cash Cows

The demand planning team can drive certainty in the demand plan by playing to their strengths. Organizations can use a growth-share matrix to identify their Cash Cow and Star products and customers, and double down on them.

The supply planning team must ensure that the warehouses, plants, and suppliers that cater to demand for such products and customers are resilient, responsive and generate positive financial value for the organization. Together, the demand-supply and the value equation should drive the supply chain capacity decisions and subsequent constrained demand plan.

4. Engage with Suppliers

Finally, in the fourth step, the demand planning team and associated functions in the organization must collaborate with the broader supply ecosystem to gain clarity. The information coming from upstream and downstream ecosystem partners under the CPFR framework brings early insight into demand changes and supply risks across all nodes of the supply network. It provides additional lead time to collaborate and respond to uncertain situations.

In conclusion, while change may be the only constant in life, understanding the reasons driving change go a long way to driving certainty in demand planning.