Without KPIs, it is impossible to improve forecast accuracy. Here are 8 highly effective metrics that allow you to track your forecast performance, complete with their formulas.

Forecast Accuracy

This KPI is absolutely critical because the more accurate your forecasts, the more profit the company makes and the lower your operational costs. We choose a particular forecasting method because we think it will work reasonably well and generate promising forecasts but we must expect that there will be error in our forecasts. This error is a function of the time difference between the actual value (Dt) and the forecast value (Ft) for that period. It is measured as:

 Forecast Accuracy: 1 – [ABS (Dt – Ft) / Dt]

Where,

Dt: The actual observation or sales for period t

Ft: The forecast for period t

Our focus on this KPI is to provide insights about forecasting accuracy benchmarks for groups of SKUs rather than identifying the most appropriate forecasting methods. For example, achieving 70-80% forecast accuracy for a newly-launched and promotion-driven product would be a good considering we have no sales history to work from.

SKUs with medium forecastability (volatile, seasonal, and fast-moving SKUs) are not easy to forecast owing to seasonal factors like holidays and uncontrollable factors like weather and competitors’ promotions etc., their benchmark is not recommended to be less than 90-95%.

Tracking Signals

Tracking signals (TS) quantify bias in a forecast and help demand planners to understand whether the forecasting model works well or not. TS in each period is calculated:

 TS: (Dt- Ft) / ABS (Dt – Ft)

Where,

Dt: The actual observation or sales for period t

Ft: The forecast for period t

Once it is calculated, for each period, the numbers are added to calculate the overall TS. When a forecast, for instance, is generated by considering the last 24 observations, a forecast history totally void of bias will return a value of zero. The worst possible result would return either +24 (under-forecast) or -24 (over-forecast). Generally speaking such a forecast history returning a value greater than (+ 4.5) or less than (-4.5) would be considered out of control. Therefore, without considering the forecastability of SKUs, the benchmark of TS needs to be between (-4.5) and (4.5).

Bias

Bias, also known as Mean Forecast Error, is the tendency for forecast error to be persistent in one direction. The quickest way of improving forecast accuracy is to track bias. If the bias of the forecasting method is zero, it means that there is an absence of bias. Negative bias values reveal a tendency to over-forecast while positive values indicate a tendency to under-forecast. Over the period of 24 observations, if bias is greater than four (+4), forecast is considered to be biased towards under-forecasting. Likewise, if bias is less than minus four (- 4), it can be said that the forecast is biased towards over-forecasting. In the end, the aim of the planner is to minimize bias. The formula is as follows:

Bias:  [∑ (Dt – Ft)] / n

Where,

Dt: The actual observation or sales for period t

Ft: The forecast for period t

n: The number of forecast errors

Forecaster bias appears when forecast error is in one direction for all items, i.e they are consistently over- or under-forecasted. It is a subjective bias due to people to building unnecessary forecast safeguards like increasing the forecast to match sales targets or division goals.

By considering the forecastability level of SKUs, the bias of low forecastability SKUs bias can be between (-30) and (30). When it comes to medium forecastability SKUs, since their accuracy is expected to be between 90-95%, bias should not be less than (-10) nor greater than (+10). Regarding high forecastability SKUs, due to their moderate contribution to the total, bias is not expected to be less than (-20) or greater than (20). The less bias there is in a forecast, the better the forecast accuracy, which allows us to reduce inventory levels.

Mean Absolute Deviation (MAD)

MAD is a KPI that measures forecast accuracy by averaging the magnitudes of the forecast errors. It uses the absolute values of the forecast errors in order to avoid positive and negative values cancelling out when added up together. Its formula is as follows:

MAD: ∑ |Et| / n

Where,

Et: the forecast error for period t

n: The number of forecast errors

MAD does not have specific benchmark criteria to check the accuracy, but the smaller the MAD value, the higher the forecast accuracy. Comparing the MAD values of different forecasting methods reveals which method is most accurate.

Mean Square Error (MSE)

MSE evaluates forecast performance by averaging the squares of the forecast errors, removing all negative terms before the values are added up. The squares of the errors achieves the same outcome because we use the absolute values of the errors, as the square of a number will always result in a non-negative value. Its formula is as follows:

MSE: ∑(Et)² / n

Where,

Et: forecast error for period t

n: the number of forecast errors

Similar to MAD, MSE does not have a specific benchmark to check accuracy but the smaller value of MSE, the better forecast model, which means more accurate forecasts. The advantage of MSE is that it squares forecast errors, giving more weight to large forecast errors.

Mean Absolute Percentage Error (MAPE)

MAPE is expressed as a percentage of relative error. MAPE expresses each forecast error (Et) value as a % of the corresponding actual observation (Dt). Its formula is as follows:

MAPE: ∑ |Et / Dt |/n * 100

Where,

Dt: Actual observation or sales for period t

Et: the forecast error for period t

n: the number of forecast errors

Since the result of MAPE is expressed as a percentage, it is understood much more easily compared to other techniques. The advantage of MAPE is that it relates each forecast error to its actual observation. However, series that have a very high MAPE may distort the average MAPE. To avoid this problem, SMAPE is offered which is addressed below.

Symmetrical Mean Absolute Percentage Error (SMAPE)

SMAPE is an alternative to MAPE when having zero and near-zero observations. Low volume observations mostly cause high error rates and skew the overall error rate, which can be misleading. To address this problem, SMAPE come in handy. SMAPE has a lower bound of 0% and an upper bound of 200%. It does not treat over-forecast and under-forecast equally. Its formula is as follows:

SMAPE: 2/n * ∑ | (Ft – Dt) / (Ft + Dt)|

Where,

Dt: Actual observation or sales for period t

Ft: the forecast for period t

n: the number of forecast errors

Similar to other models, there is no specific benchmark criteria for SMAPE. The lower the SMAPE value, the more accurate the forecast.

Weighted Mean Absolute Percentage Error (WMAPE)

WMAPE is the improved version of MAPE. Whilst MAPE is a volume-weighted technique, WMAPE is more value-weighted. When generating forecasts for high value items at the category, brand, or business level, MAPE cancels plus and minus values. WMAPE, however, weights both forecast errors and actual observations (sales). When considered at the brand level, high value items will influence overall error because they are highly correlated with safety stock requirements and development of safety stock strategies. Its formula is as follows:

WMAPE: ∑(|Dt-Ft|) / ∑(Dt)

Where,

Dt: The actual observation for period t

Ft: the forecast for period t

Like other techniques, WMAPE does not have any specific benchmark. The smaller the WMAPE value, the more reliable the forecast.

Eric Wilson CPF introduces a new measurement of probabilistic forecast performance that measures error, range and probability all in one formula. It goes beyond grading the forecaster, instead providing a valuable snapshot of a forecast’s accuracy, reliability and usefulness – in a way that is significantly easier to perform than existing methods.

Editor’s comment: This formula may be the first to holistically measure forecast error, range, and probability, allowing you to compare models, compare performance, completely measure the forecaster, and identify outliers. Think MAPE, MPE and FVA but for range forecasts, and providing more insight. This easy to use formula could be a game changer in forecast performance metrics. Eric welcomes feedback directly or via the comments section below.

A New, Improved Approach to MAPE

 This article introduces a new theoretical approach I am considering for measuring probabilistic or range forecasts. Measures of forecast performance have already been developed over the past decades with TPE, Brier Score, and others proving to be effective. The problem that I encountered was that, whilst some evaluate accuracy and others evaluated performance of the distribution, few looked at both on the same scale in a way that was easy to use. To overcome this, I have developed a single scoring function for probabilistic and range forecasts that will allow a forecaster to measure their forecasts consistently and easily. This includes even judgmental or empirical range forecasts.

Why We Predict Ranges Instead Of An Exact Number

Probabilistic and range forecasts are forms of predictive probability distributions of future variability and have enjoyed increased popularity in recent years. What we’re talking about is the difference between trying to precisely predict exact sales in the future and predicting a range of sales during that period, or the expected variability. It is also the difference between being accused of always being wrong to proudly forecasting with 100% accuracy. For more insight into this see my previous article entitled Stop Saying Forecast Are Always Wrong.

Just because you are accurate does not mean you are precise, however, or that you couldn’t do better, and it does not mean all probabilistic forecasts are created equal. With the proliferation of these probabilistic models and range forecasts arises the need for tools to evaluate the appropriateness of models and forecasts.

Range Forecasts Give Us More Information

A point forecast has only one piece of information and you only need to measure that point against the actual outcome which is generally expressed in some type of percentage error such as MAPE. The beauty of range forecasts is you are providing more information to make better decisions. Instead of one piece of information you have possibly three pieces of information, all of which are helpful. They are:

1) The upper and lower range or limit

2) The size of the total range or amount of variability

3) The probability of that outcome. To appropriately evaluate this, you should look and measure all three components and look at accuracy and precision and reliability.

So, when we say that we will sell between 50 and 150 units and are 90% confident, how do we know if we did well? A potential way I am proposing is to use a type of a scoring rule (which, for the purposes of this article, I am referring to as a (E)score. Conceptually, scoring rules can be thought of as error measures for range or probability forecasts. This score is not an error measurement but helps measure the relative error and reliability of probabilistic forecasts that contain all three components.

The scoring rule I have developed simply takes the square root of the sum of mean squared errors using the upper and lower limit numbers as the forecast, divided by the actual plus a scoring function for probabilistic forecasts.

(E)score = RMSE / Actual + BS_f

Or

Where:

Uf=upper limit

Lf=lower limit

Outcome = if in or out of range (1 or 0)

P = probability assigned to the range

Ex. Let us assume that we have historical information that shows the probability of falling within the 50 units range +/- for each period (this could also be empirical data). We create our forecast with a wide range estimating actuals to fall somewhere between 50 and 150 units. Being a larger range, we are fairly confident and give it a 90% probability of falling within that range. In this example actuals come in exactly between the range at 100. For this forecast our (E)score= .72.

Sqrt((|100-150|+|100-50|)^2)/2+(1-.9)^2=.72

Dissecting  The Components Of The (E)Score Formula

 Step 1:  Breaking this down to its components begins with first determining the relative error. What is done is to measure the Mean Squared Error for the sum of the deviation of the actual to the upper limit and to the lower limit divided by 2 (total number of observations 1 upper and 1 lower).

((|actual – upper limit|+|actual-lower limit|)^2/2=MSE

((|100 – 150|+|100-50|)^2)/2=5000

One of the things that stands out here is that no matter the actual value, the numerator would be the same for actuals that fall within the range. So that if our range is 150 upper limit to 50 lower limit, no matter if the actual was 125 (absolute value of 125 – 150 + absolute value of 125 – 50 = 100) or if actual was 75 (absolute value of 75 – 150 + absolute value of 75 – 50 = 100), both are the same result of 100.  This is correct since the forecast is a range and consists of all possible numbers within that range and would be equal deviation from the actual. This also rewards precision and smaller ranges. Conversely, it penalizes larger ranges.

Step 2: The next step is taking the square root of this mean squared error and then dividing this by the actual. The square root brings back “units” to their original scale and allows you to divide by the actual. What you end up with a comparison of the RMSE to the observed variation in measurements of the actual and may be represented as a percentage where the smaller error, the closer the fit is to the actual.

Sqrt(MSE)/actual=Uncertainty Error (E)

Sqrt(5000)/100=71%

Step 3: For the reliability of the of the range forecast we combine this with a probabilistic scoring function. One way to do it is to compare the forecast probabilities with outcomes on a forecast-by-forecast basis. In other words, subtract the outcome (coded as 1 or 0) from the assigned forecast probability and square the result.

Here we are not looking at the precision of the forecast but measuring the skill of binary probabilistic forecasts and if the actual occurred or not within the stated range. If the actual result falls between the upper and lower limit, the outcome is treated as true and given the value of 1. If the actual falls outside of the range it is given a 0.

For example, if our upper limit was 150 and lower limit 50 and we gave this a 90% probability of occurring within that range and the actual was 100, then the statement is true. This would give a value of 1 minus the probability of 90% then squared which equals .01. If the actual had fallen anywhere outside the predicted range, our score would be much higher at .81 (value of 0 minus the probability of 90% then squared). Given it answers a binary probabilistic question (in or out of range) it does not give value to the size of the range or how far you may be out of range – only a score from 0 to 1. This is the reason for the uncertainty error and size and range precision, and what makes this new scoring function unique.

(Outcome Result – Probability Assigned)^2=Goodness of Fit (score)

(1-.9)^2=.01

It should also be noted that if you had given your forecast the same range (150 to 50) with only 10% probability and it did fall outside the range (if the actual was 200 for example) you would get the same score of .01 (value of 0 minus the probability of 10% then squared). Following the logic, it is understandable. What you are saying with the statement with 10% probability is that most likely the actual result will not fall within the range. The inverse of your statement is there is a 90% probability the actual will be anywhere outside of your range and your most likely range is all other numbers. So, if the actual is 200 or 20,000 you are more correct. While this is important, once again it is only half of the equation and why we look not just at the goodness of fit and reliability but put this together with measuring how wide of a range and how far from that range you are to get a complete picture.

Step 4: The final step is simply adding these two parts together to end up with a single score between zero and an unlimited upside.

Uncertainty Error (E) + Goodness of Fit (score) =(E)score

.71+.01=.72

The lower the score is for a set of predictions, the better the predictions are calibrated. A completely accurate forecast would have a (E)score of zero, and one that was completely wrong would have a (E)score only limited to 1 plus the forecast error. So, if you had forecasted exactly 100 units with no range up or down and forecasted this at a 100% probability of occurring, in our example with the actual of 100, you would be absolutely perfect and have a (E)score of zero. Your range in our equation is 100 to 100 making your numerator zero and for the reliability you have the equation of value of 1 being true minus 1 for 100% probability which also equals zero.

Sqrt((|100-100|+|100-100|)^2)/2+(1-1)^2=0

Forecasts are generally surrounded by uncertainty and being able to quantify this uncertainty is key to good decision making. For this reason, probability and range forecasts help provide a more complete picture so that better decisions may be made from predictions. With this we still have the need to understand and measure those components and metrics like the (E)score may help not to grade the forecaster but communicate the accuracy, reliability and usefulness of those forecasts.  

Imagine you’re in the middle of the monthly S&OP meeting, and you’ve come to the point on the agenda that says: “review forecast accuracy KPIs”. You can already feel the Salespeople zoning out because they don’t have a clue what you’re about to talk about. And when we talk at length about MAPE and the like, can you blame them?

Salespeople are not familiar with forecast accuracy measures. They have no reason to be because it is not clear how these KPIs add value. They don’t help sell a product and they don’t make money in any obvious way. For me this is quite understandable and touches upon an important concept – that there has to be demonstrable monetary value in every step of the S&OP process. If our KPIs don’t reveal to other functions how S&OP makes the company money, then we’re not doing or jobs properly.

One way to get Commercial’s buy in into S&OP process is to show forecast accuracy as a cause-effect relationship

Show Forecast Accuracy As A Cause/Effect Relationship

One way to get Commercial’s buy in into S&OP process is to show forecast accuracy as a cause-effect relationship, showing predicted sales and actual sales, and the reasons for those particular numbers. What’s more, we need to demonstrate tangible KPIs like inventory, OTIF (On Time In Full) and SLOB (Slow moving and obsolete). These are far more real and relevant to the Salesperson than MAPE. You can easy explain how decreasing SLOB has a positive effect on bottom line, but are you going to convince Sales of the value of MAPE? Almost certainly not.

Often we lose track of this basic idea – that everything we do is designed to make money.

Measures like inventory level, OTIF, SLOB and write-offs have two main causes: forecast or supply. Tracking root causes behind performance of these measures is a standard activity and translating these into dollarized amounts will get peoples’ attention. Inventory, SLOB & write-off cost driven by forecast error is something everybody in monthly S&OP meeting should pay attention to.

In most companies you can trace the reasons for unfulfilled orders. Lost revenue due to forecast error is easy to identify – SLOB can easily be attributed to over-forecasting, for example. It’s easy to communicate and gets quite lot of interest from other functions at the meeting. The agenda for the monthly S&OP meeting should include presentation of KPIs, not only forecast accuracy but all financial KPIs affected by the forecast. Make sure to discuss the root causes.

Dollarize Your Forecast Accuracy

The main goal of S&OP is to increase profitability by fulfilling the highest level of customer orders whilst optimizing inventory. It is impossible to present the results of a Demand Planner’s work without referring to main these main principles. But often we lose track of this basic idea – that everything we do is designed to make money. In Forecasting and Demand Planning, there is not enough space dedicated to dollarizing forecast accuracy. Dr. Jain, Editor of the Journal of Business Forecasting, said in one of his interviews that “an average company can save $3.52 mil. for every one-percent improvement in the under-forecasting error, and $1.43 mil. in improving the over-forecasting error.” That speaks for itself and if you can put MAPE in these terms, that X increase in your forecast accuracy has delivered Y dollar amount, then Sales will immediately be more engaged.

Showing your forecast accuracy in this way, including historical data to show improvement, is a way of communicating the value of the S&OP process. This can help secure resources for the process like more staff. Not only that, making it clear to Sales that we as Demand Planners actually make money will make them more inclined to give us their time.

The key to success here is to show the impact of forecasts in dollarized amounts.

Evaluating Your S&OP With KPIs Is Very Important

A big topic of conversation in the Demand Planning community is how to make S&OP work and research shows that a lot of he time, it just doesn’t work at all. If S&OP isn’t working for your company, remember that a good process measures its outputs and evaluates itself. As S&OP is the process that all function are plugged into, all functions should be interested in how it’s performing. Forecast accuracy metrics are not always interesting for commercial functions because they are complicated or appear irrelevant. The mathematical calculation formulas might be off-putting for the less numerically inclined, but the good news is that the interpretation is quite easy and straightforward. The key to success here is to show the impact of forecasts in dollarized amounts.

For many of us, the words “the forecast is always wrong” has become something we instinctively say. There’s nothing wrong with acknowledging there is variation in demand or admitting we may miss a projection. But when it becomes your automatic response to any miss and is believed to be an unavoidable part of forecasting, it is highly limiting. This seemingly harmless habit can actually lower the effectiveness of forecasts and the business’s confidence in them. What’s more, it justifies other people’s poor actions and focuses attention on the wrong things. 

As Demand Planners, We Need To Give Ourselves More Credit

I cannot help but believe that when everyone constantly says that forecasts are always wrong, it needlessly creates guilt in the poor Demand Planner’s mind and undermines their self-esteem. It’s hard to feel good about yourself when you keep falling on your own sword.

Maybe we should stop saying we are sorry and stop saying forecasts are always wrong. Repeating this mantra also sends the message that you’d rather be agreeable than be honest, when in fact our job is not to provide a number but to offer solutions. We need to stop using the crutch of inevitable forecast error and start having honest conversations and focus on what we can predict and what we can control.

When others say “the forecast is always wrong” what they really mean is that demand variability is perfectly normal.

It Actually Is Possible To Be 100% Accurate

Yes, it really is. But let us start with what constitutes accuracy. Accuracy is the degree of closeness of the statement of quantity to that quantity’s actual (true) value. While I accept that one’s ability to create an accurate forecast is related to demand variability, an accurate forecast does not reduce demand variability. Demand variability is an expression of how much the demand changes over time and, to some extent, the predictability of the demand.  Forecast accuracy is an expression of how well one can predict the actual demand, regardless of its volatility.

So, when others say “the forecast is always wrong”, what they really mean is that demand variability is perfectly normal. What we should be focusing on is that “while we can’t predict demand perfectly due to its inherent variability, we can predict demand variability” (Stefan de Kok). This is the difference between trying to precisely predict the exact point and accurately predicting a range or the expected variability.

A common example of this is trying to guess the outcome of rolling two fair dice compared to accurately predicting the range of possible outcomes. For the throw of the two dice, any exact outcome is equally probable and there is too much variability for any prediction to be useful. But the different possibilities for the total of the two dice to add up to are not equally probable because there are more ways to get some numbers than others. We can accurate predict that 16.7% of the time the two dice will add up to seven, and we can predict the range of possible outcomes as well as the probability of each outcome. While we may not know exactly what will happen, we can exactly predict the probability of it occurring. And if you predict the outcome within the probabilities, guess what? You are correct. Even though 100% precise is not an option looking at ranges or probabilistic forecast, 100% accuracy most certainly is within the realm of possibilities!

Bingo! We have officially proven everyone wrong and have our 100% accuracy.

Accurately predicting an outcome within a range of probabilities is more valuable than trying to forecast a single number.

Range Forecasts Give Us So Much More Information Than Single Point Forecasts

Besides being able to more accurately predict the probabilities of outcomes and ranges, we are also providing more relevant and useful information. When you predict the variability, this not only grounds our initiatives in reality but also gives us the power to make better business decisions. One way to counteract variability is to ask for range forecasts, or confidence intervals. These ranges consist of two points, representing the reasonable “best case” and “worst case” scenarios. Range forecasts are more useful than point predictions.

With any single point forecast you are providing a single point of information which you know is not 100% correct. With a range you are providing four pieces of valuable information: we not only know the point or mean but we also know the top, the bottom, and the magnitude of possible variability.

Measuring the reduction in error rather than the increase in accuracy is more valuable to us because there is a stronger correlation between error and business impact than there is between accuracy and business effect.

It doesn’t take much to see that such a probabilistic forecast, or even just a forecast with ranges and a better prediction of uncertainty, is useful information in supply chain planning. Now we know how much variability we need to plan for and can better understand the upside or downside risk involved. In addition, accurately predicting uncertainty can add enormous value. That’s because you are focusing on improving not only the average demand prediction, but the entire range of possible demand predictions including the extreme variability that has the biggest impact on service levels.

Your KPIs For Measuring Forecast Error Are Based On A False Assumption

Part of the problem with saying we are always wrong is that we measure our performance ineffectively. This is because our definitions of forecast error are too simplistic or misrepresented. Many people look at forecast accuracy as the inverse of forecast error, and that is a major problem. Most definitions of forecast error share a fundamental flaw: they assume a perfect forecast and define all demand variability as forecast error. The measures of forecast error, whether it be MAPE, WMAPE, MAD or any similar metric, all assume that the perfect forecast can be expressed as a single number.

I mentioned above that we can provide more information in a range of forecast probabilities and subsequently be more accurate. All we need now is a way to measure this and prove it. A metric which helps us measure the accuracy and value of these types of forecasts is Total Percentile Error (TPE). Borrowing Stefan de Kok’s definition, TPE “measures the reduction in error – rather than the increase in accuracy – since there is a stronger correlation between error and business impact than between accuracy and business effect.” For more detailed information about this calculation see Foresight Magazine’s Summer 2017 issue.

Nassim Nicholas Taleb described this type of forecast accuracy measurement in his book, The Black Swan. He explains the difference in measuring a stochastic forecast (using probability distributions) and more traditional approaches (using a single point forecast).  He states that if you predict with a 20% probability that something will happen (and across many instances it actually happens 20% of the time) that the error is 0%. Naturally, it would also need to be correct for every other percentile (not just the 20 percentile) to be 100% accurate.

Bingo! We have officially proven everyone wrong and have our 100% accuracy.

You need to stop using the crutch of inevitable forecast error and start honest conversations about what we can predict and what we can control.

Focus On The Process

Even though we should know there is no such thing as being“wrong”, we should still look at what we are measuring and incentivize the right behavior. Mean Absolute Percentage Error (MAPE) or Mean Percentage Error (MPE) will tell us how much variability there is and the direction, but they do not tell us if the Demand Planning process is adding value. The question shouldn’t be whether we are right or wrong, but whether the steps we are taking actually improve the results. And if so, by how much.

Forecast Value Added (FVA) analysis can be used to identify if certain process steps are improving forecast accuracy or if they are just adding to the noise. When FVA is positive, we know the step or individual is adding value by making the forecast better. When FVA is negative, the step or individual is just making the forecast worse. [Ed: for further insight into FVA, see Eric’s guide to implementing FVA analysis in your organization.]

The obvious advantage to focusing on these types of metrics and KPI’s is that we are not casting blame but discovering areas of opportunities, as well as identifying non-value added activities. By eliminating the non-value adding steps or participants from the forecasting process, those resources can be redirected to more productive activities. And by eliminating those steps that are actually making the forecast worse, you can achieve better forecasts with no additional investment.

I Beg Of You, Please Change Your Vocabulary!

At the end of the day, our goal is not necessarily to be precise but to make a forecast more accurate and reliable so that it adds business value to the planning process. We need to stop saying we are sorry for what is out of our control and start controlling what we know is possible. To do this, we must not only change our vocabulary but also change the way we are doing our jobs.

Most people are fixed on traditional forecasting process and accuracy definitions. The goal is for you to start thinking in terms of the probability of future demand. From there, you need to be the champion inside your organization to help others understand the value of what forecasts provide. You need to stop using the crutch of inevitable forecast error and start honest conversations about what we can predict and what we can control.

Question

Greetings Dr. Jain,

I am currently trying to develop the best way to calculate our forecasting accuracy. We end up with a significant number of SKU’s that have some shipments, yet did not have a forecast, and vice versa. In addition, We have a small number of SKU’s accounting for the majority of our volume. I am interested in the SMAPE formula and have two questions:

1. Seems there are a few versions of this formula, which is the best one?
2. Is this formula used often, if not, what is the best or most commonly used formula? (I tried to download the report on this, but this side seems to have some technical issues.)

Thanking you in advance,

Marcel Meijer,

Char-Broil LLC

Answer

Dear Marcel,

The SMAPE for measuring forecasting error is found only on the Internet. You won’t find any discussion on this metric in any forecasting book, nor in any forecasting conferences. In fact, I don’t know any company that uses it. My main concern with the formula that I see on the Internet is in dividing error by the average of actual and forecast to arrive at the percentage error. To me, the objective of the forecast error is to see how forecast deviated from the actual, and not from the average of actual and forecast.  This can be only accomplished by dividing the error by just actual.

Based on the IBF survey data, most of the companies use MAPE (Mean Absolute Percent Error), though I feel WMAPE (Weighted Mean Absolute Percent Error) is even better. There are a number of companies that use it. You can find their formulas in any forecasting textbook.

Having a large percentage of sales coming from a handful of SKUs is quite common. Pareto laws applies to most companies.

I hope this helps.

Dr. Chaman Jain

St. John’s University

Mike Gilliland

Below details Questions & Answers from IBF’s Webinar “What Management Must Know About Forecasting.”  If you missed it, no worries.  You can view it complimentary by clicking HERE.

1. If a product is not forecastable, what’s the most appropriate step to move the product to become forecastable?

Answer: The most effective way to improve forecast accuracy is to “make the demand forecastable” and a great way to do that is to lower the volatility of demand.  Most of what we do with our organizational policies and practices is to add volatility.  We encourage our customers to buy in spikes, and we encourage our sales people to sell that way.  This is completely contrary to quality management practices, which are all about removing volatility and making everything more stable and predictable.

Review sales and financial practices that are encouraging volatility, and either re-engineer or eliminate them and replace with practices that encourage everything to operate more smoothly.  (Examples of practices that encourage volatility are pricing and promotional activities, and the quarter end “hockey stick” to meet short term revenue goals.)  You should question whether these sorts of practice make sense by contributing to the long term profitability of your business.  If not, pursue ways to reduce volatility and encourage smooth, stable growth.  This will allow you to forecast more accurately and will reduce overall costs, which you can then pass along to your customers.

2. All of this is relative to the base line forecast, correct? What if your items are heavily promotional driven?

Answer: The accuracy of a naïve forecasting model serves as the baseline against which the performance of alternative forecasting methods should be compared.  Thus if the naïve model (say, a moving average) achieves MAPE of 40%, then I want to know how well my statistical model is forecasting, and how well my overall process is forecasting, and compare them to the baseline of 40% MAPE that the naïve model delivered.  This is what I’m talking about as a “baseline.”

This should not be confused with what is commonly called the “baseline” forecast when you try to distinguish baseline demand from promoted demand.  How do you know what demand was baseline and what was due to the promotion?  How do you distinguish the two?  I don’t believe that you can distinguish the baseline demand from promoted demand in a clean or easy or certain manner, so I would suggest not bothering trying to do so.  What matters is “how much total demand is there going to be.”  It isn’t necessary for me to care how much of it is “baseline” and how much of it is due to “promotion” – and I can never know for sure anyway?  Don’t assume you are making your forecasts  more accurate by trying to distinguish the two – you may just be making things more complex.

3. What is FVA?  A tool?  Expert judgment?  Or what?

Answer: Forecast Value Added is a metric, defined as the change in a forecasting performance metric (such as MAPE, forecast accuracy, or bias), that can be attributed to a particular step or participant in the forecasting process.  When a process step or participant makes the forecast more accurate or less biased, they are “adding value.”  FVA is negative when the step or participant is just making the forecast worse.  FVA analysis is the method of reviewing the performance of your process and identifying those non-value adding (or negative-value adding) activities that should be eliminated.  For more information on FVA analysis, see the webinar “Forecast Value Added Analysis: Step-by-Step” or the accompanying white paper.  You are also encouraged to attend the IBF Supply Chain Forecasting conference in Phoenix, February 22-23, 2010, to learn how to do FVA and hear case studies about several organizations (such as Intel) that are using this method.

4. What are the methods used commonly to measure Forecast Accuracy?  (Is MAPE the most common?) And what is a good process to determine forecast accuracy?

Answer: Mean Absolute Percent Error (MAPE) or its variations like Weighted MAPE or Symmetric MAPE seem to be the most popular metrics of forecasting performance.  MAPE has many well known limitations (such as being undefined when the denominator (the Actual demand) is zero), and is not suitable for use with data with a lot of zeroes (intermittent demand).  Also note that with MAPE you can have absolute errors greater than 100%, so you cannot simply define forecast accuracy as 100% – MAPE.

For management reporting I use a “Forecast Accuracy” (FA) metric, defined as:

1 – { Σ | Forecast – Actual |  /  Σ Maximum (Forecast, Actual) }

Note: FA is defined as 100% when both Forecast and Actual are zero.

By using Maximum of Forecast or Actual in the denominator, FA is always scaled between 0 and 100%, so it is very easy for management to understand.  That is why I favor it, even though some professional forecasters are very critical of this metric.

5. What are your perspectives on how do you differentiate volatile demand from uncertain demand?  In my opnion, uncertainty is related to an event and volatility is related to demand fluctuations. Is that right?

Answer: Volatility is expressed by the Coefficient of Variation (CV), which is the standard deviation divided by the mean.  For example, look at the last 52 weeks of sales, and compute the CV of that pattern.  In general, the more volatile (i.e. erratic and variable) the demand, the more difficult it is to forecast accurately.  Recall the Accuracy vs. Volatility scatterplot in the webinar.

Sometimes we can forecast volatile demand quite accurately, where there is structure to the volatile pattern.  You might see this for highly seasonal items, where you can always count on a big spike in demand at a certain time.  (E.g. bunny costumes and egg painting kits before Easter.)  Note: I’m not claiming we can forecast bunny costume or egg painting kits accurately, just using them as an illustration of volatility due to seasonality.

Volatility is measured looking back at what really happened.  If we expect high volatility to continue, we would probably have less confidence or certainty in our future forecasts.  If volatility is very low, we can probably feel more secure (and certain) of our forecasts.

6. Is there any ratio to determine the horizon for the forecast to be measured?  Any industry correlation to lead times?

Answer: Forecasting performance should be reported relative to the supply lead times.  Thus, if it takes 3 months to make changes in your supply, you should measure the accuracy of your forecasts made 3 months in advance. Once inside this lead time, it is ok to continue to make adjustments to the forecast, and many companies even report their forecast accuracy based on a forecast immediately prior to the period being forecast.  (Some companies even allow adjustments to the forecast within the time period (e.g. week or month) being forecast – and then report that as their forecast accuracy.)  However, it is the forecast made at the lead time that really tells you how well (or how poorly) you understand your business.  Don’t congratulate yourself on good forecasts made within the month being forecast!
Regarding forecasting horizon – how far into the future you should forecast – this will vary based on your business needs.  A power company forecasts years (even decades) ahead to know if it will need to make capital investments in new power plants.  For most companies, forecasting 12-18 months ahead is sufficient.  And the forecasting process should always be “rolling,” so that you always maintain that horizon of forecasts ahead of you.

Routinely doing 5-year ahead forecasts if you don’t really need them seems like a silly exercise.  If management insists on forecasting farther ahead than you really need, don’t waste much time doing it.  It is very unlikely you can forecast very accurately that far ahead.  It is much better to keep your organization nimble and able to adapt to however your market changes over time, rather than fool yourself into thinking you can accurately predict that far into the future.

7. How can you do calculate “appropriateness for forecasting” when your time series is too short for out-of-sample testing?

Answer: When there is enough data, out-of-sample testing is a great way to help evaluate and select forecasting models.  Good software, such as SAS Forecast Server, allows you to define and utilize a holdout sample in your model selection process.  Poorly designed software will select a model based solely on “best fit” to recent history, and as was illustrated in the webinar, the best fitting model may be a very poor choice for generating forecasts.

When there is not enough history to use a holdout sample, the appropriateness of a model is based on the judgment, experience, and domain expertise of the forecaster.  In the webinar example, Model 4 fit the history perfectly, but the forecast exploded to huge values which probably weren’t realistic (unless you had domain knowledge that demand would be significantly increasing, you were rolling out to new regions, etc.).  Without any other information, using the mean (Model 1) or a simple trendline (Model 2) seemed to be “most appropriate.”

8. Statistical modeling can be difficult in planning service parts demand. Can you give further input for planning service demand volatility.

Answer: Demand for service parts if often intermittent, with lots of periods of zero demand.  Intermittent demand is difficult to forecast accurately.  Although there are various methods to help you forecast and manage inventory in these situations (see Croston’s method and its variations), you should not have high expectations for accuracy.  It may be easier (and just about as effective) to simply forecast the mean demand each period.

Sometimes there is sufficiently high demand for the service parts that you could use standard time series methods to forecast.  It may be helpful to incorporate known sales of the items requiring the parts, so you can base your forecasts on failure rates.  Thus, if you know 100,000 units of a product were sold, and that 10% require servicing every year, this could help you decide that about 10,000 of the service parts will be needed each year.

One other approach, more applicable to high value machinery (e.g. jet engines, ships, factory production lines), is knowledge of the routine maintenance schedule.  If you sell 1000 jet engines and the maintenance schedule says a part is replaced every 6 months, then you can use this to forecast demand for that part.

9. Do you have examples available of cost of inaccuracy metrics?

Answer: I do not have access to the Cost of Inaccuracy metric used at Yokohama Tire Canada by Jonathon Karelse.  However, Jonathan will be speaking at the IBF’s Demand Planning & Forecasting: Best Practices Conference in San Francisco (April 28-30), so you could follow up with him there.

IBF members have access to a cost of inaccuracy spreadsheet available on their website.  Also, analyst firm AMR has published research (which you could access if you are an AMR subscriber) on the costs of forecast inaccuracy.
Any such cost calculators are based on a number of assumptions which you provide, so be cautious in your use of them and in your interpretation of the results.  Personally, I’m very skeptical of claims such as “Reducing forecast error 1% will reduce your inventory costs x%.” If nobody in your organization trusts your forecasts now, reducing the error by 1% is not going to make anybody more trusting of the forecast, and they won’t change any behavior, so you won’t reduce inventory.  It may take more substantial improvement to reap the cost benefits.

10. Does anyone work in the call or contact center environment for an inbound customer service center?

Answer: These principles can be applied to forecasting demand for services, such as in forecasting needs for call center staffing.  The major difference is the time bucket that is of interest.  Call centers often forecast in 15 or 30 minutes increments (rather than in weeks or months for a manufacturer), to make sure they are sufficiently staffed during peak call periods, and not overstaffed during the low call times.

Michael Gilliland
Product Marketing Manager, SAS
IBF Board of Advisor

See MICHAEL GILLILAND & EMILY RODRIGUEZ from INTEL Speak in Phoenix at IBF’S:

$695 USD for Conference Only!

February 22-23, 2010
Phoenix, Arizona USA