Selecting the right model plays a very important role in predictive analytics and forecasting. Use the wrong model, and you might as not have bothered at all. Use the right one, and you have a robust forecast you can plan your business operations around. So how do you choose the right one?

Each model captures a specific data pattern, each model has a shelf-life of its own, each method yields unique results, each model reacts differently for different time horizons. There are hundreds of variations of baseline methods that can be combined into thousands of models with unlimited steps and inputs you can choose from. Finding or building the perfect one may be a one in a million proposition.

Finding the needle in the haystack, or the best method or model, goes back to all the work we put in prior to arriving at this step. It is about understanding the problem (whether it be for descriptive or anomaly detection or clustering or regression or predictive,) categorizing the inputs and outputs, and knowing your data and its limitations. It comes down to a combination of business need, specification, experimentation, and time available.

Keep the following specific points in mind when finding, building, using, or analyzing any model or method:

One Size Does Not Fit All

The one thing we know for sure when it comes to modeling and predictive algorithms is that despite all the possibilities available, there is no one approach that caters to all your problems. Even the most experienced data scientists cannot tell you which algorithm will perform the best before experimenting with others. The good news is you don’t need to get it right first time. You can pick or build an algorithm that nearly solves your problem and then, over time, customize it to improve it to solve your particular problem.

Keep It Simple

It is easy to get lost in the details or think bigger is better, but it is best to select simple methods initially and use simple procedures unless you can clearly demonstrate that you must add complexity. Complex methods may include errors that propagate through the system or mistakes that are difficult to detect. The more complex and the more features there are, the more specialized techniques you need. If can’t explain it, then you probably can’t use it properly. Start with what you know and what you can do and then experiment, adjust, and iterate models over time.

Forecasting/Analytical Models Should Meet The Situation

The predictive or analytical model should provide a realistic representation of the situation. You need the right test of models, inputs, parameters and situations to match the current problem. You’re looking for the right balance between accuracy and the potential for overfitting. You also need enough time to develop and train/tune the model.

There Is No Magic Bullet For For Forecasting Models

Because we know that no model in the world works in every situation, it is important to look at different ones. By combining forecasts, you can incorporate more information than you could with one forecast. Studies have even show that combining methods reduces error by 12.5 percent compared with a single method. Combining can also have the potential to reduce risk due to effects of bias associated with a single method.

Forecasting Models can get old

Just because it works the first time does not mean it works every time.  Patterns change, data changes, features change, and reactions within models change. Update models frequently as the underlying data or environment changes or revise parameters as new information is obtained. Make sure you use quantitative approaches and objective tests of assumptions and models.

Bottom Line

There is no “one fits all” algorithm or model. Choosing the right one depends on several factors including:

• Purpose,
• Data size, quality and diversity;
• And resources available.

There are also additional considerations like accuracy, training time, volume, parameters, data points and much more. This is where we come in, and it is the demand planner’s role to help choose the right model that fits the data and the underlying truths, utilizing our experience and professional knowledge.

Mark Frost

D. Reilly

The “time value of money” is at stake when you are trying to forecast demand at ATM machines and of course, customer satisfaction. Trying to get the right amount of cash for pay day and holidays requires some pretty complicated models to get this right.  The reality is that these methods and approaches of forecasting daily cash demand are just as necessary when forecasting what is perceived to be “simpler” problems.  The S&OP process has treated the importance of a baseline forecast as just a “stepping stone”.  A big reason why the S&OP process is leaned on so heavily is baseline forecasts are often generated using a simplistic model that doesn’t capture patterns into a model, but rather fits a pre-specified model to the data.  A quality baseline model and forecast can alleviate a lot of the work downstream. Another comment about adjusting forecasts is that if it is for a reoccurring reason it can be added as a causal variable to the model and utilized “in-line” or also “in-model”.

When building a forecasting model, it’s important to recognize how variables like “day of the week”, “week of the year”, “day of the month”,  and holidays can capture the swings in demand and allow you to plan for them.  It’s not just the holidays, but the days before and after the holidays that need special consideration as demand ebbs and flows around these events.

Furthermore, we often hear “I am fed up with fixing forecasts”.  This can be alleviated by taking a more rigorous approach to identifying patterns rather than have some list of 50 models to be forced onto a data-set “hoping for the best” without any care for what patterns are occurring in the data.  The “one size fits all” modeling approach by taking 50 models and forcing them on a data-set is like fitting a square peg in a round hold.  Customized suits are exactly that.  Custom suits are yes custom and they take a little work requiring the expense of a tailor, but you have a proper fitting product at the end of the process. “One size fits all” can result in a hat that just doesn’t fit your head as we have seen!

Join us at IBF’s Supply Chain Forecasting & Planning Conference in Phoenix to further discuss the above.  Plus, our discussions will also cover “motherhood and apple pie,” what you need to know to make better decisions about what makes a good baseline forecast.

Your comments and feedback are welcome here!

Mark Frost
Director of Business Strategy and Decision Science
Fiserv

David Reilly
Sr. Vice President
Automatic Forecasting Systems

See MARK FROST & DAVID REILLY Speak in Phoenix at IBF’S:

$695 (USD) for Conference Only!

February 22-23, 2010
Phoenix, Arizona USA