In today’s increasingly competitive markets, one way to stay ahead of competi­tors is to launch more and more new products. According to an Institute of Business Forecasting survey, about 17% of companies’ sales revenue now comes from new products, and is growing. To be ready for new product introductions, three steps are necessary:

1. Prepare forecasts based on different scenarios,
2. Negotiate conditional contracts with suppliers, and
3. Be quick in dropping a product that no longer yields enough margin.

1. Scenario Forecasting

New products are the most difficult to forecast because there is no history to go by. Market experience tells us that, by and large, 7 out of 10 new products fail. So, the best strategy is to use scenario forecasting, where forecasts are based on what the likely minimum and maximum sales volumes. Higher volumes typically lower the cost per unit and, thus, have a positive effect on the margin per unit. But most new products are over-forecasted. So, don’t think in terms of cost; rather, think in terms of realistic numbers you can expect, and the risk you can bear.

In the absence of historical sales data, the best way to forecast for new products is to use the Delphi method, where different stakeholders submit their forecasts along with comments. By reviewing different forecasts based on different assumptions with a number of iterations, companies usually arrive at realistic numbers.

2. Negotiate Conditional Contracts With Suppliers

Because of high uncertainty in the demand for new products, it is in the best interest of a company to have flexible contracts with suppliers, which allows, to some degree, the raising or lowering of orders as needed. If you negotiate a contract based on cost, which is mostly the case, you are likely to be stuck with too much inventory of raw materials.

The supplier is willing to give a price break only if you agree to place a large order and give a longer lead time. This can be a problem because products can fail, and their forecasts turn out to be much higher than the actual demand. If, on the other hand, you try to place a minimum order, you may miss opportunity to capitalize on products that do much better than expected. With that, you will not only lose sales but also your reputation in the market. Depending on the supply chain, it may take weeks or months to fully recover. The best thing, under the circumstances, is to look for suppliers that are willing to sign a flexible contract that allows you to raise or lower the quantity by a certain percentage after a certain number of weeks following the launch.

3. Quickly Get Rid Of Poor-Performing Products

It is much easier to launch a product than to drop one. I have seen companies struggle with the idea of discontinuing a failing product. If I’m in sales, I would object too, because dropping it will hurt my revenue numbers. What should matter is not how it would impact a specific function, but how it would impact the company as a whole.

We need a certain return on investment for our new products and the issue should be whether it is generating enough profit for the company. When a product comes closer and closer to the end of its life cycle, sales starts going down, competition from low-cost players intensifies, and margin deteriorates. Because of this competition and decrease in sales, the demand pattern becomes more erratic, and the quality of the forecast deteriorates. The increasing uncertainty about forecasts causes an increase in inventory and a decline in customer service. So, what we wind up with is low margin and more inventory. Consequently, the best thing to do is to rank products in term of margin and be ruthless in dropping those that aren’t driving sufficient profit.

This article originally appeared in the Fall 207 issue of the Journal of Business Forecasting. Click here to become an IBF member and get the journal delivered to your door quarterly, as well discounted access to IBF training events and conferences, members only workshops and tutorials, access to the entire IBF knowledge library, and more.

The world of forecasting today is filled with machine-related buzzwords like AI, predictive analytics and machine learning. Will all of forecasting be done by machines a few years from now? Do humans still have any competitive advantage over software? In the realm of new product forecasting – the human wins.

Human trumps machine forecasting in the following areas:

• Measuring promo impact
• Gauging life cycle trends
• Disentangling overlapping marketing tactics
• Incorporating qualitative analysis

Machines have great potential to improve forecasting processes. Automating forecasts for existing products that have a lot of data can free human time to focus on new product forecasting. However, engaging software in forecasting all products, both old and new, can pose serious risks for forecast accuracy.

How Much Value Does AI Really Add?

Hollywood has been questioning whether the machines will take over the world for almost half a century so it must be true. How will it start? Machines flipping burgers, executing court rulings or maybe doing demand planning? Demand planners are naturally intrigued about the latter. Industry trend-setters prefer to talk about AI fueled by machine learning algorithms over Excel books fueled by ridiculous amounts of coffee. Buzzwords sell, but do they really add value?

To answer that question we need to ask what AI and machine learning are. In the context of forecasting, these disciplines are essentially a series of algorithms that create baseline models and measure promotional impacts. The ‘machine learning’ component is a fancy term for the trivial process of feeding the algorithm with more data. These tools are very useful for forecasting products with lots of history and homogeneous promotions. These types of products are usually the easiest to forecast. Forecast error for products with lots of history is typically low, regardless of the level of automation. The tools save a lot of time and brain cells from doing tedious repetitive tasks, but they hardly improve forecast accuracy.

Can Algorithms Work With Little To No Sales Data?

But what about new products that have no or little sales data? Ironically these are also the ones that are subject to all kinds of marketing extremities. Can any machine algorithm forecast those? The short answer is no. And this is why:

1. Measuring promo impact as % vs Baseline may be misleading

How does software measure promo impact? Most of the time it would just compare actual sales to the baseline. % difference would be the promo impact that the machine will use to forecast future promotions. However, reality might be a bit more complex than that. Imagine you are forecasting a winter ice cream promotion based on a previous summer promotion. Your summer baseline was 100 units and you sold 200 units as the result of the promotion. The software calculates a 100% lift. Impressive right? Let’s apply that to your winter baseline of 15 units. Not that impressive anymore. A human can create a promo lift that might not be calculated by a particular formula but nevertheless is reasonable.

2. Product life cycle trends can throw machines off

Most products have a sustain period after the launch. This is the period of the product life cycle right after the initial launch when sales gradually decline until they level off and start exhibiting seasonality. Software is not very efficient at forecasting this trend. It usually either underestimates future sales by applying the negative trend indefinitely or overestimates by applying seasonality to sales that are still elevated after the launch. A human can apply common sense to determine when the sustain period ends and normal seasonality starts.

3. Disentangling components of promo impact via algorithms may result in lower accuracy

Every new product promotion is unique with its own mix of marketing support, pricing and product strategy. Imagine you are forecasting a promotion based on two historical promotions. One historical promotion had a 10% lift and the other a 100% lift. An algorithm would use the average 60% lift for the forecast. In most cases this would be inaccurate. A human can make a judgement call on what particular factor differentiated the two historical promotions and choose the most similar one as a proxy for the forecast. I know what you’re going to say, “But we can teach the machine to recognize different kinds of promotions!” This brings me to my next point…

4. Teaching machines may be more work than doing it yourself

When a human analyses historical promotions he/she can look at different files, process different formats and even ask for other human opinions. A machine, however, demands to be spoon-fed data in a consistent format from one source. Creating a variable for every marketing strategy for every historical promotion can be very time-consuming.

5. Are you working on a forecast or on a model?

A statistician will say that all of the above issues may be resolved within a model by overriding product life cycle trend, flagging different kinds of promotions, using proxy seasonality, etc. This is true. The caveat is that these processes require a lot of human hours. Human hours dedicated to perfecting the machine, rather than perfecting the forecast. Anyone who has worked with models knows that statistical significance does not always go hand in hand with common sense. Overrides and interventions increase the risk of over-specification of the model. This means that model results stop making sense because there is not enough data to support all the additional variables.

6. Humans can think outside the box while machines are the box

Forecasting new products that have no data is solely reliable on proxy selection. Imagine you are forecasting a new LTO flavor of ice cream – dark chocolate. You have history on two LTOs – coffee and caramel chocolate ice cream. Which flavor will you choose as the proxy? A human will consider a plethora of factors. Which of the 2 historical flavors was more similar in taste? Is the bitterness or chocolatey-ness driving the sales? Is there any consumer research on the two flavors? Which of the promotions had similar marketing support? Was the creative more appealing for one of the historical flavours? All these questions belong to the qualitative, not quantitative realm. Hence, the software will not be helpful.

So what is the right balance between human and machine in forecasting? The answer is obvious – leave the quantitative tasks to the machines and qualitative to humans. The problem is that there is a fine line between the two in the realm of forecasting. The best term for a forecast in my mind is ‘a scientific guess’. We don’t want to leave the guessing part to the machines as it relies on intuition – an inherently human feature. It is worthwhile investing time in the exercise of classifying forecasts into ones that are generic enough to be done by machines and the ones that require a human touch. Machine and human can even efficiently cooperate within one forecast. For line extensions, software can predict the baseline level and even the promotional impact for the total line. Human can decide what proxy to use for the new flavor.

New product Forecasting Software Not Worth The Price

The good news is that software can effectively and efficiently forecast products with lots of history and homogeneous promotions. And they’re not expensive and are very easy to operate. There are even add-ons in Excel that can create baselines for hundreds of SKUs at different levels – provided you have enough data. It is the software that claims to forecast new products that can get pricey while providing dubious value add.

Demand planners needn’t fear being replaced by machines – only humans can provide valuable qualitative judgement.

Reconciling Quantitative & Qualitative Is Key In new Product Forecasting

Automation can release human time that can be spent on increasing forecast accuracy on the products that are the most difficult to forecast. There is a lot of room for improvement in the realm of forecasting. Average error for new food is around 50%. This equates to lost profits and excess inventory. One third of global food production is wasted, a problem that software alone will not resolve. Perfecting qualitative methods of forecasting is as important as quantitative, and even more important, is developing methods for reconciling the two. There are numerous consumer research specialists in the retail industry selling their qualitative expertise on new product appeal. Developing a system that converts such consumer research into real numbers that can be used for forecasting can significantly improve forecast error.

The “Amazon effect” has already changed shipping, logistics, employment and consumer behavior, not to mention giving a good kicking to brick-and-mortar stores. As a result, Demand Planning and Forecasting is very different to how it was just a few years ago, and our jobs have evolved drastically. We must understand that Big Data and shifting consumer demand have changed everything.

New Opportunities And Challenges in Demand Forecasting

When it comes to Demand planning and Forecasting, Amazon and – e-Commerce generally – has been a phenomenally disruptive force. This new world is impacting what we forecast, how we forecast it, and when we forecast it. This creates challenges but there is also a key opportunity presented by the new e-Planning environment, and that lies in the abundance and availability of data. Big Data is driving the following fundamental changes in Demand Planning:

• Machine Learning and Neural Networks are replacing traditional time series methods
• Web crawlers are replacing syndicated data
• We are using more real-time or daily forecast at a more granular level
• We are using more analytical decision making and demand shifting techniques

Adapt Your Techniques

In the new e-Planning world, we are forecasting orders less and looking more at a wider variety of inputs. The most well-established forecasting techniques are based on historical demand and they have served us well, but with e-Commerce, we have a new wealth of information that tell us not only about the future, but also the present. Predictive analytics encompasses a variety of statistical techniques like predictive modeling, Machine Learning and data mining that analyze current and historical facts to make predictions about the future – all in a way that we could never hope to replicate with time series forecasts.

Once you understand the drivers, you can  influence demand like never before.

Instead of looking at just shipments or sales history, we have access to website clicks, rankings and the number of customer reviews. Using a multivariate regression or recurrent neural networks, for example, you can isolate variables and determine their impact to predict future sales. The greatest benefit of looking at these attributes and variables is that we may not only be able to predict sales by week going forward, but also understand what will change and by how much.

Prediction is becoming more about behavior than history. This is powerful because once you understand the drivers, you can  influence demand like never before. In the age of Big Data, therefore, we can be proactive instead of reactive. Amazon are a perfect example of this; when shopping on the Amazon website, you will have seen the recommendation engine. It recommends an item based on what you purchased before on the same website. Your recommendations are filtered by a variety of drivers such as genre, price, brand and interests etc.

Be More Agile

In today’s business environment, changes in the marketplace are swift and sudden and may not follow the historical pattern, meaning time series models cannot always be relied on for accurate forecasts.  The new e-Planning environment is not only dynamic, it operates on the power of technology and innovation. If it is still taking you a month to gather data, create a forecast, add assumptions, and develop a forecast, I’ll tell you now that you won’t be able to keep up. But you may know that already.

The winners in this new era will be the ones that can see, interpret, and act on data the most efficiently.

In many industries, forecasting actual product sales is often a lengthy business process. With e-Commerce, however, you are competing almost in real time with price, features, and delivery promises. And feedback comes just as quickly in the form of reviews and competitive responses. To be more agile, companies are looking at demand sensing techniques to translate the drivers into rules based or machine-learned responses. This brings us closer to not only to the level of demand, but also closer to demand intent.

Even more importantly, the e-Business environment provides an opportunity to readily collect information about potential or future buyers. Along the different channels of communication between an organization and its potential customers, most companies now routinely log every visit to the product webpages, every call made to an inquiry response center and every email that was received. Many organizations also use every customer touch point as an opportunity to perform a brief customer survey to collect information about their customers and comments on their products.

Where traditional demand sensing focuses purely on Point Of Sale (POS) data from retailers aggregated weekly, you are now absorbing sales on an hourly basis or even quicker. Where third-party syndicated data from Neilsen and others could help you better understand markets and competitors, now you have web crawlers that traverse multiple sites and bring you relevant data whenever you want. We have more data than ever, and we’re getting it faster than ever before – the winners in this new era will be the ones that can see, interpret, and act on it the most efficiently.

Get Detailed On Promotions

Targeted marketing has created complications in the analysis of promotional effects. The traditional way of applying a general “lift factor” to nominal demand when a certain promotion is performed may not be adequate.  Add to this the changes we are seeing with dynamic pricing models, and time series forecasting reveals its limitations.

Online reviews correlate to sales so much that you can even use them in modeling.

Changes in the selling price and the presence of product promotions are known to have a significant effect on demand in many industries. Today, in large part due to analytics and the proliferation of data, price changes and promotions are cheaper to do and have greater impact. Price changes on the web or a pop up on your website shifting demand incur little incremental cost. Even in traditional retail stores, the day will soon come when a button on a computer is pressed to issue a price change, and new prices will be reflected on an electronic label in a physical store a few seconds later. Such opportunities imply that price changes and promotion actions may be used very frequently, and so they can no longer be analyzed separately from “normal” demand. This requires a disciplined process to capture the information in a timely manner.

Offer What the Customer Wants

One of the biggest challenges we have seen in demand forecasting over the past few years is shorter and shorter product life cycles. This is absolutely necessary to meet consumer demand. In the e-Planning realm, this will be compounded by a whole new set of challenges. Why? Because constant new products do little for building up the necessary reviews in the e-Commerce marketplace. Think of your experience shopping online – you have one item with 2000 reviews and 4-star rating and another with just 5 reviews and a rating of 4. It is actually more beneficial to keep older items with good rankings than introduce new items every 18 months. Online reviews correlate to sales so much that you can even use them in modeling.

This works well for items with multiple reviews but we still need to replace poorer performing products.  With e-Commerce going directly to consumers, speed to market is crucial. What’s more it makes it very difficult to plan. Because individual products are phased out and new products come in constantly over time, and the fact that the hierarchy might be reorganized fairly frequently to reflect the fast-changing business environment (e.g. gaining or losing major customers or markets), the product hierarchy is dynamic.

And there you have it. Amazon has totally revolutionized the marketplace, and with it demand forecasting and Demand Planning. If there’s one there’s one concept that all forecast analysts and Demand Planners must understand, is that companies will live and die by their ability to gather, interpret and act on data.

Today’s markets have a greater number of new products than ever before. But where do the ideas for all of these new products come from? Usually from Sales, Marketing, and Product Development executives. But do these people know the impact a new product will have on the existing portfolio? Do they know if the new product will actually drive growth? Not always, because the Demand Planners who hold the relevant data are often kept outside of the new product process. Here I reveal how important they are and where exactly they should fit into the Stage Gate process.

Success Or Failure of New Products Depends On The Company’s Growth Stage

But first, we must understand the importance of the particular growth stage the company is in. If a company is the mature stage of its growth cycle, new products that find new customers are crucial to company growth. At this point, it is more crucial than ever that Demand Planners are brought into New Product Development process in order to maximize successful launches. Remember that no new product is launched in isolation and they have inevitable impact on the whole portfolio.

Demand Planners must enter the New Product development process to neutralize bias and reveal which product ideas are viable growth drivers and which are not.

Companies undergo the following phases of growth:

1. Startup: New solutions for new customers
2. Portfolio Expansion: New solutions for existing customers
3. Maturity: Existing solutions for existing customers
4. Market Expansion: Existing solutions for new customers

New Products Must Lift A Company Out Of The Maturity Phase

This is where the organization finds itself at a crossroads. By staying in the upper right quadrant, they continue to introduce new products, but rather than producing true growth, they may instead be cannibalizing existing product sales, whereby a new product eats into the demand for an existing product. In some cases, the older product may be more profitable and easier to manufacture. Not only that, companies run the risk of rapidly increasing the number of SKUs they need to manage, creating complexity whilst not actually acquiring any new customers. In this all too familiar scenario, they are only creating new ways to provide the same service to the same customers. No growth is achieved, instead, the company is wasting time and resources on making everything more complicated.

Organizations that stay in this quadrant run the risk of stagnating, and often fail to appreciate this situation because of two key biases. The individuals most responsible for developing new product strategies tend to overestimate the return on investment for new products and underestimate the sustainment costs of managing increasing numbers of SKUs. Demand Planners must enter the New Product development process to neutralize bias and reveal which product ideas are viable growth drivers and which are not.

Where Demand Planners Fit Into The Stage Gate Process

Advanced organizations that execute New Product best practices use a Stage Gate process that includes Demand Planners at each step. A key component of this process, Demand Planners assess viability from a data driven, statistical standpoint, and if done correctly, will maximize the chances of new product success. Demand Planners fit into Stage Gate Process as follows:

1. Investigation: In this initial phase of investigating potential new products, Demand Planners must help categorize poor performing SKUs or markets, develop strategic plans and identify gaps in these plans.
2. Feasibility: Secondly Demand Planners must create scenario plans to identify opportunities and risks to understand the impact the new product will have on the existing portfolio and wider business. They should also contribute qualitative assumptions underpinning the growth proposition.
3. Plan: They must then integrate these assumptions into the plan and develop the opportunities available
4. Development: In this stage, Demand Planners must fine-tune parameters and planning to ensure launch preparedness.
5. Launch: Once gone to market, Demand Planners should monitor sales and update forecasts accordingly. Here, the Demand Planner must ensure enough supply is available, or in case of poor sales must suggest ways to improve sales or decide to withdraw the product.

The Enthusiasm of Sales and Marketing Must Be Tamed By Demand Planners

Let’s consider the incentives placed on the Sales & Marketing and Product Development teams responsible for these new product decisions. Sales wants to be able to sell as many SKUs as possible, as each one presents an additional opportunity to money. Marketing wants to create the “splash” associated with promoting a new product. Product developers are incentivized for innovating and introducing new features. None of these functions feel the pain of proliferating the number of SKUs that need to be supported to the extent that the Demand Planning function does. What’s more, none of these functions understand the impact on the portfolio like the Demand Planner, and in many cases will be unaware that their innovations are actually holding the business back. Somebody has to tell them.

For these reasons, the best new product strategies break out of the upper right quadrant and refocus the organization’s energy on addressing new solutions for existing customers and finding new customers. These activities represent true growth opportunities. Often, those individuals who had a hand in creating the initial success by providing new solutions to new customers in the early, disruptive stage of growth find it hardest to get out of the box and find these true growth opportunities.

Demand Planners can exert influence to increase the probability that the energy exerted to launch new products is rewarded with growth.

Demand Planners Must Reveal The Hidden costs of SKU Proliferation

If the organization is going to incur the costs and risks associated with supplying new products, it should ensure a good portion of these products provide true opportunities for growth instead of just replacing solutions that already exist. For sure, replacement products are often critical for defending market share, but such efforts cannot consume the entire product strategy. A valuable line of inquiry from the Demand Planning perspective might look like this: “If we develop product A, we can expect to incur startup costs of B, experience cannibalization of C sales, so we’d need to acquire D sales from new customers. Is that possible, or might we better off focusing on a different product?”

New product introductions are a necessity for any organization, but they create challenges for the demand planning, supply chain, and manufacturing functions. By increasing their involvement in choosing the right new products to develop, Demand Planners can exert influence to increase the probability that the energy exerted to launch new products is rewarded with growth instead of stagnation and decline.

My message to Product Developers is this: get Demand Planners involved in your development process if you want new products to truly drive growth. My message to Demand Planners is this: get yourselves involved because nobody in the organization has the bias-free insight your function holds.

The latest IBF benchmarking report is out, titled “Benchmarking New Product Forecasting and Planning.” It is available for download here.

Surveying 791 forecasters and demand planners around the globe, the report sheds light on the role of New Products today and how forecasts are prepared, including models used, review frequency and acceptable forecast errors. Crucially, the report reveals insight into the difficulties of launching and forecasting new products amidst an increasingly difficult business environment.

New Products Are Crucial To Maintaining Market Share

The report find that New Products are increasingly important, and comprise increasing percentages of companies’ offerings. The differences by industry are revealing: Technology/Electronics and Healthcare feature on top at 27% and 22% respectively. By comparison, Industrial products and Chemicals are at 14%. Food/Beverages remain low at 12%.

New Product Launches Are Not So New

Arguably the most revealing part of IBF’s Benchmarking report into new product launches is that the vast majority of New Products are variations on existing products, specifically Line Extensions, market Extensions or Product Extensions. Just 20% of ‘New Products’, are completely new. The majority are Line Extensions (30%) or Product Extensions (27%).
[bar group=”content”]

Companies Are Adapting to Intense Demand By Extending Lifespan

The report reinforces existing trends within the Fast Moving Consumer Goods (FMCG) industry, with the trend particularly pronounced in Tech and Electronics. To meet consumer demand for new products, companies are cutting lead time and cost by reworking or rebranding existing models. One example of this is Apple releasing new versions of the iPhone, having released 14 models between 2007 and 2016.

The benchmarking data supports the thesis that companies are repositioning products, lowering the price, reducing the content, and or selling in bulk in order to extend product lifespan.  This is how companies are overcoming the increasingly short product lifecycle environment.

Companies Creating Practical Solutions to Meet Demand, But Forecasting Gets Harder

IBF’s benchmarking data reveals that New Products face high forecast error, at 64%, compared to Product Improvement and Market Extension at 35% and 36% respectively.

Download the full report to see all the benchmarking data, including the forecasting models used in New Product forecasting, how new products are reviewed (independently or as part of the S&OP process), and how far ahead new products are forecast. See your company measures up.

Many organizations have processes that define how new products will be conceived, developed, designed, and brought to market.  Many organizations also have processes that define how they will integrate sales and operational plans to achieve their business objectives.  These processes may be at varying levels of formality and maturity.  However, it is not entirely uncommon for these two sets of processes to be at least out of alignment, and perhaps completely disjointed.

Sometimes we just stop talking!  More often we stop listening!  But why?

New Product Development (NPD) processes tend to be creative endeavors.  Often they focus on “what” is being developed and why the customer has this need, so that we can better meet or exceed those customer’s expectations – while “when,” “where,” and “how much” may receive less attention.  Without a “what,” the rest won’t matter!

Sales and Operations Planning (S&OP) processes, on the other hand, tend to be analytical endeavors.  Often they focus on those exact things that might be deemphasized in the NPD processes:  “when,” “where,” and “how much”!  This is done so that the customer’s demand can be met when and where they need it and so the organization can meet its objectives.  If the customer’s time & quantity expectations aren’t met or the organization cannot meet its objectives (which for most of us include financial needs, like profits) the rest won’t matter!

Of course, like the proverbial eye, which cannot say to the hand, “I have no need of you;” the NPD and S&OP processes serve difference constituencies and business needs: but they are in need of each other!  Additionally, they become even more powerful tools for an organization and its leaders when they work together!  The challenge is that they focus on different questions, with different key participants, to such a degree that in some organizations, they are nearly speaking a different language!

Sometimes a good “translator” can help.  This is where cross-pollination of teams can help.  There are also multiple tools that may be natural outputs from each of these processes that can strengthen information linkages.  These may include such tools as consolidated program schedules, multi-year product plans, and product lead time maps.  It is also critical to identify and then track early indicators of uncertainty.  These may include such things as project status, inventory positions, and market trend summaries.

However, by far the largest key to success is to keep communicating … and more importantly, to keep listening!  I’ll be speaking in more detail on this topic at the upcoming APICS & IBF Best of the Best S&OP Conference. Perhaps, we’ll see you there.

Dave Flory, CFPIM, CIRM, CPF, CSCP, PMP
Senior Scheduling Manager
JLG Industries, An Oshkosh Corporation Company