As forecasters, we rely on data to look at the past and use models to provide insight into the future – this is not one of those times. The following, rather, is what we refer to in the planning field as a qualitative forecast – unscientific and with more than a pinch of gut feel and intuition. Here I project the trends for 2020 in demand planning, S&OP and related fields.

Robots Will Finally Come For Our Jobs

2020 just may be the year a robot or AI comes for your job – or at least part of it. Many think of automation as a robot on a manufacturing line that mechanizes a repetitive task and, in the process, makes many blue collar roles obsolete. With AI we are now seeing automation coming into white collar roles whereby technology is professional activities/role such as interpreting and translating, processing transactions, dealing with data, replying to emails and even parts of the legal profession.

Automation will make our jobs better by removing the boring stuff and giving us more time to focus on value-added activities.

Repetitive tasks are not the preserve of the low paid worker – think of all the repetitive tasks that you do in planning. Up to 45% of the activities we do can be automated. The good news is there is still 55% of our jobs that a robot can’t do, so us planners being rendered completely useless may have to wait another decade. On a serious note, automation and machine learning will make our jobs better – by removing the boring stuff and giving us more time to focus on value-added activities. We, by using creativity to add value rather than performing routine processes to add value, become more valuable to our organizations in the process.

While AI and automation are not new for 2020, they will see massive growth this year. With the shrinking workforce and the pressure companies face to protect their bottom line, automation will be a key initiative in the years to come and you will see more companies adopting it.

We Will  Say Goodbye To Cloud Computing & Say Hello To Edge Computing

One of the lesser talked about technologies will take data collection and processing and take it away from your computer and closer to the source. With the IoT becoming more widespread and the speed of data gathering and processing becoming more important, edge computing will gain traction. Edge computing is designed to help solve some of the problems associated with handling Big Data and bypass the latency caused by cloud computing. Edge computing can be used to process time-sensitive data in remote locations with limited or no connectivity to a centralized location.

Planning technology will see a revolution – becoming more open source and decentralized.

So, if you have not yet made the connection, this not only means it can increase the use of IoT and new data sources, but further enables predictive analytics. We will not only cut ties to our personal computer but also cut ties to legacy ERP systems and planning suites that require all processing to be done inside their systems. Planning technology, especially surrounding predictive analytics and forecasting, will see a revolution – becoming more open source and decentralized.

With that in mind, this year I also see software providers in this space changing their business models and rethinking what technology they’re offering and how it’ll be made available.

Data Analytics Will Make Roles Converge

With the growth of technology and the increased speed of business, many roles will become blurred in the year (or years) to come. Marketing teams will be synonymous with digital teams, where marketers are just as comfortable thinking about data as they are customers. Data scientists will become more like demand planners and demand planners will learn to be more like data scientists. And all areas of a business will begin to consume more data and be more analytically driven, thereby making it harder to distinguish roles.

While some say that data is the “new oil”, the real-life blood of a business is insights.

While some say that data is the “new oil”, the real-life blood of a business is insights. Organizations need to be flatter in order to process information faster unless they want to risk nimble competitors seizing business opportunities. Flatter can mean the centralizing or amalgamating of roles. Companies will see business forecasting and planning roles grow, or augment other roles such as finance, marketing and other functions to help enable better insights.

We Will Finally Clean Up Our Data Swamps

As we ring in a new year, the issue of data quality and quantity is more important than ever. As we try to bring in all sorts of new data from different sources, our data lakes are starting to look more like data swamps. This has been a recurring issue for years, but I finally have hope that we can start to solve the problems of data integrity, data governance, data security, and just plain messy data.

As organizations need more data for planning, I predict the trends will turn from data collection to data analytics as the majority of companies quickly realize that their predictive analytics adoption must be equally met with solid efforts on a next generation data environment as well. Business Intelligence software is making a revival right now, offering more advanced data analytics capabilities than ever before, along with renewed data visualization techniques. Traditional systems are understanding the needs as well, and we are seeing more of a focus on the inputs to help create better outputs. We will also see it on the people side with designated roles like data architects or data interpreters – these people will become increasingly important to help drive better data and information.

2020 Will Be The Year Of Predictive Analytics

Those customer reviews and comments on websites, all of that social media data, metadata on consumers and macro data on markets – all of this is bound to be useful for something. There is an arms race to leverage this new data and use Machine learning and Artificial Intelligence to glean better insights. The sheer volume and complexity of today’s data are challenging enough, but top organizations in 2020 and the years to come will need to turn this data into useful insights quicker to support faster and better decision making. Therein lies the competitive advantage of 2020 and beyond. To this end, demand planning and predictive analytics will become the top priority and main investment for forward-thinking companies in 2020, and for companies as a whole over the next few years.

Looking At Historical Shipments Will Not Be Enough

2020 just may be the beginning of the predictive analytics revolution for demand planners, where predictive analytics becomes more of central focus from a people, process and technology perspective. In today’s business environment, changes in the marketplace are swift, sudden, and may not follow the historical pattern. Just looking at historic shipments will no longer be enough in the next decade and you need and tell the whole picture. Instead planners will begin to look even more at patterns of consumer behaviors and other external attributes to not only predict the sale, but understand why it was purchased it in the first place.

IBF spoke to Dr. Eric Siegel, author of the best-selling book Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die, and host of the Dr. Data Show on YouTube. He reveals why predictive analytics has become a critical best practice for all businesses and how – when misused – it can further marginalize society’s most vulnerable groups.

What is predictive analytics?

Let me give three definitions. The first is that it’s the business application of machine learning. The second, more common functional definition, is that it’s technology that learns from data to make predictions about an individual person or case in order to improve company performance. The third definition—where predictive analytics meets forecasting— is that it’s the deployment of machine learning methods such as decision trees, log-linear regression, ensemble models and neural networks. These methods serve to predict demand. When applied for forecasting, rather than generating a prediction for an individual customer, corporate client, or transaction, which is what predictive analytics is usually employed to do, it simply provides a forecast for the item and time range in question.

How does predictive analytics work?

Imagine you have rows of data available from which to learn. In the case of a classical business application, each row corresponds to a customer and contains the outcome or behavior about the customer you’d like to predict—such as whether they committed fraud, whether they canceled their subscription, whether or not they turn out to be a good credit risk, whether they defaulted on their debt, or whatever outcome is relevant to your business.

In the case of forecasting, each row includes or summarizes everything you knew up to this point in time about a particular SKU or whatever you’re trying to predict demand for, along with the outcome, i.e., what the actual demand ended up being over the next week or next month. These rows compose your training data from which correlations are discovered between the data points you hold about the product and the outcome.

Are machine learning and predictive analytics synonyms?

It depends on the context, but, when you’re talking about these kinds of business problems, they’re entirely synonymous. A lot of confusion surrounding this technology is due to nomenclature. A few years ago, the industrial world started using the word machine learning. Before that, the term was only common within academia and research and development.

When it comes to this technology and demand forecasting, you can just as well call it machine learning as predictive analytics. Whether we’re predicting the likelihood of somebody defaulting on a loan or predicting demand for a product, the same exact core analytical methods apply.

To complicate things further, what about prescriptive analytics?

Prescriptive analytics is a superfluous term, which has never been agreed on as far as its definition. The way it’s usually used is to imply that predictive analytics is not enough and now you need additional technology in order to translate what you’ve predicted into actual action to be taken. But predictive analytics has all along been intended to directly inform actions to be taken such as whether to make marketing contact, whether to audit for fraud, or approving someone for a credit card application. The term prescriptive analytics implies there’s an entirely new field or technology. There isn’t. Predictive analytics as a field already includes field deployment, i.e., the translation into action.

How does predictive analytics differ to more traditional statistical modeling?

Machine learning is statistical modeling. It’s just that we don’t use the word statistics because it sounds boring and it kind of undersells it because the core methods have advanced so much since the term statistical modeling originated. Predictive analytics is just the use of those methods for business applications. They all make use of historical learning cases, i.e., data, and they all find patterns that map from the things we do know to the things that we’d like to predict.

What impact has predictive analytics had on businesses over the last few years?

Predictive analytics has become a critical best practice for all businesses that operate at scale. It empowers businesses to execute on mass-scale operations more effectively. Business is a numbers game. Mass marketing, for example, is mostly just junk mail. Investigating fraud is mostly looking at transactions that are not fraud. Most resources and time in that sense are intrinsically wasteful.

The question is, how can you make all these mass operations more effective? By predicting. If you can predict for each individual case, that is, put a probability on it, then you know whether or not to risk spending $2 to send somebody a marketing brochure, or the risk of not auditing somebody for fraud. So predictive analytics generate probabilities, which allows us to manage risk. If you take a marketing campaign and improve its return on investment by a factor of three, that’s a valuable business outcome.

How can companies get started in predictive analytics?

E: There needs to be a champion at the company who becomes familiar with the field by reading or undergoing some preliminary training. Then it’s about defining the right pilot project. Where’s the lowest hanging fruit, the greatest opportunity? What’s the large-scale operation that occurs repeatedly— whether it’s a forecasting function or marketing or what have you—that could stand to benefit? Then you just need to define the scope for the project, do a preliminary analysis of the data, and, usually, engage with an external resource like professional services to get that first project off the ground.

What is big data and why is it valuable for businesses?

Big data and data science are simply umbrella terms that mean we have a lot of data and let’s do something useful with them. Big data did originally mean data that is too big for traditional database technology, but that’s not how it’s commonly used now. These days, both big data and data science allude more to a culture than any particular technology or value proposition. When we talk about using big data, we’re just saying, “We’ve got smart people who are working on using data to derive value”.

What kind of data do companies typically hold that could be used for predictive analytics?

The good news is that anything you do repeatedly is the exact thing that creates the data you need to improve that process. In marketing, it’s simple—you’ve got all the demographic profile and behavioral data about that customer, like what they’ve bought up to that moment and other data points. For forecasting, it’s very similar. Everything that’s been going on with the sales of this particular product up to this moment in time, as well as other external factors which you believe could influence demand, can be leveraged by predictive analytics.

One of the most famous cases of a company benefitting from predictive analytics was when Target marketed to women they identified as likely to be pregnant. How did they figure that out?

That’s an unusual case because normally in marketing applications, you’re predicting for a particular product or a set of products. This is a little different because they’re trying to predict something about the current state of the customer rather than their future behavior. If we think about training data with all the things we know about the customer, Target shored that up with data from customers who voluntarily told Target that they were pregnant. Then the analytics process ends up just being the same thing as for any of these other problems we’ve been discussing. They analyzed all the demographic and profile information about the individual customer and the purchases that they’ve been making, especially more recently, to identify which customers are likely to be pregnant. Then they market appropriate products to that group.

When it comes to implementing predictive analytics, is the challenge not so much building or accessing the technology itself but building a culture around it?

Yes. That’s a parallel challenge and is often the greater challenge. Predictive analytics needs to be seen as an organizational process and the business value needs to be the leading motivator. The actual deployment has to be an integral part of the project from inception. To that end, you need to lead with that value proposition. Let’s say we’re selling this idea internally to marketing. We tell them we’re going to market to only the top 30% predicted most likely to buy and in so doing, we’ll save 70% on marketing costs and here’s our estimated ROI.

These metrics are why we’re doing it—you don’t lead with machine learning or the analytical methods, you lead with a value proposition, the functional purpose of the project. Then, to the degree they need to how the underlying technologies work, you explain the technical aspects that we’ve been discussing.

Do businesses need Chief Data Officers?

I’m actually agnostic about that. It certainly can’t hurt in terms of improving visibility of the analytics function and encouraging a general culture of using data and finding ways to do it. Of course, it depends on the individual organization. The very first Chief Data Officer was at Yahoo. It turns out the job title of Chief Data Officer was first suggested as a joke and everybody in the room laughed. But it stuck and it became a whole new thing.

In your book you say it’s impossible to predict with high accuracy. In business, what kind of accuracy do you need?

Well, accuracy is not the right measure. Rather, you want measures that tell you how much better than guessing your predictions are. As previously mentioned, business is a numbers game.  You want to tip the odds in your favor by predicting better than guessing. That can have a dramatic effect even if it’s not “accurate” in the conventional sense.

For example, I can identify the 20% of customers that are three times more likely than average to buy, but it might be that those people overall have just a 6% chance of buying. I don’t have high confidence that any one customer’s definitely going to make a purchase. But knowing that group is three times more likely to buy than the average tells you that the model has learned something. Instead of prediction accuracy, we should ideally measure the value in business terms like return on investment or overall profit of the project or marketing campaign. If you can translate it into those terms, which you often can, that’s the number that matters, not the predictive performance of the underlying model.

There are now off-the-shelf predictive analytics/machine learning tools that are very cheap. Are these black box systems and can they be trusted?

Most of the software products out there are not black box. There are well-known standard machine learning methods such as the decision trees and neural networks methods we have mentioned. The software products have lots of different bells and whistles and variations to them, but more or less they use known methods or algorithms, and they don’t hide how they work. There are exceptions. There are some machine learning products that build branding around having a “secret special sauce.”

For more technical users and experts, including myself, that creates suspicion and raises questions regarding credibility. After all, if you want to sell a product to experts in the field, it doesn’t make sense to keep the core analytics a secret and market the product as some magical tool. Transparency, as well as cost, user interface, and how well it integrates with existing operational systems and databases should be the selling points.

Many people are still forecasting in Excel when these tools are now easily accessible. What do you think is stopping widespread adoption?

There’s a lot of complexity in moving from Excel to proper machine learning. It’s a completely new endeavor and you have to get together training data, apply methods that may be new to you, and then be able to interpret the outcome, results and performance. It’s a big project. The thing is, there has to be a business reason or incentive to do it. How much ROI would there be if your forecasts were 10% better, for example, and how much is that worth to you?

What is the persuasion paradox and why is it important?

The paradox is that, although we can’t perceive or record individual cases of causation or influence, we can predict it. You can’t conclusively observe persuasion, even in retrospect. If you look at a customer, why did they buy the product? Was it because of your marketing initiative? You can never be entirely sure because they might have bought the product anyway. But we can predict the likelihood of causation over large numbers of individuals.

On the other hand, in forecasting, observing influence is generally is not a concern. We just want to know how many widgets are going to be sold next week. We don’t care why we’re going to sell a lot of widgets next week. We just want an accurate estimate of sales.

What are some examples of unethical uses of data?

The first one that comes to mind is using a protected class such as the race, gender, or religion of an individual in order to derive a consequential decision such as whether they get approved for a loan or sentencing in prison. Let’s imagine that a manager is interviewing a candidate for a job, and they use predictive analytics to gauge the candidate’s likelihood of success in the position. The manager might reject the candidate based on the results provided by the system. When the candidate asks the manager for the basis of their decision, they would have to say, “Well, for one thing, our predictive model penalized you by seven points because you’re black.” Normally, protected classes are disallowed as direct inputs, but there are experts, including university professors involved in the legal justice system, campaigning to purposefully introduce then as direct inputs. The risk is real.

What excites you about predictive analytics?

I’m a former academic and I think that technology that learns from examples and, therefore, from experience to draw generalizations that hold in future cases is the most interesting kind of any science and technology. It’s the most exciting science period. The fact that it naturally ends up being valuable only makes it that much more exciting.

To read the full interview with Dr. Eric Siegel, download the latest issue of the Journal of Business Forecasting. When you become an IBF member, you’ll get the Journal delivered to your doorstep quarterly. Get your IBF membership.

In supply chain and operations, raw materials are substances that are used in the  manufacturing of goods. They are the commodities to be transformed into another state that will either be used or sold. For algorithms or predictive models, data is the raw material that every insight begins with.

A piece of data, or collection of it, can help drive a predictive analytics process and uncover insights. Data are the building blocks and inputs, and without data it is nearly impossible to find answers and make decisions. That said data is not the destination. Data is not a decision. And, while data may take on many forms and be used for many things, data by itself is not insight.

Data Is Information In Its Raw Form

Information is a collection of data points that we can use to understand something about the thing being measured.

Insight is gained by analyzing data and information to understand what is going on with a particular thing or situation. The insight can then be used to make better business decisions.

Data on its own is meaningless. It is just a raw material that needs to be transformed, analyzed, and turned into understanding.

Data on its own is meaningless. It is just a raw material that needs to be transformed, analyzed, turned into understanding and shared by people with the skills, training and commitment to do so. At the same time, predictive modeling or any business insight without data is equally as meaningless. No matter how skilled you are, or how good your model is, it is like trying to produce a finished product without the proper parts.

There’s no arguing the power of data in today’s business landscape. Businesses are analyzing a seemingly endless array of data sources in order to glean insights into just about every activity – both inside their businesses and out. Right now, it seems that enterprises cannot get their hands on enough data for analysis purposes. They are looking at multiple sources and forms of data to collect and use to learn more about customers and markets, and predict how they will behave.

What Are The Different Types Of Data?

We can think about data in terms of how it is organized, as well as the source. Data may be either structured or unstructured and the source can be either internal or external.

Knowing what types of data you have, and where they come from, is crucial in the age of Big Data and analytics.

Internal Sources: Internal sources of data are those which are procured and consolidated from different branches within your organization. Examples include: purchase orders, internal transactions, marketing information, loyalty card information, information collected by websites or transactional systems owned by the company, and any other internal source that collects information about your customers.

Before you begin to look for external sources, it’s critical to ensure that all of a business’s internal data sources are mined, analyzed and leveraged for the good of the company. While external data can offer a range of benefits (that we’ll get into later), internal data sources are typically easier and quicker to collect, and can be more relevant for the company’s own purposes and insights.

External Sources: External sources of data are those which are procured, collected, or originate outside of the organization. Examples include external POS or inventory data from a retail partner, paid third party information, demographic and government or other external site data, web crawlers, macroeconomic data, and any other external source that collects information about your customers. Collection of external data may be difficult because the data has much greater variety and the sources are much more numerous.

Structured Data: Is both highly-organized and easy to digest and generally refers to data that has a defined length and format. It is sometimes thought of as more traditional data which may include names, numbers, and information that is easily formatted in columns or rows.  Structured data is largely managed with legacy analytics solutions given its already-organized nature. It may be collected, processed, manipulated and analyzed using traditional relational databases. Before the era of big data and new, emerging data sources, structured data was what organizations used to make business decisions.

Unstructured Data: Does not have an easily definable structure and is unorganized and raw, and typically isn’t a good fit for a mainstream relational database. It is basically the opposite of structured and includes all other data generated through a variety of human activities. Common examples are comments on web pages, word processing documents, videos, photos, audio files, presentations, and many other kinds of files that do not fit into the columns and rows of an excel spreadsheet.

These new data sources are made up largely of streaming data coming from social media platforms, mobile applications, location services, and Internet of Things technologies. Since the diversity among unstructured data sources is so prevalent, businesses have much more trouble managing it than they do with traditional structured data. As a result, companies are being challenged in a way they weren’t before and are having to get creative in order to pull relevant data for analytics.

Don’t Get Left Behind When It Comes To Data

You may believe that only super large companies with massive funding or technology are implementing data analytics and pushing the limits of the types of data that are collected.  While 90% or more of data today is internal structured data, it is important to understand that 90% plus of the data ‘out there’ (external data) is unstructured.

It is important to understand that 90% plus of external data is unstructured.

With this increase in data and the need to be competitive, along with the expansion of data storage capabilities and data analytics tools, the playing field has leveled. While data is not insights, new forms and types of data have given rise to demand for newer insights and this focus on data has embedded itself into the culture of more and more businesses.

Eric will reveal  how to update your S&OP process to incorporate predictive analytics to adapt to the changing retail landscape at IBF’s Business Planning, Forecasting & S&OP Conferences in Orlando (Oct 20-23) and Amsterdam (Nov 20-22). Join Eric and a host of forecasting, planning and analytics leaders for unparalleled learning and networking.