Is Consumer Demand Really That Unpredictable?
Photo: RetailWire

Through a special arrangement, what follows is a summary of an article from Retail Paradox, RSR Research’s weekly analysis on emerging issues facing retailers, presented here for discussion.

According to RSR’s benchmark report on omnichannel order profitability, unpredictable consumer demand is the top business challenge among retail respondents. After much consideration, I would offer that consumer demand isn’t really that unpredictable. Instead, retailers are just really bad at predicting it.

Part of the issue is one of “garbage in” — as in, “garbage in yields garbage out.” There seems to be a lot of garbage in the inputs retailers use to decide what to stock where. First, a lot of them start with last year’s plan, which means the bad assumptions made last year are automatically carried over into this year.

Then there is the level of granularity used. If you’re looking at demand at a category or even sub-category level you’re still going to miss a lot.

Another area where I see retailers ending up with garbage inputs is around estimating missed demand. Remember, what you sold is not “demand.” It’s just what you sold. There could have been plenty more demand out there to be captured, but you were out of stock.

I know that retailers are not good at estimating lost sales — all I have to do is listen to their shock and surprise when they start showing store inventory online. For example, for fashion retailers shipping from store, the impact was immediate. “We had no idea the depth of demand for some of these items” was a phrase I heard over and over again.

When you don’t have granularity — when you aren’t starting from clean inventory data and thus clean demand signals, and thus consistently under-estimate the demand that you’re losing out on — then yeah, I guess it could look to you like consumer demand is just outright unpredictable. That’s especially possible when you add the increased complexity of more channels, more inventory and demand crossing channels, and endless aisle assortments that make managing the granularity of your total assortment ever more difficult.

But the reality is, you must get better at it in order to win at retail. Unpredictable consumer demand is an excuse for bad processes and data — and a lame excuse at that.

BrainTrust

“There are both exogenous and endogenous factors driving bad forecasts.”

Paula Rosenblum

Co-founder, RSR Research


“Demand planning appears to have become trickier given the fickleness of shoppers and their lack of loyalty. “

Dave Wendland

Vice President, Strategic RelationsHamacher Resource Group


“The more data that is captured and analyzed, the better that new technologies can utilize machine learning to create “hyper-local” demand forecasts…”

Ralph Jacobson

Global Retail & CPG Sales Strategist, IBM

Discussion Questions

DISCUSSION QUESTIONS: Is consumer demand really unpredictable, or are retailers just bad at it? How had omnichannel made estimations more complex?

Poll

Has omnichannel retailing made estimating consumer demand more complex?

View Results

Loading ... Loading …

Leave a Reply

14 responses to “Is consumer demand really that unpredictable?”

  1. Paula Rosenblum Avatar
    Paula Rosenblum

    Well, omnichannel makes things problematic if forecast and planning systems are driven off of where demand was fulfilled, rather than where it is generated. This seems hard for retailers to grasp for reasons I don’t quite understand.

    Still, I wouldn’t go so far as to say that retailers are bad at estimating demand. Yes, I know I am part of RSR, but I have a slightly different take. Our earlier studies showed that Retail Winners were actually rather good at it and it was only under-performers who had a hard time adjusting to shifts and changes.

    The world of omnichannel has most certainly changed that and the data reflects it. It is not easy to predict, at this point, where a sale will be consummated. Odds are that with digital sales still increasing by double digit percents, online sales are under-estimated and with continued falling traffic in stores, store-based sales are over-estimated.

    And let’s not forget the additional chaotic influence of endless promotions.

    So there are both exogenous and endogenous factors driving bad forecasts. At least that’s my view.

  2. Peter Fader Avatar
    Peter Fader

    Indeed, as Nikki points out, when you don’t have granularity, demand forecasting can be challenging. But many retailers *do* have granularity or at least should be making more serious efforts to achieve it. And when you have such granularity, then forecasting is remarkably easy/accurate/actionable — if you understand the basic behavioral patterns and choose the appropriate methods to capture/project them. Here is one such example from my predictive analytics firm, Zodiac and I encourage retail analysts to learn more about these methods and their implications for strategic and tactical decision-making.

    1. Cynthia Holcomb Avatar
      Cynthia Holcomb

      Add to that … what if you knew your individual customers sensory and aesthetic preferences? No more guess work, wondering why something sold or did not sell specifically. I know as a Design Director to a large retailer’s private brands, many multi-million dollar decisions were based on anecdotal evidence in the moment of decision.

  3. Dave Wendland Avatar
    Dave Wendland

    Demand planning appears to have become trickier given the fickleness of shoppers and their lack of loyalty. Consumers are making purchases anytime, anywhere and from anyone. So is the anticipation of demand difficult? Yes. Do I expect it to get easier in the near-term? No.

  4. Adrian Weidmann Avatar
    Adrian Weidmann

    Retailers pride themselves at being great at operations. The truth is that they drink their own Kool-Aid. Because of the way Wall Street historically measures (and rewards) retailers’ success — same-store sales — retailers are constantly basing their forecasting on historic sales data. In today’s dynamic retail and shopper environment the insights need to be based on far more granular data points. To Nikki’s point, retailers need to start with a clean whiteboard when it comes to forecasting demand. The data does exist, it’s a matter of bringing those insights to light and knowing what to look for and having the resolve to act on what the numbers are telling you.

  5. Dr. Stephen Needel Avatar
    Dr. Stephen Needel

    Of course it’s relatively easy — but as Nikki et. al. say, it has to start from the bottom up with lots of granularity.

  6. Ralph Jacobson Avatar
    Ralph Jacobson

    Of course, retailers have an inherent disadvantage of forecasting consumer demand versus other industries. Hospitality, airlines, etc. all have consumers who make reservations, so anticipated incoming demand can be very accurately determined. The trouble is, retail shoppers typically don’t make reservations to go to a grocery store. One of the keys to leveraging that same kind of accuracy in retail is to ingest as much historical information about consumer demand as possible. The more data that is captured and analyzed, the better that new technologies can utilize machine learning to create “hyper-local” demand forecasts for retailers down to the individual store and individual SKU.

    Are many retailers using this capability? No, not yet. However, this capability does exist and it leverages unprecedented types of data from external sources including social chatter, news, local events (sports, etc.), weather, etc. and how all of these sources affect each other. The forecasting accuracy has shown significant improvements with real retailers as we speak.

    1. Kenneth Leung Avatar
      Kenneth Leung

      Exactly. Retailers have more channels for serving customers for input of data (browsing, buying) and delivering the experience (store, ship to customer, cross ship between stores) than airlines, hotels or restaurants. The data points are more complex and demand a lot more investment than many retailers are used to. Some retailers are better at doing it than others….

  7. Dan Raftery Avatar
    Dan Raftery

    Certainly the options available from the omnichannel environment make it more difficult to predict demand at any one point of retail. That’s why suppliers are drawn to Big Data. However for established products, retailers should be close enough 90 to 95 percent of the time. That means new products (which could arguably be most SKUs in the fashion world) are the wildcards. Some manufacturers use staged release or even crowdfunding feedback to help predict broader demand. And they still get it wrong sometimes. But if you’re wrong on the short side you can react, and so what? If you are wrong on the long side, well that’s a problem.

  8. Lee Kent Avatar
    Lee Kent

    Retailers tend to talk about Big Data only in the context of providing personalized shopping experiences but I like to remind them that the right usage of Big Data is how they can stay ahead of the customer. For example, using Big Data to listen for demand signals. This means capturing and interpreting both structured and unstructured data across the enterprise and across social.

    From a post I recently wrote: “Smart retailers are learning to leverage big data to sense and predict near real-time (or short-term) demand. New techniques that use pattern recognition coupled with the statistical forecasting algorithms are also being used to produce a more accurate picture of near real-time demand. This will provide retailers with information they need to make better decisions about where to place inventory. It is imperative to have the right products, at the right place and at the right time to remain competitive.”

    So yes it can be done but retailers need to learn how to listen for the signals.

    And that’s my 2 cents.

  9. Larry Negrich Avatar
    Larry Negrich

    The prediction of the demand is becoming more achievable but the operationalization still takes place in the physical world which is a challenge as trends in fashion happen faster and are wedged into shorter windows. Big data consumption is not yet at a stage that it is very helpful in the execution. Nice to know that a certain fashion item is resonating well by listening to social chatter, but being able to take action on those signals will require shortening of and precision in the supply chain.

  10. Roger Saunders Avatar
    Roger Saunders

    Consumer demand is very predictable.

    First, retailers have to start with the demand they are attempting to forecast. If it is how many navy blue tops in petite size will sell through at full price in the next 90 days, as some merchandisers might ask, there will be a bit more “art” than “science” in the forecast.

    If the retailer needs to know what their shoppers are planning on buying in the next 90 days within a merchandising line — say groceries — “science” kicks in. Having a scaleable database of shoppers — a couple thousand of Walmart, Target, Macy’s, etc. shoppers — and then listening to those shoppers purchasing intentions, retailers can capture forecasting correlations of .60 to .75.

    When developing a forecast, retailers will always get better forecasting leading to better planning by looking forward. A retailers windshield is bigger than their rear view mirror. There is a reason. It’s the same with the forecast.

    Edward Learmer, Director of UCLA Anderson Economics Forecasting, has pointed out, “Accurate forecasting is coming from comparing the data that are available today with historical points in the past. There are hardly any legitimate comparisons. And, THE RESULT IS THAT NOBODY IS DOING FORECASTING; WE’RE ALL DOING HUNCHES AT THIS POINT.”

    Better consumer forecasting has to start with the human element — what is the consumer’s variable spending plan in the 90 day to 120 day span. Follow that with integrating diverse datasets of actual data/insights of retailers along with a proven, competent data house.

    You’ll receive accurate forecasts. Just ask the right questions in a longitudinal basis. The answers will delight you.

  11. Dan Frechtling Avatar
    Dan Frechtling

    Tried and true time-series sales data succeeds for popular “big head” products in large regions. By itself it is insufficient for slower moving “long tail” items, especially in local markets.

    Sales data needs to be supplemented with causal non-sales data. For example, overlaying web search and social sentiment data from dealership trading areas predicts auto sales because people do a lot of research before buying cars. Adding weather data helps Tesco anticipate cold remedy sales.

    It takes a lot of trial and error to surface other demand predictors, but it can be done with practice.

  12. Peter Charness Avatar
    Peter Charness

    A number of good discussion points on granularity. While it’s mathematically interesting to try to get an accurate demand forecast by SKU/store/day, it’s probably not something that most retailers really need to get right. Demand plans need to be accurate enough to make a good merchandising decision, a purchase, a price change, a location adjustment. If a product is “one and done” single purchased to last 6 to 12 weeks, it really isn’t necessary to get a daily forecast of how it will sell — mildly interesting and minimally actionable. Boiling the ocean to get a “right” number that has no practical usage takes time that could be better used to work on getting the demand plan right where and at what level it counts.

14 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Paula Rosenblum
Paula Rosenblum
Active Member
7 years ago

Well, omnichannel makes things problematic if forecast and planning systems are driven off of where demand was fulfilled, rather than where it is generated. This seems hard for retailers to grasp for reasons I don’t quite understand.

Still, I wouldn’t go so far as to say that retailers are bad at estimating demand. Yes, I know I am part of RSR, but I have a slightly different take. Our earlier studies showed that Retail Winners were actually rather good at it and it was only under-performers who had a hard time adjusting to shifts and changes.

The world of omnichannel has most certainly changed that and the data reflects it. It is not easy to predict, at this point, where a sale will be consummated. Odds are that with digital sales still increasing by double digit percents, online sales are under-estimated and with continued falling traffic in stores, store-based sales are over-estimated.

And let’s not forget the additional chaotic influence of endless promotions.

So there are both exogenous and endogenous factors driving bad forecasts. At least that’s my view.

1
Peter Fader
Peter Fader
7 years ago

Indeed, as Nikki points out, when you don’t have granularity, demand forecasting can be challenging. But many retailers *do* have granularity or at least should be making more serious efforts to achieve it. And when you have such granularity, then forecasting is remarkably easy/accurate/actionable — if you understand the basic behavioral patterns and choose the appropriate methods to capture/project them. Here is one such example from my predictive analytics firm, Zodiac and I encourage retail analysts to learn more about these methods and their implications for strategic and tactical decision-making.

4
Cynthia Holcomb
Cynthia Holcomb
Member
Reply to  Peter Fader
7 years ago

Add to that … what if you knew your individual customers sensory and aesthetic preferences? No more guess work, wondering why something sold or did not sell specifically. I know as a Design Director to a large retailer’s private brands, many multi-million dollar decisions were based on anecdotal evidence in the moment of decision.

0
Dave Wendland
Dave Wendland
Member
7 years ago

Demand planning appears to have become trickier given the fickleness of shoppers and their lack of loyalty. Consumers are making purchases anytime, anywhere and from anyone. So is the anticipation of demand difficult? Yes. Do I expect it to get easier in the near-term? No.

0
Adrian Weidmann
Adrian Weidmann
Member
7 years ago

Retailers pride themselves at being great at operations. The truth is that they drink their own Kool-Aid. Because of the way Wall Street historically measures (and rewards) retailers’ success — same-store sales — retailers are constantly basing their forecasting on historic sales data. In today’s dynamic retail and shopper environment the insights need to be based on far more granular data points. To Nikki’s point, retailers need to start with a clean whiteboard when it comes to forecasting demand. The data does exist, it’s a matter of bringing those insights to light and knowing what to look for and having the resolve to act on what the numbers are telling you.

0
Dr. Stephen Needel
Dr. Stephen Needel
Member
7 years ago

Of course it’s relatively easy — but as Nikki et. al. say, it has to start from the bottom up with lots of granularity.

0
Ralph Jacobson
Ralph Jacobson
Member
7 years ago

Of course, retailers have an inherent disadvantage of forecasting consumer demand versus other industries. Hospitality, airlines, etc. all have consumers who make reservations, so anticipated incoming demand can be very accurately determined. The trouble is, retail shoppers typically don’t make reservations to go to a grocery store. One of the keys to leveraging that same kind of accuracy in retail is to ingest as much historical information about consumer demand as possible. The more data that is captured and analyzed, the better that new technologies can utilize machine learning to create “hyper-local” demand forecasts for retailers down to the individual store and individual SKU.

Are many retailers using this capability? No, not yet. However, this capability does exist and it leverages unprecedented types of data from external sources including social chatter, news, local events (sports, etc.), weather, etc. and how all of these sources affect each other. The forecasting accuracy has shown significant improvements with real retailers as we speak.

0
Kenneth Leung
Kenneth Leung
Member
Reply to  Ralph Jacobson
7 years ago

Exactly. Retailers have more channels for serving customers for input of data (browsing, buying) and delivering the experience (store, ship to customer, cross ship between stores) than airlines, hotels or restaurants. The data points are more complex and demand a lot more investment than many retailers are used to. Some retailers are better at doing it than others….

0
Dan Raftery
Dan Raftery
7 years ago

Certainly the options available from the omnichannel environment make it more difficult to predict demand at any one point of retail. That’s why suppliers are drawn to Big Data. However for established products, retailers should be close enough 90 to 95 percent of the time. That means new products (which could arguably be most SKUs in the fashion world) are the wildcards. Some manufacturers use staged release or even crowdfunding feedback to help predict broader demand. And they still get it wrong sometimes. But if you’re wrong on the short side you can react, and so what? If you are wrong on the long side, well that’s a problem.

0
Lee Kent
Lee Kent
Member
7 years ago

Retailers tend to talk about Big Data only in the context of providing personalized shopping experiences but I like to remind them that the right usage of Big Data is how they can stay ahead of the customer. For example, using Big Data to listen for demand signals. This means capturing and interpreting both structured and unstructured data across the enterprise and across social.

From a post I recently wrote: “Smart retailers are learning to leverage big data to sense and predict near real-time (or short-term) demand. New techniques that use pattern recognition coupled with the statistical forecasting algorithms are also being used to produce a more accurate picture of near real-time demand. This will provide retailers with information they need to make better decisions about where to place inventory. It is imperative to have the right products, at the right place and at the right time to remain competitive.”

So yes it can be done but retailers need to learn how to listen for the signals.

And that’s my 2 cents.

0
Larry Negrich
Larry Negrich
7 years ago

The prediction of the demand is becoming more achievable but the operationalization still takes place in the physical world which is a challenge as trends in fashion happen faster and are wedged into shorter windows. Big data consumption is not yet at a stage that it is very helpful in the execution. Nice to know that a certain fashion item is resonating well by listening to social chatter, but being able to take action on those signals will require shortening of and precision in the supply chain.

0
Roger Saunders
Roger Saunders
7 years ago

Consumer demand is very predictable.

First, retailers have to start with the demand they are attempting to forecast. If it is how many navy blue tops in petite size will sell through at full price in the next 90 days, as some merchandisers might ask, there will be a bit more “art” than “science” in the forecast.

If the retailer needs to know what their shoppers are planning on buying in the next 90 days within a merchandising line — say groceries — “science” kicks in. Having a scaleable database of shoppers — a couple thousand of Walmart, Target, Macy’s, etc. shoppers — and then listening to those shoppers purchasing intentions, retailers can capture forecasting correlations of .60 to .75.

When developing a forecast, retailers will always get better forecasting leading to better planning by looking forward. A retailers windshield is bigger than their rear view mirror. There is a reason. It’s the same with the forecast.

Edward Learmer, Director of UCLA Anderson Economics Forecasting, has pointed out, “Accurate forecasting is coming from comparing the data that are available today with historical points in the past. There are hardly any legitimate comparisons. And, THE RESULT IS THAT NOBODY IS DOING FORECASTING; WE’RE ALL DOING HUNCHES AT THIS POINT.”

Better consumer forecasting has to start with the human element — what is the consumer’s variable spending plan in the 90 day to 120 day span. Follow that with integrating diverse datasets of actual data/insights of retailers along with a proven, competent data house.

You’ll receive accurate forecasts. Just ask the right questions in a longitudinal basis. The answers will delight you.

0
Dan Frechtling
Dan Frechtling
7 years ago

Tried and true time-series sales data succeeds for popular “big head” products in large regions. By itself it is insufficient for slower moving “long tail” items, especially in local markets.

Sales data needs to be supplemented with causal non-sales data. For example, overlaying web search and social sentiment data from dealership trading areas predicts auto sales because people do a lot of research before buying cars. Adding weather data helps Tesco anticipate cold remedy sales.

It takes a lot of trial and error to surface other demand predictors, but it can be done with practice.

0
Peter Charness
Peter Charness
Active Member
7 years ago

A number of good discussion points on granularity. While it’s mathematically interesting to try to get an accurate demand forecast by SKU/store/day, it’s probably not something that most retailers really need to get right. Demand plans need to be accurate enough to make a good merchandising decision, a purchase, a price change, a location adjustment. If a product is “one and done” single purchased to last 6 to 12 weeks, it really isn’t necessary to get a daily forecast of how it will sell — mildly interesting and minimally actionable. Boiling the ocean to get a “right” number that has no practical usage takes time that could be better used to work on getting the demand plan right where and at what level it counts.

0