Big Data is done, put a fork in it

Big Data is done, put a fork in it

Rick Ferguson, CLMP

Through a special arrangement, presented here for discussion is a summary of a current article from The Wise Marketer, a website and newsletter serving the global loyalty industry.

About a decade ago, the phrase “data is the new oil” swept the globe as the twin corporate power centers of IT and marketing realized their companies had more data than they knew what to do with. “Drinking from the fire hose” became a metaphor for the struggles of extracting actionable insights from data.

About five years ago, the usual suspects in IT consulting and cloud-based analytics began to trumpet the phrase “Big Data” to sell into companies hoping to extract that resource.

Now, as Slate’s Will Oremus points out, the phrase Big Data has become passé — in part because we just call it data now, and in part because the rush to rely solely on data for business decision-making has often revealed the limitations of data-based decisions.

Our over-confidence in these tools often hinders our ability to see the forest for the trees. We often fall victim to what data scientist Shane Brennan calls the “Ten Fallacies of Data Science.” For a variety of reasons, the data may be inaccessible, indecipherable, outdated, lacking enough granularity for analysis or prohibitively expensive to get or have processed, Mr. Brennan contends. Often, a marketer’s lack of understanding or pre-conceived notions leads to poor decisions.

“Garbage in/garbage out” results in marketing campaigns or loyalty offers that are ineffective or even harmful to your business.

Should all data analysis be eschewed in favor of, say, gut feel, astrological charts or throwing darts at a board? Not at all. Data analytics is being successfully used to fuel personalization, construct relevant offers and build differentiated experiences.

But by understanding the limits of data science, your efforts on data can best correlate with current customer value or be most predictive of future customer value. Call it “small data” or “customer-centric-data,” or just continue to call it data. If customer behavior shifts in a profitable direction because of your analysis, then you’ll know you’re on the right track. Be diligent, question your assumptions and be aware of your biases. Big Data may be over, but data science, like any scientific pursuit, is forever.

BrainTrust

"A stake to the heart of the term Big Data is not going to kill the essence of managing by information."

Lyle Bunn (Ph.D. Hon)

Strategy Architect – Digital Place-based Media


"Can anybody honestly say that “data science” is dead, even if “Big Data” is a cliche?"

Dick Seesel

Principal, Retailing In Focus LLC


"The label “Big Data” doesn’t really matter — it’s all about the analytics maturity of a retailer and how they are organized."

Gib Bassett

Director, Solutions Marketing with Alteryx


Discussion Questions

DISCUSSION QUESTIONS: Should “Big Data” be retired as a progressive movement in marketing? Do retailers and brands generally understand the limitations of data science within their organizations? What advice would you have for marketers attempting to best leverage the data at their disposal?

Poll

36 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Bob Phibbs
Trusted Member
6 years ago

This is semantics. We use Big Data now to refer to AI, machine learning and predictive analytics in one easy term. Watson is much more than just “data.”

Lee Kent
Lee Kent
Member
Reply to  Bob Phibbs
6 years ago

Yes, Bob, it is semantics. Data in retail will always be fuel behind almost everything we do. It’s how smart we get in using it. For my 2 cents.

Paula Rosenblum
Noble Member
6 years ago

I don’t think “Big Data” was ever well defined, and the systems that might actually make use of the insights were not in place in retail operations.

And then there was omnichannel, which overtook corporate IT spending (along with security).

I think the biggest issue with retailers and brands is not that they don’t understand the limitation of data science — it’s more that the data scientists have gone into financial services, where they can make real money. It’s on the vendors to provide the science and make it easily digestible for the retailers, et al.

Dick Seesel
Trusted Member
6 years ago

If you read yesterday’s Wall Street Journal interview with the chairman of Fast Retailing (Uniqlo), you might have seen this perceptive comment: “Data would never substitute the merchant. How do you interpret the data? That’s the merchant’s skill set. You need to uncover the insight that is buried in the data and the merchants need to uncover it. Even if you employ artificial intelligence to help you, the numbers [don’t tell] the future.”

His point is well taken: No matter how much importance a retail organization places on its ability to extract data from its transactions, the information means nothing if it can’t be turned into action — and some of that decision-making rests on instinct and experience.

That being said, declaring that “Big Data is dead” is an overreaction. The phrase itself may be overused, but data science is alive and well in the interest of smarter merchandising decisions, loyalty programs and so forth. Would Amazon be where it is today without groundbreaking use of data to develop its predictive technology? I don’t think so.

Dick Seesel
Trusted Member
Reply to  Dick Seesel
6 years ago

And just to add to my comment: I teach an undergrad retailing class at UW-Milwaukee and one of my guest speakers this term is a data scientist whose firm works with multiple omnichannel and e-commerce retailers. He made the point that 90 percent of the world’s data has been generated over the last two years, and the trend toward accelerating data generation continues.

Can anybody honestly say that “data science” is dead, even if “Big Data” is a cliche?

Cynthia Holcomb
Reply to  Dick Seesel
6 years ago

Good points. I agree with Mr. Yani’s comments as well. What is missing? Data scientists are working to interpret data on a subject requiring deep knowledge only an experienced merchant has earned. Thus the disconnect. To cross this chasm will require specialized merchants to work hand in hand with data scientists.

Dr. Stephen Needel
Active Member
6 years ago

Back in 2013 I delivered a paper at ESOMAR’s congress on why Big Data was not such a big deal — this echos a lot of what I said at the time. I’m not sure this was ever a “progressive” movement. We were sold a bill of goods on possibilities that turned out not to exist. I’ve always maintained that marketers and retailers need to focus on the questions they would like answered and let their researchers find the best way to answer them. We shouldn’t put the cart before the horse, trying to leverage the data we have, just to try and leverage it.

Ian Percy
Member
Reply to  Dr. Stephen Needel
6 years ago

Excellent response, Stephen!

Sterling Hawkins
Reply to  Dr. Stephen Needel
6 years ago

Well said. Data and insights for the sake of data and insights doesn’t net any impact on the business. It’s what is done as a result that makes the difference. Having the business look at what questions to ask connected to actions they could take is definitely the best approach.

Charles Dimov
Member
6 years ago

Thanks for retiring the term. Big Data is probably the most overused and abused term in the business world. I have heard VPs talk about the critical importance of it, and how every firm needs it, but they did not really understand much about the hows, whys, tactics or implications.

Data is messy. The important thing is that executives understand that you still have to make a decision on the data you have at hand, and it still will often come down to a judgment call. The worst thing you can do is to fall into an analysis-paralysis vortex. Don’t let this happen to you! There is a point where you have analyzed the data enough and it is time to make a judgment call. By the time you have full clean data, the conclusion is usually obvious. Your competitors have already long since taken action while you sit pondering what happened — in the dust.

Ralph Jacobson
Member
6 years ago

I’m just a bit concerned about stating that “Big Data” should be retired without the intentional, highlighted hand-off to another term, like, “GINORMOUS DATA.” Retailer and CPG brands haven’t even scratched the surface of capturing and deriving insights from the more than 80 percent of all data which is “dark” and invisible to current systems in these enterprises. Capturing this data is the first step in acknowledging what the current limitations of data science really are. From there, there’s a ton of work to do still. Let’s get going!

Ian Percy
Member
Reply to  Ralph Jacobson
6 years ago

You trigger the thought that “information” and “insight” are totally different things though often confused.

Kiri Masters
6 years ago

I look to arguably the best example of a company who obtains, compiles, and uses data to its full advantage: Amazon. As a core leadership principle, Amazon uses data to take huge bets and lead decisions. And when the data doesn’t support an idea, the venture is quickly killed and moves on. It seems so much part of the DNA of the company that giving it a special title is pointless. It’s just the way Amazon does business.

Peter Fader
6 years ago

I’m as strong an advocate as you can get for “Small Data,” i.e., squeezing as much value/insight as possible about simple transaction log data before turning to more elaborate “Big Data” schemes. But that doesn’t mean that Big Data is done — quite to the contrary, the real era of Big Data hasn’t even started yet. Once we master the art and science of Small Data, we’ll finally be in a position to start to properly leverage Big Data, in ways that most retailers can’t even imagine today.

It’s a lot like CRM: people were writing off CRM systems (literally and figuratively) 15 years ago because they weren’t yet in a position to make strategic decisions that required it. But today we know that CRM is the bare minimum pf what you need for customer-centric success. Big Data will have a similar resurgence in a few years.

But first retailers really need to get started with the Small Data revolution that is right at their doorstep and ready to happen …

J. Peter Deeb
J. Peter Deeb
6 years ago

The new term should be “Smart Data.” Companies that have learned how best to efficiently mine their data for better sales and profits, customer retention, better logistics operations etc. are already on to this “smart” usage. There are many examples — from dunnhumby to Watson — of businesses that understand how to utilize their information for results.

Brandon Rael
Active Member
6 years ago

Similar to the apprehension retail industry experts have about the term “omnichannel,” Big Data now shares a similar distinction.

Big Data is a term that was intended to simplify the complexity of all of your data sources, consolidate it, and present it back in a form that is both consumable and able to be operationalized.

Essentially we should retire this term, as it overflows into the predictive analytics, augmented intelligence, cognitive data and artificial intelligence arenas, which are far better defined and are undergoing significant evolutions.

Lyle Bunn (Ph.D. Hon)
Lyle Bunn (Ph.D. Hon)
6 years ago

A stake to the heart of the term Big Data is not going to kill the essence of managing by information. Let it morph into a descriptor of its business value rather than what it is, which has driven investment and has been the enterprise call to action, through which abilities and incapabilities are being continuously revealed.

Dave Nixon
6 years ago

Big Data as a term might be passe’ but the use of Big Data is only going to grow with the adoption of machine learning tools and AI (coupled with prebuilt retail data models) to harvest greater volumes of insights from growing volumes of data that are bigger than what data scientists could ever dig through for non-critical or “cold” data needs. THAT is where the term Big Data makes sense. The key is to have a clear data strategy and architecture that allows for smaller and more critical data sets to be analyzed by the data science teams within retailers for better results.

Zel Bianco
Zel Bianco
Active Member
6 years ago

I have been known to say “stop worrying about Big Data and use the data you have.”
The reason I made that statement was that so many companies have sat in endless meetings discussing the fact that they don’t have enough data to make decisions and yet they are not effectively using the data they do have.

It shocks me that major CPG companies are still only using a portion of the data they have access to. Some are happy that they use syndicated and panel data while others are combining up to eight to 10 different data sources. Still others are paying dearly for data and only using a fraction of the measures contained in the data. Do retailers and marketers understand the limitations of their data? I think many times they don’t understand that while combining data sources will lead to a higher level of insights and recommendations, they may not have the human resources or skill set to get the job done.

The limitation may actually be in people who understand how to use data as opposed to the data itself. Small Data and sometimes the combination of different Small Data sources can lead to big decisions.

Ian Percy
Member
6 years ago

I’m with Stephen Needel on this. Retail (and most other organizations) are prone to a “band-wagon” fetish. Too often we jump into things like Big Data out of desperation with little insight or foresight. Too much fertilizer is called weed killer. Too much data is called stagnation and paralysis.

Cynthia Holcomb
Member
6 years ago

The best way for a retailer to leverage data? Think outside the box of AI, ML, segmentation, visually alike, collaborative filtering and the like. The best data to leverage? Customer preference intelligence. Daily individual customers tell a retailer what they like and do not like. Customer preference microdata, cognitively translated by an expert system able to understand the inter-relationship between thousands of sensory attribute data points in a product to uncover the specific, individual sensory preferences of a human customer; in other words, like a human thinks and processes data.

AI and machine learning are still in their infancy when it comes to processing data like the human brain, lacking the ability to understand the subjective nuances between a person and a product to truly leverage data in personalized recommendations. Retailers have reams of micro customer preference intelligence accessible right before their eyes.

Gib Bassett
6 years ago

Marketers looking to best leverage the data at their disposal should collaborate across agency, internal IT and data science around their customer journeys or customer experiences. The label “Big Data” doesn’t really matter — it’s all about the analytics maturity of a retailer and how they are organized.

Too many retailers have bolted on data science, marketers don’t speak their language, and the data that agencies manage remains fenced off from transactional and customer interaction data created by internal systems. Speed, speed, speed and test, test, test, while quickly deploying to production or scaling the highest value use cases has got to be a priority.

Advancements in AI are like an octane booster to get retailers moving faster and taking advantage of methods they haven’t yet productionized like relatively straightforward personalization. The leaders are extending the gap here and most retailers have got to catch up quickly.

Seth Nagle
6 years ago

I can’t remember the last time I attended an industry event and have seen Big Data either in the keynote or in multiple breakout sessions. The term is used to cover an array of industry practices and just leads to more confusion.

I think the limitations of Big Data are now being accepted through dollars. Anything is possible if your budget is unlimited but to clean, harmonize and store data in a single repository costs a lot of money in not just upfront costs but ongoing maintenance.

The key to getting the most out of your data is to control what data goes into your database, create clear concise dashboards that aren’t in the weeds and keep out clunky, inconsistent data sources that just seem to break every other day.

Stuart Jackson
6 years ago

The industry, in some places, has developed what I call “Big Data blindness.” They are thinking in terms of only the high-order trends in the sector, and missing the personalized, very individualized treatment that customers expect.

There’s no doubt that Big Data is very powerful. It has applications right across the retail industry, and it can add serious value. Retailers can use it to optimize sales and ordering, adjust store opening hours and target marketing at the right customers at the right time. It’s a fantastic way to understand your consumers at a macro-level, and get to the heart of their needs, often before they know it themselves.

But in the stampede to adopt Big Data solutions, some retailers and e-tailers have started trying to make all their decisions using Big Data. It’s becoming clear that many retailers really believe every bit of the hype. They think that decisions about what will sell, what won’t, and how to wow customers in Paris can be made from sitting behind a computer screen in an office in Chicago.

That’s why, despite the gloom and doom right now, I do expect to see a resurgence on the High Street over the coming years. Consumers will start to rebel against the tech-driven companies and seek more intimate shopping experiences. The death of the High Street is greatly exaggerated.

Shawn Harris
Member
6 years ago

The idea of “Big Data” stumbled because it was too much data for us humans to make sense of. That ushered in “Small Data;” more useable chunks of data that a data scientist could use to model with.

I think the recent advancements in machine learning now allow us to go back to making sense of “Big Data,” or lots of data. I’m not sure retailers are at the point of needing to understand the limitations of data science, as I think they still have not yet grasped the potential. Marketers should be clear on the business problem they are trying to solve for, then enlist data science to determine the right data to model in order to draw out insights.

Ron X
Ron X
6 years ago

If you mix useful information with garbage, the results often stink. Advocates of Big Data often forget that they need to consider the quality and potential value of all the data they include in their analyses. Collecting non-probabiliy samples, adding unrelated observations and mining the set for value can put firms on the wrong paths.

Herb Sorensen
6 years ago

Data does NOT think, nor is it creative. It is gas in the tank, not even the vehicle, much less the driver.

Phil Rubin
Member
6 years ago

Thought provoking Rick, but I’m not sure most companies and brands, and retailers (other than AMZN and WMT especially) have fully embraced leveraging data to even get near a point where they have pushed the limits.

I’d challenge anyone reading this to name three retailers other than AMZN that demonstrate to you as a customer that they know “enough” about you to be truly relevant, improve your customer experience AND make you feel like that retailer is loyalty to YOU.

Most retailers still lack a single view of customers, much less one that is fully informed with the right data, including contextual.

Doug Garnett
Active Member
Reply to  Phil Rubin
6 years ago

Interestingly, Phil, Amazon constantly reminds me in the recommendations that despite 20 years of purchase data (plus whatever else they troll from online activity), they really don’t know me. Their recommendations are as irrelevant as they ever were.

To me, that suggests there’s a major flaw in big data theory. What I’ve written about elsewhere is that the flaw is that big data is purely behavioral. And behavior data is always missing critical insight: why someone does something. Amazon can see WHAT I did, but not having any sense of why, they remain incapable of reaching me.

Adrian Weidmann
Member
6 years ago

All content will be data-driven. Regardless of the buzzword that’s used to describe or define the workflow, the fact remains that a broad spectrum of data needs to be leveraged to drive efficacy and optimization. It’s the only way digitally empowered shoppers will receive content that they value. It’s also the only way that brands can publish and deliver valued content.

Tom Erskine
6 years ago

History repeats itself, and tech is no exception. The rise and fall of the term “big data” is no different from the rise and fall of other terms “cloud,” “AI,” “Cognitive Computing,” “SOA,” “Micro-services,” et al. Each of these terms describe technical tool or concept with the potential to solve real business problems. Once people find the killer application of the tool or concept, the need to continue to use the term goes away.

Doug Garnett
Active Member
6 years ago

Big data is far from dead — it’s just been renamed “AI” which is primarily using algorithms to process data. And that’s too bad. Because the term AI has led to an entirely new round of outsized hype.

Here’s what I believe we learned:

  1. There is value to be learned by analyzing data.
  2. It’s exceptionally rare that the value is large — as we know from decades analyzing data in direct response marketing it reveals small tweaks that are helpful but not earth shattering.
  3. Big data and AI both require massive investments. And both return smaller incremental value. So we need to be very cautious about those investments.
  4. Data/AI is not an end of its own. It merely complements other market intelligence like qualitative and quantitative research.
  5. It’s deadly to retail when management lets the data make decisions.

The big lesson from the Data/AI adventure is that effective retail success when people make smart, responsible decisions after reviewing the available research, data, analytics, and human behaviors.

Sarah Nochimowski
6 years ago

I don’t think we are even touching the tip of the iceberg there. We still don’t know how to analyze all this information and how to use it. Big Data (or whatever you want to call it) is just a newborn.

Ricardo Belmar
Active Member
6 years ago

What we choose to call it isn’t what’s important — it’s all in the execution. Being able to apply big data based insights into tangible marketing and operational processes is still elusive for many if not most retailers. Most retailers I talk to will tell you they have more data than they know what to do with — it’s not about collecting and analyzing, it’s about applying the insight to their business. They just don’t have enough people in the right roles with the right experience to execute well. And that goes for marketing, operations, everywhere. There are small gains being made, but largely I believe there is much room for improvement. More data scientists would help, but as others here have pointed out, there’s more money in financial services for those folks than there is (so far) in retail.

Mark Nicholson
6 years ago

Marketers should make use of data, but knowing where to look is critical. Identifying churn can have a real impact on a business. About 80% of companies believe they deliver out­standing customer service, but only 8% of their customers agree. Less than 5% of dissatisfied customers speak up, they simply disappear.

Watching for signs, clues and red flags through data can alleviate this dilemma to some degree.

Most marketing is focused on acquisition. Around 80% of revenue comes from customers already earned, yet many organizations spends maybe 2% of revenue on customer service. Big data can solve much of this.

Paul Donovan
Paul Donovan
6 years ago

The term “Big Data” itself is not really the issue here, it’s more about the issue of where this data is coming from, who is creating it, and how is it being interpreted. The majority of IT systems in place today have no way to ingest or analyze the newer streams of data from social, IoT, etc. Even in the best case scenarios in retail most of the analysis is still done in a sampling approach to analysis and execution rather than across the whole company. The main reason for this is there is no execution system landscape in place that can execute at scale based upon the insights. This is where it’s important for the business to break down the silos between IT and the business, it’s not about us and them — it’s just about “US” together!