Big Data is done, put a fork in it


Rick Ferguson, CLMP
Through a special arrangement, presented here for discussion is a summary of a current article from The Wise Marketer, a website and newsletter serving the global loyalty industry.
About a decade ago, the phrase “data is the new oil” swept the globe as the twin corporate power centers of IT and marketing realized their companies had more data than they knew what to do with. “Drinking from the fire hose” became a metaphor for the struggles of extracting actionable insights from data.
About five years ago, the usual suspects in IT consulting and cloud-based analytics began to trumpet the phrase “Big Data” to sell into companies hoping to extract that resource.
Now, as Slate’s Will Oremus points out, the phrase Big Data has become passé — in part because we just call it data now, and in part because the rush to rely solely on data for business decision-making has often revealed the limitations of data-based decisions.
Our over-confidence in these tools often hinders our ability to see the forest for the trees. We often fall victim to what data scientist Shane Brennan calls the “Ten Fallacies of Data Science.” For a variety of reasons, the data may be inaccessible, indecipherable, outdated, lacking enough granularity for analysis or prohibitively expensive to get or have processed, Mr. Brennan contends. Often, a marketer’s lack of understanding or pre-conceived notions leads to poor decisions.
“Garbage in/garbage out” results in marketing campaigns or loyalty offers that are ineffective or even harmful to your business.
Should all data analysis be eschewed in favor of, say, gut feel, astrological charts or throwing darts at a board? Not at all. Data analytics is being successfully used to fuel personalization, construct relevant offers and build differentiated experiences.
But by understanding the limits of data science, your efforts on data can best correlate with current customer value or be most predictive of future customer value. Call it “small data” or “customer-centric-data,” or just continue to call it data. If customer behavior shifts in a profitable direction because of your analysis, then you’ll know you’re on the right track. Be diligent, question your assumptions and be aware of your biases. Big Data may be over, but data science, like any scientific pursuit, is forever.
- The rise and fall of Big Data – The Wise Marketer
- How “Big Data” Went Bust – Slate
- The Ten Fallacies of Data Science – Medium
DISCUSSION QUESTIONS: Should “Big Data” be retired as a progressive movement in marketing? Do retailers and brands generally understand the limitations of data science within their organizations? What advice would you have for marketers attempting to best leverage the data at their disposal?
Join the Discussion!
36 Comments on "Big Data is done, put a fork in it"
You must be logged in to post a comment.
You must be logged in to post a comment.
President/CEO, The Retail Doctor
This is semantics. We use Big Data now to refer to AI, machine learning and predictive analytics in one easy term. Watson is much more than just “data.”
Principal, Your Retail Authority, LLC
Yes, Bob, it is semantics. Data in retail will always be fuel behind almost everything we do. It’s how smart we get in using it. For my 2 cents.
Co-founder, RSR Research
I don’t think “Big Data” was ever well defined, and the systems that might actually make use of the insights were not in place in retail operations.
And then there was omnichannel, which overtook corporate IT spending (along with security).
I think the biggest issue with retailers and brands is not that they don’t understand the limitation of data science — it’s more that the data scientists have gone into financial services, where they can make real money. It’s on the vendors to provide the science and make it easily digestible for the retailers, et al.
Principal, Retailing In Focus LLC
If you read yesterday’s Wall Street Journal interview with the chairman of Fast Retailing (Uniqlo), you might have seen this perceptive comment: “Data would never substitute the merchant. How do you interpret the data? That’s the merchant’s skill set. You need to uncover the insight that is buried in the data and the merchants need to uncover it. Even if you employ artificial intelligence to help you, the numbers [don’t tell] the future.”
His point is well taken: No matter how much importance a retail organization places on its ability to extract data from its transactions, the information means nothing if it can’t be turned into action — and some of that decision-making rests on instinct and experience.
That being said, declaring that “Big Data is dead” is an overreaction. The phrase itself may be overused, but data science is alive and well in the interest of smarter merchandising decisions, loyalty programs and so forth. Would Amazon be where it is today without groundbreaking use of data to develop its predictive technology? I don’t think so.
Principal, Retailing In Focus LLC
And just to add to my comment: I teach an undergrad retailing class at UW-Milwaukee and one of my guest speakers this term is a data scientist whose firm works with multiple omnichannel and e-commerce retailers. He made the point that 90 percent of the world’s data has been generated over the last two years, and the trend toward accelerating data generation continues.
Can anybody honestly say that “data science” is dead, even if “Big Data” is a cliche?
Founder | CEO, Female Brain Ai & Prefeye - Preference Science Technologies Inc.
Good points. I agree with Mr. Yani’s comments as well. What is missing? Data scientists are working to interpret data on a subject requiring deep knowledge only an experienced merchant has earned. Thus the disconnect. To cross this chasm will require specialized merchants to work hand in hand with data scientists.
Managing Partner, Advanced Simulations
Back in 2013 I delivered a paper at ESOMAR’s congress on why Big Data was not such a big deal — this echos a lot of what I said at the time. I’m not sure this was ever a “progressive” movement. We were sold a bill of goods on possibilities that turned out not to exist. I’ve always maintained that marketers and retailers need to focus on the questions they would like answered and let their researchers find the best way to answer them. We shouldn’t put the cart before the horse, trying to leverage the data we have, just to try and leverage it.
President, The Ian Percy Corporation
Excellent response, Stephen!
Co-founder, CART
Well said. Data and insights for the sake of data and insights doesn’t net any impact on the business. It’s what is done as a result that makes the difference. Having the business look at what questions to ask connected to actions they could take is definitely the best approach.
Vice President of Marketing, OrderDynamics
Thanks for retiring the term. Big Data is probably the most overused and abused term in the business world. I have heard VPs talk about the critical importance of it, and how every firm needs it, but they did not really understand much about the hows, whys, tactics or implications.
Data is messy. The important thing is that executives understand that you still have to make a decision on the data you have at hand, and it still will often come down to a judgment call. The worst thing you can do is to fall into an analysis-paralysis vortex. Don’t let this happen to you! There is a point where you have analyzed the data enough and it is time to make a judgment call. By the time you have full clean data, the conclusion is usually obvious. Your competitors have already long since taken action while you sit pondering what happened — in the dust.
Global Retail & CPG Sales Strategist, IBM
I’m just a bit concerned about stating that “Big Data” should be retired without the intentional, highlighted hand-off to another term, like, “GINORMOUS DATA.” Retailer and CPG brands haven’t even scratched the surface of capturing and deriving insights from the more than 80 percent of all data which is “dark” and invisible to current systems in these enterprises. Capturing this data is the first step in acknowledging what the current limitations of data science really are. From there, there’s a ton of work to do still. Let’s get going!
President, The Ian Percy Corporation
You trigger the thought that “information” and “insight” are totally different things though often confused.
Founder and CEO, Bobsled Marketing
I look to arguably the best example of a company who obtains, compiles, and uses data to its full advantage: Amazon. As a core leadership principle, Amazon uses data to take huge bets and lead decisions. And when the data doesn’t support an idea, the venture is quickly killed and moves on. It seems so much part of the DNA of the company that giving it a special title is pointless. It’s just the way Amazon does business.
Professor of Marketing, The Wharton School of the Univ. of Pennsylvania
I’m as strong an advocate as you can get for “Small Data,” i.e., squeezing as much value/insight as possible about simple transaction log data before turning to more elaborate “Big Data” schemes. But that doesn’t mean that Big Data is done — quite to the contrary, the real era of Big Data hasn’t even started yet. Once we master the art and science of Small Data, we’ll finally be in a position to start to properly leverage Big Data, in ways that most retailers can’t even imagine today.
It’s a lot like CRM: people were writing off CRM systems (literally and figuratively) 15 years ago because they weren’t yet in a position to make strategic decisions that required it. But today we know that CRM is the bare minimum pf what you need for customer-centric success. Big Data will have a similar resurgence in a few years.
But first retailers really need to get started with the Small Data revolution that is right at their doorstep and ready to happen …
Managing Partner, Deeb MacDonald & Associates, L.L.C.
The new term should be “Smart Data.” Companies that have learned how best to efficiently mine their data for better sales and profits, customer retention, better logistics operations etc. are already on to this “smart” usage. There are many examples — from dunnhumby to Watson — of businesses that understand how to utilize their information for results.
Strategy & Operations Delivery Leader
Similar to the apprehension retail industry experts have about the term “omnichannel,” Big Data now shares a similar distinction.
Big Data is a term that was intended to simplify the complexity of all of your data sources, consolidate it, and present it back in a form that is both consumable and able to be operationalized.
Essentially we should retire this term, as it overflows into the predictive analytics, augmented intelligence, cognitive data and artificial intelligence arenas, which are far better defined and are undergoing significant evolutions.
Strategy Architect – Digital Place-based Media
A stake to the heart of the term Big Data is not going to kill the essence of managing by information. Let it morph into a descriptor of its business value rather than what it is, which has driven investment and has been the enterprise call to action, through which abilities and incapabilities are being continuously revealed.
Retail Solutions Executive, Teradata
Big Data as a term might be passe’ but the use of Big Data is only going to grow with the adoption of machine learning tools and AI (coupled with prebuilt retail data models) to harvest greater volumes of insights from growing volumes of data that are bigger than what data scientists could ever dig through for non-critical or “cold” data needs. THAT is where the term Big Data makes sense. The key is to have a clear data strategy and architecture that allows for smaller and more critical data sets to be analyzed by the data science teams within retailers for better results.
President, founder and CEO Interactive Edge
President, The Ian Percy Corporation
I’m with Stephen Needel on this. Retail (and most other organizations) are prone to a “band-wagon” fetish. Too often we jump into things like Big Data out of desperation with little insight or foresight. Too much fertilizer is called weed killer. Too much data is called stagnation and paralysis.
Founder | CEO, Female Brain Ai & Prefeye - Preference Science Technologies Inc.
The best way for a retailer to leverage data? Think outside the box of AI, ML, segmentation, visually alike, collaborative filtering and the like. The best data to leverage? Customer preference intelligence. Daily individual customers tell a retailer what they like and do not like. Customer preference microdata, cognitively translated by an expert system able to understand the inter-relationship between thousands of sensory attribute data points in a product to uncover the specific, individual sensory preferences of a human customer; in other words, like a human thinks and processes data.
AI and machine learning are still in their infancy when it comes to processing data like the human brain, lacking the ability to understand the subjective nuances between a person and a product to truly leverage data in personalized recommendations. Retailers have reams of micro customer preference intelligence accessible right before their eyes.
Director, Solutions Marketing with Alteryx
Marketers looking to best leverage the data at their disposal should collaborate across agency, internal IT and data science around their customer journeys or customer experiences. The label “Big Data” doesn’t really matter — it’s all about the analytics maturity of a retailer and how they are organized.
Too many retailers have bolted on data science, marketers don’t speak their language, and the data that agencies manage remains fenced off from transactional and customer interaction data created by internal systems. Speed, speed, speed and test, test, test, while quickly deploying to production or scaling the highest value use cases has got to be a priority.
Advancements in AI are like an octane booster to get retailers moving faster and taking advantage of methods they haven’t yet productionized like relatively straightforward personalization. The leaders are extending the gap here and most retailers have got to catch up quickly.
Senior Marketing Manager, RW3
I can’t remember the last time I attended an industry event and have seen Big Data either in the keynote or in multiple breakout sessions. The term is used to cover an array of industry practices and just leads to more confusion.
I think the limitations of Big Data are now being accepted through dollars. Anything is possible if your budget is unlimited but to clean, harmonize and store data in a single repository costs a lot of money in not just upfront costs but ongoing maintenance.
The key to getting the most out of your data is to control what data goes into your database, create clear concise dashboards that aren’t in the weeds and keep out clunky, inconsistent data sources that just seem to break every other day.
Managing Director, Regency Analysis
Senior Director, Global Retail and Hospitality Strategy & Business Development, Turing.ai
The idea of “Big Data” stumbled because it was too much data for us humans to make sense of. That ushered in “Small Data;” more useable chunks of data that a data scientist could use to model with.
I think the recent advancements in machine learning now allow us to go back to making sense of “Big Data,” or lots of data. I’m not sure retailers are at the point of needing to understand the limitations of data science, as I think they still have not yet grasped the potential. Marketers should be clear on the business problem they are trying to solve for, then enlist data science to determine the right data to model in order to draw out insights.
If you mix useful information with garbage, the results often stink. Advocates of Big Data often forget that they need to consider the quality and potential value of all the data they include in their analyses. Collecting non-probabiliy samples, adding unrelated observations and mining the set for value can put firms on the wrong paths.
Scientific Advisor Kantar Retail; Adjunct Ehrenberg-Bass; Shopper Scientist LLC
Data does NOT think, nor is it creative. It is gas in the tank, not even the vehicle, much less the driver.
Founder, Grey Space Matters
Thought provoking Rick, but I’m not sure most companies and brands, and retailers (other than AMZN and WMT especially) have fully embraced leveraging data to even get near a point where they have pushed the limits.
I’d challenge anyone reading this to name three retailers other than AMZN that demonstrate to you as a customer that they know “enough” about you to be truly relevant, improve your customer experience AND make you feel like that retailer is loyalty to YOU.
Most retailers still lack a single view of customers, much less one that is fully informed with the right data, including contextual.
President, Protonik
Interestingly, Phil, Amazon constantly reminds me in the recommendations that despite 20 years of purchase data (plus whatever else they troll from online activity), they really don’t know me. Their recommendations are as irrelevant as they ever were.
To me, that suggests there’s a major flaw in big data theory. What I’ve written about elsewhere is that the flaw is that big data is purely behavioral. And behavior data is always missing critical insight: why someone does something. Amazon can see WHAT I did, but not having any sense of why, they remain incapable of reaching me.
Managing Director, StoreStream Metrics, LLC
All content will be data-driven. Regardless of the buzzword that’s used to describe or define the workflow, the fact remains that a broad spectrum of data needs to be leveraged to drive efficacy and optimization. It’s the only way digitally empowered shoppers will receive content that they value. It’s also the only way that brands can publish and deliver valued content.
CEO, One Door
History repeats itself, and tech is no exception. The rise and fall of the term “big data” is no different from the rise and fall of other terms “cloud,” “AI,” “Cognitive Computing,” “SOA,” “Micro-services,” et al. Each of these terms describe technical tool or concept with the potential to solve real business problems. Once people find the killer application of the tool or concept, the need to continue to use the term goes away.
President, Protonik
Big data is far from dead — it’s just been renamed “AI” which is primarily using algorithms to process data. And that’s too bad. Because the term AI has led to an entirely new round of outsized hype.
Here’s what I believe we learned:
The big lesson from the Data/AI adventure is that effective retail success when people make smart, responsible decisions after reviewing the available research, data, analytics, and human behaviors.
Director of Marketing, Treepodia
I don’t think we are even touching the tip of the iceberg there. We still don’t know how to analyze all this information and how to use it. Big Data (or whatever you want to call it) is just a newborn.
Retail Transformation Thought Leader, Advisor, & Strategist
What we choose to call it isn’t what’s important — it’s all in the execution. Being able to apply big data based insights into tangible marketing and operational processes is still elusive for many if not most retailers. Most retailers I talk to will tell you they have more data than they know what to do with — it’s not about collecting and analyzing, it’s about applying the insight to their business. They just don’t have enough people in the right roles with the right experience to execute well. And that goes for marketing, operations, everywhere. There are small gains being made, but largely I believe there is much room for improvement. More data scientists would help, but as others here have pointed out, there’s more money in financial services for those folks than there is (so far) in retail.
VP, Marketing at Nicejob
Marketers should make use of data, but knowing where to look is critical. Identifying churn can have a real impact on a business. About 80% of companies believe they deliver outstanding customer service, but only 8% of their customers agree. Less than 5% of dissatisfied customers speak up, they simply disappear.
Watching for signs, clues and red flags through data can alleviate this dilemma to some degree.
Most marketing is focused on acquisition. Around 80% of revenue comes from customers already earned, yet many organizations spends maybe 2% of revenue on customer service. Big data can solve much of this.
Global Senior Director, Retail Business Unit at SAP
The term “Big Data” itself is not really the issue here, it’s more about the issue of where this data is coming from, who is creating it, and how is it being interpreted. The majority of IT systems in place today have no way to ingest or analyze the newer streams of data from social, IoT, etc. Even in the best case scenarios in retail most of the analysis is still done in a sampling approach to analysis and execution rather than across the whole company. The main reason for this is there is no execution system landscape in place that can execute at scale based upon the insights. This is where it’s important for the business to break down the silos between IT and the business, it’s not about us and them — it’s just about “US” together!