RSR Research: What’s So Big About Big Data?
Through a special arrangement, what follows is an excerpt of an article from Retail Paradox, RSR Research’s weekly analysis on emerging issues facing retailers, presented here for discussion.
A few years ago, in great frustration at the unfettered and meaningless usage of the term "cloud computing," I wrote a short essay called Clouds in my Coffee. Googling around, I had found no credible definition of the term. Even Wikipedia’s definition was … cloudy. Today, the term "big data" is similarly getting tossed around like a football, used to describe everything from video analytics to merchandise planning.
However, there’s a difference. I actually understand what big data is, but the definition is a bit geeky and just doesn’t roll off the tongue easily.
First of all, it remains my contention that retail has always had big data. All of us have pushed mountains of data around since the advent of POS. There were two challenges: finding hardware fast enough to present the data in a quick and actionable way and getting retailers to pull their heads out of the detail weeds long enough to actually look at it.
So we’ve got the hardware now, and we’ve probably got enough of a cultural change going that we might even treat the analytic output as being enough to act on.
But I think the more interesting part of big data comes out of the consumerization of IT. We’ve got a whole new dimension of data called "customer sentiment." That data is new and it’s daunting. Retailers have thus far mostly tried to manage it using the squeaky wheel philosophy. Monitor the data coming out of social networks and reviews and respond to those who are most irritable or upset. That’s also a long-standing retailer habit: Squeaky wheel syndrome.
I can remember sitting at a retailer board meeting (not going to name names here) where a board member talked about his wife’s observations upon entering our store. Taking it as true customer sentiment, he expected us to actually change our stores based on her observations. Never mind that she wasn’t in our target demographic; never mind that it was a majority of one — we were supposed to just do it.
What would big data look like in this case? It would be the aggregation of all that "noise" out of social media, reviews, emails and site comments. This is no small trick. It requires a Natural Language Processor to parse the unstructured data and assign it a structured data element (see … it’s already getting geeky) and then associate it with a product, location or promotion. Finally, it must be integrated into a merchandising or marketing hierarchy somewhere and be aggregated into some kind of exception analysis. You know I had zero luck explaining this to Time magazine, and I knew it as it flew off my tongue.
Discussion Questions
What’s your technical as well as more basic definition of big data? How well do you think retailers understand what big data is?
Big data refers to data printouts over five pounds, or any number over a quadrillion. Most people understand that, don’t they?
Let me give a slightly different spin — it’s not what big data is, it’s what you do with it that defines it. In that sense, big data is any data set that let’s you query your shoppers behaviors and or attitudes in order to make business decisions.
Big data is represented by Volume, Velocity, and Variety. Retailers have always had volume (POS transactions) and velocity (like RFID tags), but Variety is the new dimension. Traditional systems are great at handling structured data, but the variety of data has introduced more unstructured data (text, video, etc.). As Paula points out, it takes another level of complexity to process natural language (like product reviews, Twitter comments, blog postings), and still more sophistication to draw conclusion via analytics. But for those that achieve this level, the payoffs will be handsome.
Not only are there numerous definitions of Big Data, but the people who hold those opinions are definitely sure that they know it when they see it. Paula’s definition is as good as anyone’s. The sooner we can get to a common understanding of what we’re talking about, the sooner we can start to address retail best practices. Until we do, we’re leaving a lot of money on the table.
Big Data is the collection of granular, unstructured retail data that when analyzed can give retailers new insights into their business operations — end to end. Big data is a source of item detail with a longer history which augments forecasting for new items and fast moving fashion items.
With big data in retail comes a need for platforms that can support unstructured, non-aggregated, and massive amounts of data — and make the conversion of the data into information in a matter of seconds vs. the old world of waiting hours for more info. Retail platforms that support big data will also not require data warehouses.
Lot’s to understand, but the simple point is that big data and big data platforms provide retailers with real, real-time information about every element of their business.
I prefer a wider definition, and without defining that (Hah!) I prefer to refer to big, BIG data. And in that I subsume TOTAL INFORMATION, about everyone and everything, all the time. An information cloud that simply IS, whether we have the tools to capture it, understand it or respond to it.
This also impinges on the issue of privacy — as Scott McNealy said, you have none. But that’s part of what big, BIG data is all about. The data exists, if only in concept. May as well attempt to stop the tide as to slow the inexorable recognition of big, BIG data.
But that doesn’t mean that marketers can use all that data, (read SPAM!) to communicate with shoppers — as many techies flooding into this space dream. Rather, as big, BIG data grows, big, BIG, consumer driven filtering will explode.
That is, you may know this or that about me, but you darn well better not let me know you know, unless you are being genuinely helpful to me, and I perceive it that way. Because, if you abuse me based on what you know about me, I WILL firewall you to hell with my personal SPAM filter.
It’s going to be a retail mess, but the few, the winning retailers, will learn that moving sales from SELF-service, to digitally mediated sales (using big, BIG data,) requires considerably more selling skills, sales finesse, than hectoring shoppers with inane coupons and cents-off on an ever increasing array of crap they have no interest in. You could put a long list of “retail” names here. 😉
Others are giving some great definitions, so I’ll focus on why it matters now more than in the past.
Reason one is that complex data presented in an actionable way is now a requirement for retailers (and brands too, by the way) to understand consumer behavior. The traditional marketing funnel has lost relevance as a model for forecasting consumer behavior. With the well documented omni-channel (there, another buzzword with varying definitions) phenomenon consumer paths to purchase have become much more complex and are inscrutable to retailers without a good view of their data.
A second reason big data has come to matter so much is Amazon. Amazon was built from day one to be a company that gained advantage from it’s access to and understanding of data. When Amazon executes well on this, which they often do, it changes consumers expectations of what good service is, what a competitive assortment is, etc. To compete in a world of rising consumer expectations, many other retailers will need to become fast followers of Amazon’s data-centric strategy.
Good discussion.
Here’s a way I’ve been thinking about the retail data challenge:
1. Dealing with the data (capture, process, manage, integrate etc.). I see 3 levels of sophistication here:
a. hierarchical / additive data like POS product and transaction data for product/category management and financial management. Seems like most retailers know how to do this.
b. non-hierarchical data such as the customer dimension. This is still an area in which many retailers and technologies struggle.
c. unstructured data – this is emerging and some retailers/brands are beginning to do this in patches.
2. Using the data effectively.
Here it is worth differentiating between event-based use (i.e. a discreet project) which is generally easy, and continuous and improving use which is generally harder (i.e. right-time systems to drive operations or decisions).
I also want to reinforce that consistency of approach, metrics and definitions is essential to effective application of data too. There is no use having hundreds of PhDs all doing clever stuff in independent little silos all determining their own definitions, measures and starting points. Bearing this in mind, to me there are 3 spheres of application that retailers and brands need to consider:
a. Applying within their business e.g. at retail. Driving merchandizing or marketing solutions with a consistent view of the data and approach.
b. Applying the data with the broader retail ecosystem (e.g. collaborating with suppliers).
c. Applying beyond the retail ecosystem. As digital blurs these boundaries, retailers and brands need to connect data and experiences in this broader sphere of influence.
Maybe this kind of framework could help retailers assess where they are and where they want to get to.
Paula is right, “Retail has always had big data.”
Fortune Magazine had a solid pictographic display of “Big” data in the 9/24 edition, pointing out that “From the beginning of time until 2003, we created 5 billion gigabytes of data. In 2011 the same amount was created every two days. By 2013 that time will shrink to 10 minutes.”
Okay, so we have Volume and Speed. What retailers (and other marketers) need is to make this data user-friendly. In order to get a competitive advantage, users of business data need help in managing, tabulating, analyzing, visualizing, and delivering the data in interactive, web-based and mobile formats that their people can utilize immediately to solve pressing business problems.
The companies that will maintain their competitive advantage in an era of “drinking from a firehose of data” will be ones that integrate the data. Perhaps the curse of the Digital Age is that there is NO TURNING BACK. Those who think otherwise are in denial, and won’t be in place in short order.
Excellent observations. And, reading the last paragraph, there’s far more subtlety to leveraging big data than one might think at the outset. (The mere term “big data” implies it’s a thoroughly accurate thing — but it’s not.)
Aggregating social media feedback is far more of a challenge than merely technological. We did research years ago for a company whose product was being attacked for having environmental problems (which, in reality, it didn’t have). We did some research with a massive, highly reliable research entity. In that research, there were open end questions which the company coded for us. This coding clearly showed the environmental issues to be a big problem among consumers.
Except reading the open end answers, I was concerned something had been missed.
So I had my team re-code the questions using the specialized knowledge of consumer language around the issue that we’d developed over time. This was far more subtle coding and far more accurate. What did it show? That, understood correctly, the environmental concerns hadn’t penetrated to the mass market.
So when it’s suggested that we could automate a process to “parse the unstructured data” I get concerned. We should do it. BUT, we should remember that answers pulled from that data may depend far more on the assumptions going into the parsing model than on the true consumer issues.
There are three sides to the Big Data definition that must be understood by today’s decision makers. There is the software side, file side, and the hardware side of the story. The information a company needs to run the business is largely provided by an IT system either outsourced or in house, which ever is the most cost effective.
Every software program stores data in its own file structure which generally can not be used effectively by another software program. Multiple application software makes information redundancy a highly potential issue. So when you use multiple applications to manipulate large amounts of data you are in fact increasing the load on the hardware. When the data and software are to big to handle for the hardware in use in the time frames given the company has a BIG DATA issue. Remember, redundancy in both software and data is the fastest way to render an IT system inept. Other types of big data issues include managing files that are to large for the system to handle. Still another is the file sizes for graphics information and the next most common problem is the systems’ inability to communicate without slowing the entire network. And finally there is the problems of distributed processing. Getting a better handle on these issues may scare the average executive but it is necessary for accurate decision making.
‘Big Data’ is all of the quantitative and qualitative metrics and insights that is gathered from a shopper’s journey starting at what Google defined as the the ‘Zero Moment of Truth’ (ZMOT) all the way through the entire lifecycle of the consumer’s experience with owning a brand’s goods or services. Brands and retailers have always had ‘Big Data’, but it has been siloed within their organization and if any insights were gleaned they were either reactionary or they were too disruptive to the status-quo to implement.
One thing that has become clear to me based upon direct experience in creating ‘Big Data’ is that it is not enough to generate ‘actionable data’. Data is the easy part as it is all history. Marketers and Merchandisers need (and value!) recommendations and predictive guidance based on all of the ‘Big Data’. That’s is the promise of the ‘omni-channel’ discussion. Collating all the Big Data and derive recommendations for brand and retailers alike moving forward is the truly valued objective.
Guess what! “Big Data” HAS always been around. Dating back to before scanning, even. Stores were trying to capture consumer information so they could better stock their stores. Well, today, we have billions more people shopping at thousands more stores with billions more devices (PC, POS, mobile, kiosk, etc.)… now conversing with billions of people around the world about how they love or hate a retailer and/or a CPG brand. So, what was once “Small Data” has now become “Really Big Data”. And… 95% of retailers have not learned how to capture the 80% of that data that is literally unstructured, nor have they even begun to gain insights and develop subsequent actions.
Here’s the good news: There are tools today to capture, analyze and create actionable insights for the myriad sources of data throughout the supply chain ecosystem.
It is not as important to understand what big data is as is to know WHAT to do with it.
What all retailers are after is actionable knowledge, which is derived from clearly presented information culled from an endless stream of data.
Perhaps it’s called ‘big’ data because it’s too big. It’s certainly not called ‘just the right amount of’ data.
If you know the questions that you’d like answers to, it’s a relatively straight-forward process to extract the answers from the data. The challenge comes when there’s answers buried in the data to questions we haven’t even thought of asking yet.
Ultimately the issue of big data is a managerial one. What do we want to know? What do we need to know? And perhaps most importantly, what additional questions is the data/information/knowledge that we are looking at prompting us to ask?
Some very good comments made already, so I’ll be brief:
1. Retail data has always been big in comparison to most other industries, and it is getting bigger faster than ever before, for all the reasons cited in other comments.
2. The challenge is not only how to gather, store, process, and communicate the “big data,” but more importantly, how to turn it into action — action that allows a retailer to set a clear strategy, create plans aligned with that strategy, and finally helps drive the actions of those employees who spend their days serving customers. Data without actionability is just wasted storage.
It’s whatever works for you. Over time, the dominant definition will win out.
Big data is relative, not absolute. Some questions businesses ask become analytical models that only need a small amount of data and many repetitions. Big data needs to meet high-performance analytics to rapidly inform retail decisions across the organization and create unique customer value. The best retailers in the world create hard-to-copy competitive advantage by leveraging their unique big data and retail savvy in conjunction with retail analytics.