Why is gaining meaningful insights from data still so hard?
Photo: @andreeas via Twenty20

Why is gaining meaningful insights from data still so hard?

MarketingCharts staff

Through a special arrangement, presented here for discussion is a summary of articles from MarketingCharts, which provides up-to-the-minute data and research to marketers.

Half of the marketing, information and marketing tech professionals surveyed in the UK and U.S. for a report from Merkle state that their organization’s data is not organized for easy consumption, making it the most common technical barrier to deriving meaningful insights.

Other barriers include limited storage, cited by 39 percent; slow data analytics processes, 38 percent; the inability to understand what data is most important to decision-makers, 38 percent; data integration hurdles, 38 percent; and data being stored in disparate systems, 35 percent.

Merkle Challenges Gaining Insights from Data Mar2021

Forty-one percent of respondents say they do not have a single customer profile, even though 89 percent agree that creating a consolidated customer profile is very or extremely important. The inability to capture a single customer view also comes despite 81 percent of U.S. respondents reporting they have a CRM and two-thirds having a customer data platform.

What’s standing in the way? For one thing, technology spend doesn’t seem to be a limiting factor. Some 38 percent of U.S. respondents report allocating 21-25 percent of their marketing technology spend to identity-based solutions, while 27 percent allocate 16-20 percent.

Instead, it’s more likely to be a lack of expertise and skills. Data and analytics is not only currently a valued skillset but is expected to remain so in the future. Nonetheless, close to half (46 percent) of U.S. respondents have run into a lack of data and analytics expertise within their organization when trying to implement a data and analytics solution.

Additionally, many have encountered limitations in gaining consensus among stakeholders (46 percent) or a lack of an agile implementation partner to support changing business and time to market requirements (44 percent).

Finally, although IT is said to no longer be a detractor to marketing, the Merkle report suggests it is still more likely to be in control — finding that IT (56 percent) is more likely than marketing (34 percent) to allocate more than 20 percent on martech spend to identity-based solutions.

BrainTrust

"Using data is only as good as the questions that you are seeking to answer."

Joel Goldstein

President, Mr. Checkout Distributors


"First, let me say that insights without predictive value are empty calories. You must have an eye towards prediction or activation for an insight to really have meaning."

Joel Rubinson

President, Rubinson Partners, Inc.


"Competencies are the biggest barrier. This includes the will to get things done, ability to overcome legacy methods of doing things, and lack of accountability."

Chuck Ehredt

CEO, Currency Alliance


Discussion Questions

DISCUSSION QUESTIONS: What do you see as the biggest barriers to using customer data effectively and creating a single customer view? Is the primary challenge resources, competencies, organizational, shared vision or something else?

Poll

45 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
David Naumann
Active Member
3 years ago

The greatest challenge for many brands in achieving a comprehensive, single view of the customer is disparate data that isn’t connected in real-time. Data silos for omnichannel retailers has been an ongoing barrier to attain a 360 degree view of customer data. Using a consistent customer identification methodology across all channels is key to recognizing individual customers. Ideally, retailers should adopt a unified commerce platform to keep all data on the same platform for one version of the truth and easy access to data. This would alleviate the top challenge noted in the Merkle report – organizational data is not organized for easy consumption.

Joel Goldstein
3 years ago

Using data is only as good as the questions that you are seeking to answer. If you’re looking to learn the gender or age range of those purchasing from your store, data can be your best friend. However if you’re trying to build a psychological profile when deciding which colors to use it is a much more challenging prospect as the data may not be easily correlated.

Jeff Sward
Noble Member
Reply to  Joel Goldstein
3 years ago

Great way to put it. Data easily answers the question of “What is the best seller?” Easy question … easy answer. The not so easy question is “WHY is that item a best seller?” Style, fabric, color trim…? Movie star posted it on Instagram last week? And even if you know the WHY, then what is the shelf life of the data? A week? A season? Do I re-order, or try to be smarter next time…?

Joel Goldstein
Reply to  Jeff Sward
3 years ago

Thank you Jeff.

Christine Russo
Active Member
3 years ago

This is NUTS! Data is such a critical component of future success and access to both forward looking sentiment and rearview history is easier and more affordable than ever. Perhaps this is a factor of an old-guard management point of view. But then change must come, and quickly. Subscription models to data or simply pay for play outfits like Piplsay or Prosper Insights make access to the information easy.

NAVJIT BHASIN
Reply to  Christine Russo
3 years ago

Christine, I totally agree! Lack of internal data scientists should not stop anyone from gleaning customer insights. In order for data to solve problems, it needs to be unified into a single version of the truth, distilled to the most meaningful insights, and provide corrective actions. Decision intelligence is going to be the the next wave of retail transformation.

Dr. Stephen Needel
Active Member
3 years ago

Two thoughts. First, the assumption that a single customer view exists may be a flawed concept. Your business may appeal to multiple customer cohorts – think Walmart, which is both historically downscale yet upscale chic. Second, what keeps businesses from using data effectively is usually competence, and that’s only getting worse as crap AI and poor modeling skills invade marketing.

Cathy Hotka
Trusted Member
3 years ago

Participants in the VP IT Council meeting repeatedly referred to inadequate data use as an ongoing issue. Despite access to multiple data analysis systems, many retailers just don’t adequately benefit from the data they collect, in part because of an absence of vision. 2021 may be the year that we begin to turn the corner on this.

NAVJIT BHASIN
Reply to  Cathy Hotka
3 years ago

Cathy – we have been hearing of retailers calling themselves “data driven” for a long time now. But to remain competitive and ahead of the pack, they will be forced to make it happen soon.

Cathy Hotka
Trusted Member
Reply to  NAVJIT BHASIN
3 years ago

This is the year!

Ken Morris
Trusted Member
3 years ago

I believe the primary barrier is organization and infrastructure. We have marketing folks in charge of technology which they are not trained to do and we have silos of data controlled by IT and in turn by finance who control the budget. We need data normalization. We need to rid ourselves of these islands of customer data and create a holistic plan to brings these separate data lakes under one umbrella and fund the initiative appropriately. The retail world runs on hard dollar savings and what we need here is infrastructure which has soft savings at best. Retailers need to organize technology under technologists, invest in the technology to obtain the 360 degree view of the customer and invest in data scientists under the control of the CMO.

Gary Sankary
Noble Member
Reply to  Ken Morris
3 years ago

Ken, well said. They also need to empower their teams teams with tools that can use this data to improve execution.

Michael Terpkosh
Member
3 years ago

To start, in many retail organizations there are multiple sources of data that don’t work well together. Loyalty data, POS data, and syndicated data, to name a few. Each of these sources of data may be stored in different locations and not stitched together to create a holistic view of the retailer’s business or the shopper. When an organization can get to the point of bringing all the data together, the skills of the retailer’s merchandising and marketing teams become critical. Plus, there are competing views of how to execute against any insights. There needs to be retailer vision bringing the data, associates and analytical insights together. Only then can you get to efficient and effective strategies and tactics to execute at retail.

Suresh Chaganti
Suresh Chaganti
Member
3 years ago

Based on what I have seen, the true barrier is the lack of belief in what data-driven decision-making can do. But that is perpetuated by lack of skills in the organization that bridge data and business.

There are plenty of tools in the marketplace. The costs have come down significantly. There is no dearth of data either – tons of first-party data, huge volumes of syndicated and macro data.

But the premium is still on human resources who can make sense of it, and leadership who encourages that.

Gary Sankary
Noble Member
Reply to  Suresh Chaganti
3 years ago

I would suggest that while there are lots of data analytics tools out there, we still have a gap in the automation and execution tools that inform the tools that front-line workers are using to make their daily decisions. That’s where I see the bottleneck.

Mohamed Amer
Mohamed Amer
Active Member
3 years ago

It’s not one thing but a confluence of reasons that also vary by a retailer’s data maturity. Having to choose the biggest hurdle, I suggest that data silos are the common challenge that is, in part, a reflection of the retailer’s organization and the changing priorities over time.

Venky Ramesh
3 years ago

Last year I interviewed several CPG executives as part of the Capgemini Research Institute survey on Data Powered Enterprises. We found that while everyone understood the importance of data and analytics to the success of their business, less than 40 percent of organizations used data-driven insights to drive business value and innovation. One key reason for that we found was business executives do not trust the data they receive. 62 percent of technology executives believed the business trusted their data while only 20 percent of business executives agreed. The two key reasons for that lack of trust were – quality of data and lack of alignment to business strategy. The complete report can be accessed here.

Gene Detroyer
Noble Member
Reply to  Venky Ramesh
3 years ago

That is interesting … “lack of alignment to business strategy.” Does that mean the data is suggesting the business strategy is wrong and the executives don’t like the indications? The data should suggest the direction of the strategy.

Venky Ramesh
Reply to  Gene Detroyer
3 years ago

Thanks for your comments Mr. Detroyer. To clarify, when we asked the technology executives if their data/analytics aligned with the overall business strategy, 56 percent agreed. However, only 38% of business execs agreed. The gap emphasizes the mismatch in organizations on not only defining the strategy but also on designing the architecture and deploying the platform and operating model. We found that successful organizations have moved from “data strategy supports the business strategy” to “data strategy is the business strategy.”

Oliver Guy
Member
3 years ago

When it comes to data, retail as an industry has something that is a blessing and a curse. That being – huge amounts of data. Dealt with effectively this should be a huge boon to retailers but there are a number of things limiting this. Firstly data silos – it is all very well having lots of data but if it is in different systems and you cannot connect it then you have an incomplete picture. For example – if you are able to understand which customers have bought products but not link that to customer specific advertising or promotions in the past then this limits the usefulness of this data and restricts how you might draw anything useful from it to help you in the future. The huge amounts of data in a retail setting also suffer from the three Vs of big data – volume, variety and velocity. These have actually gotten worse as the ability to capture has increased – for example with a store sale it is more difficult to identify specific customers.
One key challenge is that data can be converted into a report – but this is after the fact, it is too late to do anything useful. The insight is effectively “perishable.”

The three Vs do make the ability to respond in real-time very difficult. Technologies such as real-time streaming analytics that have been used in the world of banking and trading for almost 20 years have attained limited traction within retail. The ability, however, to leverage this approach while embedding predictive models offers huge possibilities to respond to data driven events – both actual and inferred – and this unlocks phenomenal possibilities to improve customer service and provide an intelligent automated response to what is happening across the business.

Matthew Pavich
3 years ago

In my experience, the largest barriers are almost always organizational willingness to transform into a data-informed culture. Whether it is CRM, supply-chain or pricing data, the capabilities exist and can drive enormous value if retailers are able to do “the hard work” and employ all of the needed best practices to achieve their business objectives. Unfortunately, some retailers purchase Ferraris and then spend the next several months trying to convert them into bicycles. The best retailers embrace the top solutions, remove data silos internally and transform their organizations to meet the demands of their consumers profitably using the best analytics available.

Ian Percy
Member
3 years ago

First, the quest for a “single customer view” is futile. And it’s an insult to customers. As King Solomon pointed out around 950 BCE, “no one can know the human heart.” So collect data all you want and best of luck to you.

While technology claims the possibility of artificial intelligence, it hasn’t mastered “wisdom.” My definition of wisdom is knowing what best to do in a situation you haven’t experienced before. Wisdom is the rarest quality in the world, nevermind in retail. That’s why you never see it in a job description. Every customer is one you haven’t seen before. Even the repeat ones. Neither technology nor massive amounts of data are going to change that.

John Hennessy
Member
3 years ago

Lack of a plan is what I run into. RFPs for analytics tech seek to boil the ocean. When you seek everything, you end up with nothing. Instead, start with a plan. The plan dictates the data, resources and expertise. If the plan fails to deliver, learn from it and build a better plan. If the plan succeeds, learn from it and establish a new plan to expand what you can learn and do. It’s not the lack of data or lack or resources or lack of spending, it’s lack of a clear objective that I’ve seen prevent successful use of data and analytics.

Perry Kramer
Member
3 years ago

The critical challenges are not lack of effort, spending, or technology. The major inhibitors are found in the areas of mission objectives, data stewardship and consolidation. The challenge comes from each retailer needing to define what CRM means to them and having an agreed upon customer data direction and then building to it. There are a few effective organizational models that can work as long there is a clear objective and set of guiding principles that the entire organization adheres to. This is the only way to eliminate the multiple systems and data repositories with customer data that most retailers have today. If you look to the leading grocers and membership based retailers, they mastered this years ago.

Doug Garnett
Active Member
3 years ago

We look to data for insights it can’t supply. As a mathematician (MA-Applied Math UCSD) with 25 years in the direct marketing business what I have learned is that insights through data are primarily of low value. So we invest more and more to find less and less.

We arrive at this question not because better methods or tricks would find more value – but because we bought into what the data folks promised and that has set our expectation too high for what the expensive investment can deliver.

Retailers need to revise their expectations because there aren’t major things they are missing. Retail success comes from managing better that which we already know rather than searching data in hopes of discovering the meaning of profits.

Lisa Goller
Trusted Member
3 years ago

Investing in martech is one thing. Companies must also adapt their internal processes to unleash the technology’s power to serve consumers as unique individuals.

The primary challenge – and opportunity – is that a company-wide commitment to value the power of data needs to start at the top. When CEOs and CFOs see how the right martech can improve marketing effectiveness and profitability, they will prioritize its success.

Cleaning up the digital clutter of years’ worth of inaccurate, outdated data is a massive job that saps organizational resources. Rather than trying to manage data alone, companies can partner with trusted data experts to make data integrity more efficient.

Also, establishing standards that reflect corporate priorities makes it easier to report consistently, compare results and make better decisions to improve the customer experience.

Jeff Sward
Noble Member
3 years ago

This sounds like a classic left brain vs. right brain question. Part of the issue is how data is organized and accessed, but the other part is who is doing the analysis. Planners don’t design and designers don’t readily interpret data. The analysis is not pure number crunching and it’s not a simple leap to design. The interpretation of the data is an artful synthesis of a whole bunch of moving parts — a combination of data + design. Enter the merchant, the bridge between planning and design. The merchant has to bind the team together so that the flow back and forth between data and design is fluid. They learn from one another.

Peter Charness
Trusted Member
3 years ago

Organizational silos fed by system silos which create data silos owns a good portion of the problem, but let’s face it — for most retailers, the customer shopping in-store is anonymous and there is no simple way to add the in-store purchases to a customer’s profile and history, or to aggregate that anonymous customers information into useful cohorts.

Brandon Rael
Active Member
3 years ago

It’s fascinating and, at the same time, somewhat surprising to see that organizations are struggling with leveraging the richness that data and analytics provide. Unfortunately, some legacy retailers are challenged with siloed operations, disparate data sources, legacy processes, aging technological infrastructure, and most importantly, are lacking the agile, digital-first mindset to truly take a data- and analytics-first approach to their business strategies.

Today’s commerce world requires a data and analytics approach, with one version of the truth and an agile strategy to operationalize against these invaluable customer insights. To win in today’s environment, companies need to centralize their data, run their businesses by exception and, most importantly, have a holistic view of the channel-less customer to provide hyper-personalized experiences.

Gary Sankary
Noble Member
3 years ago

In my experience there are several disconnects between the data scientists and their insights and the business and their ability to use those insights. Execution systems have not kept up with the pace of innovation from data science. In retail especially automation tools that the execution teams use to do their jobs have not been successfully integrated with data tools, and when they have, they aren’t updated as new data sources become available. The speed of innovation and the introduction of new types of data, especially for customer analytics and human movement has created a bit of chaos in the industry. It’s difficult to build workflows when the source data changes constantly. Leaders become champions of their specific tools and often they don’t align at an enterprise level, this complicates the situation.

I strongly believe that the bias should be towards tools that can unify disparate data and allow interrogation of that data in a consistent manner, and in a way that is accessible across the enterprise is a top priority. I also believe a sharp focus on the business needs that these insights are going to inform and support is necessary to ensure the investments in BI and data are aligned with company strategy.

I used to like to ask my teams to simplify their requirements using the “I want too… So I can …” model. That way we could focus on capabilities and execution instead of getting caught up in the the latest and greatest, and forgetting what tactics we were trying to enable and what strategies we were supporting.

Jeff Weidauer
Jeff Weidauer
Member
3 years ago

There are many reasons why actionable insights are not available to retailers, even though they are awash in data. But the underlying problem is a lack of investment in the tools needed to properly gather and mine the raw data, coupled with a lack of understanding by leadership.

Chuck Ehredt
Member
3 years ago

Competencies are the biggest barrier. This includes the will to get things done, ability to overcome legacy methods of doing things, and lack of accountability. The technology, cost, or skills are no longer barriers at all. Therefore, any organization still suffering from these problems has too many in-competencies and not the right competencies.

In fact, these in-competencies are a leading indicator of future failure as a business altogether.

Shikha Jain
3 years ago

Very often companies devote time and energy to stitching together different data sources with the hope of developing a “single source of truth.” This in itself is a daunting task and, as a result, can sometimes be abandoned if companies think that it is a low-value task not tied to hard dollars. Instead, start from the end, what are the KPIs and objectives that are important to track and manage? Then work backwards to figure out what data you need.

Yogesh Kulkarni
3 years ago

I think organizations don’t think of data as a strategic asset at the point of capture or creation of data. Often times, disparate systems capture that data, users at the lowest end of the totem pole just do enough with the data to get by in their daily business. When the time comes to integrate it for something like Customer 360, you will often find that data is landlocked or wasn’t captured properly and isn’t of the required quality. I think we need to think of data properly at its inception to get around some of the challenges mentioned here.

DeAnn Campbell
Active Member
3 years ago

In my experience, the difficulty is not in gaining the insights, it is in getting the entire organization to align on and agree to the actions to act on those insights. I think the pull to inertia is incredibly strong, especially in mature organizations, and the force it takes to break out of the gravitational pull of tradition is more than many company executives are willing to take on.

Gene Detroyer
Noble Member
3 years ago

Data is simply meaningless numbers unless it is put in a form that makes it easy to read and understand. Though not retail, the following takes hundreds of thousands of data points, if printed out, would be impossible to read. It presents them in a way that makes ultimate sense, provides learning and gives a base for the future. Health and wealth in 200 countries over 221 years.

Mary Pietsch
Mary Pietsch
3 years ago

Some companies are doing it well, but not all. I see many investing heavily in the IT side, but not the people side. AI can go a long way, but it is a point of entry, not the final result. Too many are looking for an automated one-size-fits-all quick solution, but to be truly successful, analysis is an iterative process. The too often ignored need is the human contribution to make sure the right questions are being asked to create the necessary ROI, to understand that there probably isn’t one silver bullet target, and to know how to further mine the data to get to the optimal result.

James Tenser
Active Member
3 years ago

Sure data-derived insights are tough to obtain. Even tougher to apply to business decision-making. There are many wise comments in this thread, about “boiling the ocean,” the fallacy of the “single customer view,” “data silos,” and “organizational barriers.”

I think it comes down to what I call The Marketing Metrics Fallacy, which states: “Just because a quantity is highly measurable doesn’t mean it’s highly meaningful.”

Yes this is an aphorism, but it conveys an all too common systemic error that organizations make when confronted with many sources of big data. They expend too much effort measuring what is easy (low insight value), leaving too few resources to define and pursue what is important (breakthrough insight value).

In short, too many organizations suffer from data analytics cowardice. Getting to a number (any number) is prized above getting to understanding, so nobody is willing to take chances. This is a cultural failure that requires intervention from top management (and yes, thoughtful implementation of AI decision-support tools).

Ananda Chakravarty
Active Member
3 years ago

I’m sure I’m repeating thoughts here, but it warrants reinforcement. Garbage in – garbage out. Data and data governance is such an important part of decision making, yet it is routinely pushed back due to cost, misalignment, lack of expertise and so on. The concept has in many cases been the challenge of taking disparate systems (with very different purposes) and attempting to integrate these together in a meaningful way to access data for specific goals and objectives.

The best data transformations and insight producing solutions have self imposed limits. These systems also don’t answer all the questions. For longer-term decision making, boiling the ocean isn’t an option and teams need real data science experts to pull the right data to answer the right questions. Too often retailers just don’t have the internal expertise to develop more than fancy dashboards that can truly highlight and break out meaningful data. Compound that with the enormity of data available, establishing the right data points becomes a challenge in itself. Competencies are critical and even data scientists with PhDs may be insufficient.

Kim DeCarlis
3 years ago

Creating a single view of customer data is the goal for organizations across business segments — well beyond retail — but this is easier said than done for three main reasons: silos, talent and skewed data.

Unfortunately, customer data lives in many systems — the CRM such as Salesforce, the loyalty system, the payment system and the website analytics tool. These systems are often owned by different groups which results in silos that are hard to work across and data that is difficult to correlate. Finding skilled data analytics talent is the next challenge. People, with the skills to correlate multiple data stacks and understand the right questions to ask to help a business get from data to insights, are hard to find. Skewed data — particularly from websites and web applications — is the final challenge.

As e-commerce continues to be a primary channel for retailers, their website becomes a primary brand experience, with more traffic online than to most brick and mortar stores. But with human traffic comes automated bot traffic — to scrape content pages, hoard inventory and try to take over accounts. Since many websites have over 50% of their traffic coming from automated bots, decisions about promotions, product popularity and campaign impact can lead to erroneous conclusions and over- or under-investment.

So with these challenges, what’s a retailer to do? First, empower a data czar to bring together the silos and give this role the budget, authority and resources needed. Second, invest in internal data analytics skill-building classes, and consider sponsoring programs with local colleges and universities to build a talent pipeline and feed internships and new hires. Finally, make sure that you have the right protection tools for your websites and web applications to ensure that you are making decisions based on data from people, not bots.

Joel Rubinson
Member
3 years ago

I do a lot of work in this space and have numerous thoughts.

First, let me say that insights without predictive value are empty calories. You must have an eye towards prediction or activation for an insight to really have meaning. Secondly, data used for targeting and any form of differential marketing will pay off IF (that’s if and only if) you have the right principles for determining marketing responsiveness.

An example … many marketers (include retailers) think incrementality mostly or completely comes from non-buyers. WRONG! Advertising to the vast pool of non-buyers of your brand represents unprofitable ad spending. The great majority of non-buyers have no interest in your brand and will not respond to your advertising. Response comes from the Movable Middle … those with a 20-80% probability of buying your brand (or visiting your store). Direct disproportionately more advertising to those consumers and you will create a virtuous cycle of profitable growth. If data are used in THAT way, the payoff can be huge to a well organized and integrated first-party database.

Brian Cluster
Member
3 years ago

Disparate data sets, poor data quality, often a multi-department fight over who owns the data, what it means, and how it’s used are common challenges with customer data. You can work on asking the right questions and bringing in more analytical talent but if you don’t have a single trustworthy view of the customer, it will be wasted. This challenge requires C-Level sponsorship to overcome and investment in the data foundation of the organization. There is a real and opportunity cost to this issue. They need to answer what is the ROI, how is it measured and who is going to pay?

Liza Amlani
Active Member
3 years ago

Have you ever gone to a carnival or county fair and played whack-a-mole?

Consider that a retailer’s customer data exists in different buckets across the organization and contains multiple records of a single customers. A retailer then never reaches that holistic, single view of the customer that can help them make better decisions.

Essentially, they are hitting a moving target, constantly chasing the mole with the hammer but never hitting the mark when it comes to using that data effectively.

Even if retailers have access to customer data, more often than not, merchants have no access to it. Planners don’t have access to it. And most of the time, retailers have the data but have no idea what to do with it. There are no data scientists to decipher the data, to gain insights from the data and pass these insights to the teams making decisions directly impacting the customer.

Having one view of the customer is essential. BUT if you don’t have the right teams in place to understand that data and pass along those insights to the right people … the data will mean nothing.

Patricia Vekich Waldron
Active Member
3 years ago

Turning insights into action is a decades-old problem that stems from top-down commitment (funds, ownership and execution) to managing disconnected data, implementing intuitive tools and putting information close to all decision-makers. There are technologies, tools and expertise available to change this situation.

Casey Craig
3 years ago

In 2019 I was engaged by a tier 1 retailer to develop and prioritize predictive analytics use cases. My team spent weeks preparing for a three day workshop with key business stakeholders. Two hours into the workshop, one of the key business users stood up and said “these are all great and would be very impactful, but what I really need is access to our customer data without having IT always involved.” The group unanimously agreed. Easy access to trustworthy, clean data by business users is step 1 in realizing value and gaining actionable insights.