Will retailers use facial recognition tech to reshape store layouts?
Photo: RetailWire

Will retailers use facial recognition tech to reshape store layouts?

New university research explores how grocers can take advantage of in-store cameras employing advanced artificial intelligence (AI) technologies to read facial expressions — raising an eyebrow, opening eyes, smiling — to enhance store layouts.

“Emotion recognition algorithms work by employing computer vision techniques to locate the face, and identify key landmarks on the face, such as corners of the eyebrows, tip of the nose and corners of the mouth,” said Dr. Kien Nguyen of Australia’s Queensland University of Technology, in a press release.

He added, “Other behaviors like staring at a product and reading the box of a product are a gold mine for marketing to understand the interest of customers in a product.”

Researchers note that while use of facial recognition in the retail setting is “still controversial” due to privacy concerns, the footage data can be de-identified or made anonymous so that customers would be examined only at an aggregate level.

AI has also become widely used with in-store cameras for shoplifting prevention and increasingly used for applications such as pay-with-your-face, check-out free grocery stores, and visual and voice search, according to the study.

However, the researchers stated that currently, the conventional approach to designing store layouts is based on a “passive reaction” to customer behavior, such as basing decisions on sales data.

Along with understanding emotions through facial cues and customer characterization, layout managers could employ heatmap analytics, human trajectory tracking and customer action recognition techniques to inform their decisions.

The study states, “Importantly, the conventional design process does not reflect (1) how customers actually navigate through store aisles, (2) how much time customers actually spend in each section, and (3) what visible emotion (e.g., happiness) customers exhibit in response to a product.”

A 2012 article from Bloomberg, however, was already weighing the potential benefits to store layouts, window displays and promotions from in-store facial-recognition technologies against the legal and ethical ramifications of gathering personal data with or without consent.

Privacy concerns haven’t abated. In July, Kmart and Bunnings in Australia were forced to pause the use of facial recognition technology in their stores they claimed was being used for security purposes amid an investigation from Australia’s privacy regulator.

BrainTrust

"I recommend that retailers find less controversial, less legally risky ways of acquiring the insights they are seeking."

Mark Ryski

Founder, CEO & Author, HeadCount Corporation


"Bad science with bad assumptions. Emotion recognition works when the emotion is somewhat extreme. Who has an extreme reaction to a shelf set?"

Dr. Stephen Needel

Managing Partner, Advanced Simulations


"Maybe that extra staff person could keep the shelf full, preventing that look of disappointment on the customer’s face…"

Peter Charness

Retail Strategy - UST Global


Discussion Questions

DISCUSSION QUESTIONS: What do you think of the potential benefits to store layout design of using facial recognition technologies? Can privacy concerns be overcome, even with assurances that shoppers will only be profiled anonymously?

Poll

27 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
DeAnn Campbell
Active Member
1 year ago

What a shopper buys doesn’t always reflect what they really wanted. We are creatures of habit, but at the same time we constantly seek new ideas and products to love. In the absence of sufficient information we will usually default to products we are familiar with. The beauty of using heat mapping and AI to see a customer’s movement through the store is it reveals these hidden desires and products that interest them and shows the retailer clearly where they need to strengthen their merchandising displays, product information or position in the store to help shoppers leave with purchases they feel happy about choosing, not mildly frustrated or wondering what they should have bought instead.

Mark Ryski
Noble Member
1 year ago

Sentiment analysis using facial recognition is still largely unproven, but the backlash against the use of facial recognition is gaining traction. I argue that the benefits of using facial recognition in the way described in the article can be acquired without the use of facial recognition. Facial recognition – and other biometric data – is a hot potato, and I recommend that retailers find less controversial, less legally risky ways of acquiring the insights they are seeking.

Ken Morris
Trusted Member
1 year ago

Traditionally, retailers have pegged their strategies to the 4 Ps of product, place, price, and promotion. This technology adds the fifth and most important one, which is people. Store layout can show movement and compare traffic at various facings or end-caps where the same SKU is sold. But retailers should be able to look at data from multiple dimensions. So I think the real value, or a lot of additional value, will come from pairing this technology with more active analysis of planogram usage. Instead of setting a planogram and forgetting it, you can now capture your shoppers’ interaction with SKUs by leveraging this technology. It allows brick-and-mortar retailers to capture the type of information only available within e-commerce transactions. Also, privacy should not be an issue if the technology either can’t or won’t identify individuals. Approximate age, for example, is a useful data point without privacy issues.

Steve Montgomery
Steve Montgomery
Member
1 year ago

There are companies that use video to follow a customer’s path in a retail environment. They gather a great deal of data including how long they stand in front of a section, if they look up/down or look left/right and, if they make a purchase, what they buy, all without the danger of privacy invasion.

While there is no question many if not most people have given up a great deal of their privacy in exchange for some real or perceived benefit, I don’t think shoppers would do so in order that a store can create a better layout.

Georganne Bender
Noble Member
1 year ago

Sorry, still creepy.

Rich Kizer
Member
1 year ago

My fear is of basing ANY solutions or merchandising strategies on facial recognition technology. It is brand technology and I believe early adoption could be fatal to selling specific items and the presentation of a retail environment as a whole. My advice; tread lightly.

Cathy Hotka
Trusted Member
1 year ago

Put a sign at the front door telling customers that your cameras will be watching their facial reactions to various products, and watch them visit your competitor instead. Try focus groups instead.

Mark Ryski
Noble Member
Reply to  Cathy Hotka
1 year ago

Exactly Cathy. If retailers choose to use this technology for my benefit as a shopper, then disclose it at the front door and let me decide if I’m OK with it. As we’re learning with mobile device location tracking, when given a choice most consumers will opt out.

Gary Sankary
Noble Member
1 year ago

There’s more downside than upside to this. No retailer or CPG company has come up with a clear benefits case for what they’re going to do with this data. What changes will they make, and how often will they make them? Redoing a planogram or changing a store layout is expensive and not something they want to do every time someone smiles differently or raises an eyebrow.

Customers already have privacy concerns. Putting cameras on displays will not create warm fuzzies, in my opinion. We’ve also learned that “we promise not use that data” is the modern “check’s in the mail” statement. It is just too tempting for retailers. Better to find other ways to gather data to influence assortments and store design.

Melissa Minkow
Active Member
1 year ago

This seems like tech for tech’s sake to me. I doubt the ROI on this will make it worthwhile. There’s too much margin for error and it’s also easy to tell which sections shoppers are eyeing just by asking sales associates what they’re seeing. I can’t see consumers being comfortable with this.

Ryan Mathews
Trusted Member
1 year ago

Facial recognition works best in a homogeneous population. Once you begin to address diverse populations the effectiveness of the AI begins to break down. For example, I live in a metro area with one of the largest Middle Eastern populations outside the Middle East, where many shoppers routinely shop with some version of a hijab or hajib. Also facial responses are cultural and AI doesn’t take that into consideration yet. Facial recognition also has several political problems. Privacy is one of them, but so are objections to the use of facial recognition technologies among people of color. All that said, the technology may be promising one day, but today I wouldn’t bet the store on it.

Gene Detroyer
Noble Member
Reply to  Ryan Mathews
1 year ago

Ryan, your comment about facial expression being cultural is spot on. When I teach negotiation in International Business, we review the various messages being sent via facial expression. Can you have “YES” meaning “NO”? You can and you do.

David Spear
Active Member
1 year ago

Facial recognition insights combined with a retailer’s existing data gravity suggest that there are massive learning opportunities for companies, especially, for marketers. Understanding a consumer’s pathing, emotional reaction to a promo, and linger/hover time are all primary insights that can unlock any number of strategies that could be employed in-store. The key lies in anonymized data and how well a retailer adheres to strict privacy guidelines with this data.

Mohamed Amer, PhD
Mohamed Amer, PhD
Active Member
1 year ago

Just because technology brings new capabilities doesn’t mean they are appropriate, let alone ethical. Retailers need to think twice before technology’s allure leads to misguided decisions. Privacy and data retention issues are one set of concerns; bias or poorly constructed AI algorithms are another. Any consideration must include a well-communicated, clear, and explicit opt in permission campaign. Today, the time is not yet ripe for detailed facial recognition-type technology in store aisles.

Doug Garnett
Active Member
1 year ago

This sounds incredibly silly. Studies of facial recognition AI show that it is highly inaccurate for interpreting emotions.

Peter Charness
Trusted Member
1 year ago

Which has the higher ROI — video analytics like this, or hiring some additional staff in-store? Maybe that extra staff person could keep the shelf full, preventing that look of disappointment on the customer’s face…

Mark Self
Noble Member
1 year ago

I do not know how you “merge” the lessons learned from facial recognition with the physical aspect of store layout. I think the hoped for benefits are not even close to enough to justify the costs. At least for now. You would have to add a virtual dimension in some way now, and I am not smart enough to figure out how that would add value.

John Karolefski
Member
1 year ago

Facial recognition technology may help retailers, but shoppers won’t like it. If they find out, shoppers will look for another place to do business.

Brandon Rael
Active Member
1 year ago

Considering the fundamental challenges that retailers are experiencing in a disrupted and inflationary global economy, any investments in technology innovations have to be purpose-driven and add value to either the consumer or the associates in service of the consumer. Those who have attended NRF in the past five years have seen the whirlwind of innovations, including RFID capabilities, magic mirrors, robotics throughout the sales floor, and the onset of the AR/VR revolution.

Facial recognition is another innovation that could potentially be a game changer, yet in this economic climate, we have to question the prioritization of these initiatives. There are also privacy concerns and discomfort in having our emotions and facial expressions analyzed and dissected by retailers. We are unfortunately getting closer to the relentless face and eye scanning highlighted in the movie Minority Report. Are we as a society ready for this?

Dr. Stephen Needel
Active Member
1 year ago

Bad science with bad assumptions. Emotion recognition works when the emotion is somewhat extreme. Who has an extreme reaction to a shelf set? I can’t wait to do controlled tests from the results of one of their studies.

Craig Sundstrom
Craig Sundstrom
Noble Member
1 year ago

What’s more likely, I think, is facial recognition technology being banned; or less drastically a series of highly publicized lawsuits turning it into the latest “won’t touch that with a ten foot pole.”

As far as the idea of anonymous facial recognition: that’s a joke, right? The term is literally an oxymoron.

Ananda Chakravarty
Active Member
1 year ago

While I don’t see facial recognition being used to lay out designs, it is useful for one key purpose — identity. Most retailers have no clue who is in the store. Autonomous stores have cameras everywhere and they serve a special consumer driven purpose. Lolli and Pops experimented with loyalty and facial recognition a few years back with limited success, so although the privacy concerns remain, there will be a cadre of customers willing to trade a snapshot recognizes who they are to avoid paying in a checkout line or earn actual loyalty rewards.

James Tenser
Active Member
1 year ago

In-store video surveillance is a lightning rod for privacy zealots, as it should be. Fold in the multicultural watch-outs others describe here and you have a formula for legal peril in numerous jurisdictions, compounded by less-than-reliable data.

Perversely, training the AI over time to attain greater accuracy in reading facial sentiment and individual demographics further compounds the promise of anonymity, while creating a tempting target for data thieves.

Just because it may be technically possible to use video analytics to monitor shopper behaviors in this manner doesn’t mean it will be acceptable in a free society.

There are other approaches to optimize space management, assortment, planogram and pricing decisions that won’t open retailers up to fines and lawsuits or create an overwhelming data security imperative.

Brad Halverson
Active Member
1 year ago

There are ways of scientifically measuring how thousands of customers shop a store, and design a good experience without being so intrusive as to follow faces, trying to determine intent. A retailer hoping to test this at minimum must have clear signage posted in store and explain how the data will be used. Customers get to decide if they want to support retailers who attempt this.

Oliver Guy
Member
1 year ago

Fascinating. The ability to be able to provide this kind of insight (anonymously) to merchandisers is amazing. The question organisations need to ask is are merchandisers ready for the insight and do they know what to do with it?
Extracting insight from one source like this is one thing — but combining together with other insights is where it becomes really powerful.

The ultimate will be able to make AI driven assortment and promotional adjustment change recommendations based on the combination of this insight with other details such as incoming supply. Doing this in near-real-time requires logic and rapid execution capability at the store level — few stores have the necessary infrastructure to allow this as yet — but will come eventually.

Anil Patel
Member
1 year ago

AI-based facial recognition is not a brand-new technology, it has been adopted across many industries. Retailers incorporating facial recognition to monitor and analyze customers’ behavior may also soon become mainstream. Concerns about privacy, in my opinion, don’t really matter much in this situation. Retailers will continue to use facial recognition tech if it helps them derive favorable results.

Mark
Mark
Member
1 year ago

It is doubtful whether facial expressions accurately determine the person’s true reaction or mental state. Not scientifically. Psychologists have studied this for decades now, especially Dr. Paul Ekman, who is a leading advocate. No one really knows or can know what the customer is thinking, in fact. Their facial reaction could have nothing to do with the visual product at all. This is how I shop. I am reflecting on all sorts of things irrelevant to the store. Moreover, I think customers in other nations with various cultural differences react in different ways for different reasons.
The idea of basing store layout on customer facial recognition is based on a deceptive assumption and may lead to false conclusions, misinformation that retailers will not be aware of. Beware of studies that claim too much, in any field, because their answers are hasty and inconclusive.