How far should retailers go with the use of computer vision?
Cleber Ikeda is Investigative Analytics and Intelligence Director at Walmart. Any views and opinions expressed herein are personal and may not represent those of Walmart. The content of this article does not contain nor reference to confidential, proprietary information of Walmart.
Retailers have found plenty of use cases for computer vision (CV), from surveillance to sentiment analysis to line management. The process of capturing, storing and analyzing shoppers’ images can generate great insights and outputs and, at the same time, impose relevant ethical challenges for retailers.
One of the most popular CV applications is the combination of high-definition cameras and artificial intelligence (AI) to detect customers who have missed scans at self-checkouts in stores. Another use case is the application of artificial intelligence and augmented reality tech for trying on clothes. The primary risks of these technology uses are inefficacy and inaccuracy due to unbalanced training data sets. Serious research in the area has shown, for example, that CV technology did not perform as accurately for people with darker skin tones compared to those with lighter skin. The gap between intersectional groups (e.g., dark-woman versus light-man) was demonstrated to be even greater.
Sentiment analysis based on facial analysis can prompt personalized in-store advertising or provide inputs for customer satisfaction. Personal information like biometrics, however, must be protected and consent must always be requested. Retailers must promote a culture of transparency and make customers aware of the technologies that are being applied to collect, process and use their data.
CV has also been deployed to support store staff by managing lines and aisles in a way that makes the shopping experience more pleasant and frictionless. In the pandemic era, it can also help retailers control store capacity and redesign space to preserve minimum distance among shoppers. The ethical aspect to consider here is that technologies must not use customers’ data to track location and activities beyond the specific purposes they were designed for.
Amazon.com and Microsoft in 2020 decided to halt the sale of their facial recognition technologies to law enforcement agencies. IBM announced it would exit the business completely.
Those decisions were taken in a context of increasing protests against police truculence with Blacks in the U.S. that put pressure on legislators to create laws that clearly regulate the technology. Though the dialogue on the matter has increased worldwide, there has been little progress made on regulations that could provide legal grounds for the development of such technology while protecting citizens from the potentially harmful consequences of its use.
- Walmart is using AI-powered cameras to prevent theft at checkout lanes – The Verge
- Virtual Try-On Is More Than A Pandemic Trend And These Brands Are Reaping The Rewards – Forbes
- Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification – Proceedings
- A.I. Can’t Detect Our Emotions – OneZero
- Microsoft won’t sell police its facial-recognition technology, following similar moves by Amazon and IBM – The Washington Post
- Can facial recognition outlast its bad press? – RetailWire
DISCUSSION QUESTIONS: What do you see as legitimate use cases for computer vision and facial recognition technologies at retail? What ethical demands are placed on retailers that make use of these types of technologies?
Join the Discussion!
17 Comments on "How far should retailers go with the use of computer vision?"
You must be logged in to post a comment.
You must be logged in to post a comment.
Principal, Retail Technology Group
Legitimate use cases for computer vision are identification, managing queues and the like. However even these uses expose retailers and customers to hacks, and then even the best of intentions are challenged.
Principal and Founder, Retail Strategy Group
From loss prevention to evaluating physical footfall, there is a real opportunity to leverage the use of computer vision.
Fit and sizing solutions through VR, AI and computer vision could be the future unlock for challenges many brands face today.
I’m somewhat confident that retailers are addressing ethical risks as there are many regulations in place protecting customer data privacy. With the spotlight on Google, Facebook, Amazon, etc., there has been visibility and transparency on what these tech companies have access to and what they do with all of our data. I would only hope that retailers and brands are following regulations but there is always a risk. Many retailers are putting data centers of excellence in place to help protect consumer data which is great to see but we need more guardrails in place to ensure we are protected.
Consultant, Strategist, Tech Innovator, UX Evangelist
Computer vision and other implementations of AI are all acceptable IF — the true use case is clearly defined, the data use/retention/sharing policy is honestly defined/shared, and consumer participation is opt-in and opt-out. As soon as any of these criteria are not maintained, problems and risks are inevitable.
I have built CV systems and, in one instance, although it worked nicely it would not recognize the primary person signing our checks at the client — and he was Caucasian with red hair. So these systems are fallible in nearly every direction, at least to a small degree. Deploying them for critical tasks is an invitation for trouble and shoplifting prevention is precisely one of those potentially troublesome areas. As with anything else, it’s a risk/benefits analysis that must include reputational risks in the decision making.
Industry Consulting, Retail, CPG and Hospitality
Some of the best use cases for computer vision are when it’s leveraging aggregated, anonymized data. Examples include queueing time, emotive responses to shelf displays, order bundling inside a fast food environment and BOPIS pick-up in parking lots. CV can be extremely powerful and can deliver new insights never before seen (pun intended), but all involved need to be extremely vigilant to avoid individual privacy violations of current laws (CCPA) and future ones that are in state legislative houses today.
Marketing Strategy Lead - Retail, Travel & Distribution, Verizon
Consumer privacy rights should be respected when using computer vision for facial recognition use cases. Two of the most potentially common use cases for computer vision are virtual mirrors for fitting rooms and cashierless checkout/autonomous shopping. In these scenarios, it is important that retailers get customer consent with opt-in requirements. Using computer vision for store traffic and queue control should be conducted with anonymous customer data that doesn’t infringe on customers’ privacy.
Chief Strategy Officer, Hoobil8
There is a fine line between convenient and creepy. Consumers place great trust in brands and retailers when they opt in to biometrics in exchange for a higher level of personalization and convenience. But when the technology is used by outside entities, it breaks that bond of trust, which the retailer may never regain through no fault of their own. I’m all for catching real criminals, but would prefer it be done in ways that take the retailer out of the middle.
Managing Partner, Advanced Simulations
If you are collecting individual images of a person, you’re going to screw it up eventually. It’s one thing to understand queue management, to look at stocking levels, and track movement patterns – none of that needs an individual face. Once you step into individual facial recognition/body imaging, you’ve crossed a line that opt-in will never repair once it blows up. This is a PR disaster waiting to happen and I doubt the benefits will ever outweigh the potential cost.
Principal, Cathy Hotka & Associates
When it comes to technology like this, ask yourself if your grandma would be OK with it. And she wouldn’t.
Vice President, Research at IDC
Most stores already engage in collecting images of people on camera to catch what’s going on in the store as well as protect the store from theft and other factors. Security footage is a common tool that has been used for many years. High value stores and businesses even have people continuously watching video. London has over 691,000 CCTV cameras across the city. Paris has 42,500 cameras watching streets and public spaces. Even NYC has over 58,000 cameras. The PR issues occur when it becomes individual and more importantly when we treat one individual differently from another.
Retail Strategy - UST Global
Predicting the actions/reactions of a single individual (vs a cohort) is highly problematic, and not just because of privacy issues. If retailers stick to those use cases that only require aggregated input, or non-human, non-facial input (like “is the shelf full?”) there is great potential. (Cashier-less check out which is also a great use case, doesn’t really require facial recognition). So avoid working one face at a time, and don’t store those personal images.
Board Advisor, Light Line Delivery
Investigative Analytics and Intelligence Director, Walmart
Shawn, thanks for your great comment. I cannot agree more with you on two aspects: 1) the need of diversity over the AI lifecycle; and 2) the self-assessment on whether certain types of technologies should be deployed, given the problem it aims to solve — sometimes, ethical questions pose important barriers, as the tolerance for false positives you mentioned. Thank you!
Vice President, Research at IDC
President, Rubinson Partners, Inc.
I think that facial recognition for security purposes would be a deterrent to smash and grabbers and make me feel SAFER as an honest shopper. As far as value-add services go (like virtually trying on clothing) that would require terms of service being clearly posted (separate spaces in the retail outlet would be best) with agreement statements.
CEO, New Sega Home
AR applications are key to helping minimize returns and helping customers visualize themselves in and with the product. Particularly in the home space, these have become increasingly impactful for purchasing items such as furniture. The more this tech is adopted in other areas, like apparel for virtual try-on, the more the tech can expand. As for the facial recognition element, it really will need to rely on customer consent and transparency. Self-checkout is a major consumer want, so tying any facial recognition plan back to the benefit of a consumer should minimize any resistance.
Chief Amazement Officer, Shepard Presentations, LLC
LoyaltyCIO, Client Service Associates
The way I rationalize the acceptable uses of technology is to ask myself if the world’s best retailer had his/her eyes on the consumer and reached the same merchandising conclusions, made the same transaction decisions, etc. then I’m OK with it.
Where I have great discomfort is in the recording and subsequent replay of audiovisual recordings. I believe live camera in real time is OK, but don’t agree with preserving recordings of the consumer for perpetual replay and reductive analysis of their actions and behaviors. I anticipate the USA advances the “right to privacy” legislation greatly limiting government and commercial businesses right to record and preserve audio/visual recording of citizen/consumer activities.