What if artificial intelligence is biased?
Artificial Intelligence, which promises to eliminate the inefficiencies that come from subjective human assessments and assumptions, is facing heightened concerns that the technology includes hidden biases.
Amazon.com, for instance, early last year stopped using a recruiting and hiring tool powered by AI because it was biased against women.
The tool observed patterns in resumes of successful hires over a 10-year period to rate the resumes of prospective hires, according to Reuters. Because many of the hires were men, it showed a preference for male candidates and favored resumes using more masculine terms. Resumes including the word “women’s” as in “women’s chess club captain” and candidates who attended all-women’s colleges were downgraded.
Reuters noted that even though the algorithms were edited to mitigate those biases, the tool was scrapped because Amazon wasn’t sure if it wouldn’t devise other ways to be discriminatory.
A number of other studies, including one showing a similar tendency in LinkedIn’s search engine, have likewise found biases for groups underrepresented in AI datasets. Google was criticized a few years ago after its image recognition algorithm identified African Americans as “gorillas.”
“AI algorithms are not inherently biased,” Venkatesh Saligrama, a professor at Boston University’s Department of Electrical and Computer Engineering who has studied word-embedding algorithms, told PC Magazine. “They have deterministic functionality and will pick up any tendencies that already exist in the data they train on.”
Because deep-learning software perceives patterns in human decision making, AI algorithms can also pick covert or overt biases from their human creators when are written.
The feedback loop from a machine learning system, particularly as humans increasingly rely on their AI’s assessments, could create more biased data that algorithms will then analyze and train on in the future.
While programmers seek to reduce prejudices, calls for diversity are being heard in the technology sector where the algorithms originate as well as greater transparency and accountability.
AI-bias is a significant concern in the health care and law enforcement sectors as well. It’s less known how any covert biases in algorithms would impact the personalized experiences AI promises to deliver for retailers.
- Artificial Intelligence Has a Bias Problem, and It’s Our Fault – PC Magazine
- A.I. has a bias problem that needs to be fixed: World Economic Forum – CNBC
- Amazon scraps secret AI recruiting tool that showed bias against women – Reuters
- When Robots Turn Racist: Artificial Intelligence Not Immune to Bias – The Globe Post
- AI is the future of hiring, but it’s far from immune to bias – Quartz
- IBM AI OpenScale: Operate and automate AI with trust – IBM
- Discriminating algorithms: 5 times AI showed prejudice – New Scientist
DISCUSSION QUESTIONS: How big a concern are latent biases embedded in artificial technology? Are retailers using artificial intelligence equipped to uncover and fix these biases?
Join the Discussion!
27 Comments on "What if artificial intelligence is biased?"
You must be logged in to post a comment.
You must be logged in to post a comment.
President, Integrated Marketing Solutions
Beauty and bias typically lies in the eyes of the AI creator. Worse yet, a “neutral” algorithm will develop bias if trained on data that has particular trends and patterns. The only hope is maybe to develop AI designed to be “neutral” and uncover bias trends in decisions. While AI can be amazing technology to automate tasks and operations, we may not ever be able to fully “trust” AI to make significant decisions about humans.
Founder, CEO & Author, HeadCount Corporation
Bias can be an unintended consequence of applying AI algorithms as the examples in the article show, and so practitioners must remain vigilant in reviewing and monitoring outcomes generated by these systems. AI and machine learning are still in their infancy and so to is retailers’ ability to uncover and fix biases. Like Amazon and others who are actively using and applying AI, this is still a work in progress.
Consultant, Strategist, Tech Innovator, UX Evangelist
The myth around AI is that a neutral “machine” independently develops its own intelligence. In reality, AI and machine learning, even instances that are to a strong degree “self-educating,” originate from human-derived algorithms. Humans are flawed and often from their early intervention into creating/training artificially intelligent systems, can introduce bias. I listed it as a probable issue with Amazon Scout in my article “Amazon is Scouting the Wrong Approach to Improving Product Discovery.”
I doubt that many AI systems anywhere are evaluating for much more than obvious bias because they are designed to enhance biased outcomes: meaning, designed to create outcomes to support a business case or need, which is a purposeful interest.
Retailers do need to use AI cautiously for both bias issues that can affect and offend customers and can affect profitability via missed opportunities and wrong assumptions.
Principal, Retail Technology Group
Almost every technology and technology-based decisions have some unintended consequence. Since AI software does nor develop itself but, rather, it is developed by people, there is always the probability that AI will contain some human bias, whether intended or not. Of course, these unintended errors will be discovered — eventually, and corrected — hopefully.
You might want to read “Weapons of Math Destruction” regarding the question of discovering the errors. In many situations, errors are virtually impossible to detect.
Managing Director, GlobalData
As machines do not, as of yet, have volitional consciousness they are not biased by themselves. Bias comes from human programming or from machines observing human patterns of behavior which exhibit bias.
Sometimes this bias is useful. If I, as a consumer, have a bias towards buying certain products then that helps retailers target me more effectively. Sometimes, the bias is less helpful such as in recruiting or profiling people when there is no need.
They key thing is to ensure that algorithms are auditable and to review outcomes and to ensure that these comply to ethical frameworks and policies. Given that most retailers haven’t really fully got to grips with AI itself, I think exploring the ethical dimension is far from advanced.
VP of Strategy, Aptos
Everyone should be highly concerned about latent biases. We don’t fully understand our own biases, and without that understanding, we can never assume that AI won’t “learn the wrong things.” This is why I think there will be an “AI 2.0” in our future – one that has been developed specifically to make its assumptions or lessons learned clear to an end user, who can then better monitor the things that AI learns. The reality is, we WANT AI to learn “ideal” behaviors, but we’ve released it into a world far less than ideal. And it will do with that data what AI does — draw conclusions and apply them going forward.
Founder, Branded Ground
Beautifully stated Nikki. As always.
Vice President of Marketing, OrderDynamics
When using AI, we all need to be aware of the potential of bias. Then watch for it in the results. This second step is particularly important to avoid a future problem. If there is a bias that is not appropriate then it needs to be fixed. The danger is NOT having this feedback loop. If missed, eventually it may come up as a public embarrassment or brand-eroding incident.
Right now I have not perceived any self-correcting or bias detecting elements in AI software. It’s a call to the software vendors vendors. It isn’t just a responsibility. It is a risk-reducing feature that customers (retailers) will want.
President, The Ian Percy Corporation
You’ve provoked my thinking, Charles. Even “bias detecting” AI software will be biased and will perpetuate a never ending spiral of “bias-ness.” We’ve learned this from software production generally. Software fixes account for almost half of all software faults. Will there ever be fault-free software? Or unbiased AI? I’m working on the former but am not quite so sure of the latter. Appreciated your contribution.
President, founder and CEO Interactive Edge
Garbage in — garbage out. If the data the algorithm is being trained on is not somehow scrutinized or scrubbed, it will likely be based on bias and bad assumptions.
Board Advisor, Light Line Delivery
The view of the world an AI has, is not the world; it’s its training data. The inscrutable nature of AI makes it difficult, or impossible, to understand prediction logic and to pinpoint flaws of bias; this is still very much a research problem. However, this is where I believe that diversity of thought among applied data scientist, data engineers, and ethics validators will pay dividends.
Scientific Advisor Kantar Retail; Adjunct Ehrenberg-Bass; Shopper Scientist LLC
Artificial Intelligence (AI) is not really “intelligence” at all, but rather simply information, hidden in vast skeins of data, about what IS, that is not readily NOTICED by our limited minds. However, it is doubtful that that data accounts for all the biases produced by programmers from demonstrated human monocultures, politically, socially, etc.
Some of the biases may be historically justifiable, but maybe not, too. It’s a mess. But THAT’s the human race, that is on a largely unrecognized positive trajectory. See: Steven Pinker, “Is the World Getting Better or Worse?“
Co-founder, RSR Research
I agree with Neil’s comments. AI, at the moment, is another magic bullet that is perceived to allow us to alter retail. It’s not clear that this is actually widely in use.
It will be an interesting problem to solve once it has been deployed and is mature enough to truly support decision-making at an individual level, but meanwhile I’m happy to see it getting closer to an accurate weather forecast.
President, The Ian Percy Corporation
President, The Ian Percy Corporation
As usual, a further thought came to me after I submitted my comment. I referenced the work of the great masters. We heard that an AI-generated portrait just sold for $432,500. So should it be listed along with the Mona Lisa or The Man With The Golden Helmet? Or can you get another copy by hitting “enter?” What really is the difference? Understanding that may help us rethink retail.
Founder, CEO, Black Monk Consulting
Let me start by questioning whether or not anyone — other than zealots — believes AI “promises to eliminate the inefficiencies that come from subjective human assessments and assumptions.” AI recognizes patterns in data. That’s like thought, but it isn’t totally objective thought. So if there are biases in the data, they will be mirrored in the analysis. These are not independent systems, and they are only as objective as the developers who set them up and the data fields they are analyzing. On any given application they aren’t inherently any more or less biased than a human being, just faster and less prone to computational error.
Ethics are usually playing catchup with tech developments. But what is all that development for in the first place? (Paraphrasing a quote from The Wind Singer: “If everything we do is in order to get somewhere else, when’s the end of it all?”) What’s coming to mind: Walmart was developing facial recognition tech to catch shoplifters and detect dissatisfaction last year. I wonder what an AI would do with millions of shoppers’ unhappy/happy faces? What would it find and recommend based on its instructions to detect dissatisfaction (and what a “dissatisfied” face looks like from the POV of the race, gender, cultural background of the developers)? How that would impact the demographics they market certain products to or even what a “good worker’s” face should look like? No easy answers here other than rooting for purposeful AI developed intentionally within as diverse and crowded a room as possible.
Retailer hiring algorithms may be where the most egregious AI errors lie. The excellent book “Weapons of Math Destruction” by Cathy O’Neil discusses hiring algorithms for their secrecy and the complete inability to improve the algorithms by testing.
For proof, we have the recent choice by Amazon to stop using their AI hiring algorithm because it showed bias against women.
But I don’t think the problem is fixable. These biases are hidden and nearly impossible to test. AI can ONLY be led to succeed when the algorithms are trainable by evaluating results. But in hiring software there’s no way to train with “person X was hired elsewhere and performed outstanding work.”
It seems to me that AI cannot be completely unbiased because the inherent objective to building a database is to make determinations. If the output is biased, then the information used to create that database must also have some kind of bias.
Managing Partner Cambridge Retail Advisors
AI is not biased, it is the data that is biased and the humans that created the data. If human decisions have been biased in decisions that impact data, then the output from AI will perpetuate that bias. Garbage in, garbage out. That old phrase is as appropriate today as it ever was. Systemic bias needs to be scrubbed out of the data so neural networks will apply the proper framework of algorithms to make AI unbiased.
Retailers need to be aware of the potential mistakes that AI can make based on historical data and biases and monitor the results of AI to help catch faulty logic. Humans are not perfect and neither is AI.
Global Retail & CPG Sales Strategist, IBM
The fact remains that machines learn from the data they ingest. The challenge to “feed” unbiased data will lie squarely upon those humans who augment their intelligence with these machines. Data scientists need to be able to balance the inputs to ensure desired unbiased outputs.
Retail Strategy - UST Global
As already noted, the bias is likely in the data, which influences the algorithms. Next logical question, if AI “judgement” is based on historical patterns, where does innovation come from? In merchandising, the challenge has always been the self fulfilling prophesy. Don’t retire the creative types in your organization yet …
Vice President, Research at IDC
What a great set of thoughts already. Big concern — yes. Biases, if uncovered, can topple brand value rapidly, especially as it dehumanizes engagement with customers. We are still at the early stages of AI and the problem that occurs is that our bias corrections are not objective enough to be bias free, so we just introduce another set of biases into the training data or apply fixed business rules on top of results, diminishing the value of AI tools to uncover things that humans wouldn’t normally find easily. AI is quite promising, but we’re not at a point where the concerns are well founded nor can we deliver results that can be used readily in retail — except in select cases such as logistics or demand planning.
Co-Founder and CMO, Seeonic, Inc.
Like any new science, AI is going through learning lessons, including latent biases. Researchers are working on many techniques to eliminate the bias. By studying the results of AI, biases will typically be found if they exist. Retailers are probably not performing much analysis to determine if biases exist as any AI bias will have little impact on people. The biases may cause the retailer to have the wrong assortment or products in store at the wrong time, nothing that will effect the shopper personally.
Chief Amazement Officer, Shepard Presentations, LLC
There should be a big concern about AI being biased or making bad decisions based on biased programing or human behaviors that cause bias. It has been my position for several years that AI is not ready for advanced decision making. It can help, but it can’t be used (yet) for big issues and decisions. It’s perfect for lower level support and functions. In the HR world, it can be used to create a streamlined application process, help with repetitive tasks and more basic functions. To use it as the sole decision maker for hiring is too soon, if ever at all.
President, What Brands Want, LLC
AI is still relative new and still being developed as a solution. Inherent biases are hard to unearth with people which means AI-solutions are bound to have bias (even if vetted). The key is to find and understand how the biases impact the end result.