What is AI bias? It is what happens when an AI system treats people unfairly because of who they are — their gender, race, age or background — rather than what they have done or need. It is a pattern, showing up in hiring, courtrooms, face recognition and healthcare.
So how does a machine with no opinions of its own end up discriminating? Because AI learns from us: it is trained on our data, given our goals, and tested by people who share our blind spots. A skew in any of those steps becomes a skew in the system — applied to thousands of decisions a day.
Key takeaways
- AI bias is unfair treatment by an AI system, usually from skewed training data, a badly chosen goal, or testing that misses whole groups of people.
- Real cases are well documented: Amazon’s hiring tool, COMPAS and face recognition all show it.
- Bias is rarely deliberate: these systems learned the pattern from the data.
- Spot bias by asking who tested it, what it predicts, and whether its mistakes fall unevenly.
- No AI output should be the final word on a decision affecting a person’s job, liberty, health or money.
What is AI bias, in plain terms?
AI bias means an AI system produces systematically worse outcomes for some people than for others, for reasons unrelated to merit or need: a biased hiring tool rejects women more often than equally qualified men, and a biased face recognition system misidentifies darker-skinned people far more often. Every system makes mistakes; bias is when the mistakes follow a pattern, tracking a characteristic such as race or sex — or a close stand-in, such as a postcode or a first name. Bias can exist even when nobody meant any harm. A system can look neutral — it never asks for your race or gender — yet still discriminate through proxies that stand in for those things.
Where AI bias comes from
Biased training data. AI learns from examples. If they reflect past discrimination, the system learns it as a rule. Data can also be unrepresentative: a face recognition system trained mostly on lighter-skinned faces will be worse at recognising everyone else.
A badly chosen goal, or proxy. Sometimes the problem is what the system is asked to predict. One healthcare algorithm predicted how much a patient would cost, using cost as a stand-in for illness. Because less is spent on Black patients at the same level of illness, it concluded they were healthier than equally ill White patients.
Design and labelling choices. Humans decide what counts as a “good” outcome. A hiring tool that rewards CVs resembling past hires is, by design, a tool that rewards the past.
Testing that misses whole groups. If a system is tested mainly on one group, its failures on others stay invisible until launch — which is how face recognition with error rates under 1% for lighter-skinned men shipped with rates above 30% for darker-skinned women.
Real examples of AI bias
Each case below is documented in published research.
Amazon’s hiring tool penalised women’s CVs
In 2014, Amazon built a tool to score applicants’ CVs from one to five stars, trained on CVs from the previous ten years — mostly from men. It taught itself that male candidates were preferable. Reuters reported in October 2018 that it penalised CVs containing the word “women’s”, as in “women’s chess club captain”, and downgraded graduates of two all-women’s colleges. Amazon tried to correct it, but could not guarantee it would not find other ways to discriminate. The project was scrapped, and Amazon said it was never used by recruiters to evaluate candidates.
COMPAS: risk scores that favoured white defendants
COMPAS is a tool used in US courts to estimate the likelihood that a defendant will reoffend. In May 2016, ProPublica analysed its predictions for more than 7,000 people arrested in Broward County, Florida, in 2013–14, against what actually happened over two years. Black defendants who did not reoffend were wrongly labelled high risk 44.9% of the time, nearly twice the white rate (23.5%). White defendants who did reoffend were wrongly labelled low risk 47.7% of the time, against 28.0% for Black defendants. Overall accuracy was similar, but the errors fell in opposite directions. Northpointe, the tool’s maker, disputed the analysis, arguing the scores satisfied other measures of fairness. The dispute is instructive: there is no single agreed measure of fairness, and a system can look fair by one measure and biased by another.
Facial recognition: the Gender Shades study and NIST
In 2018, Joy Buolamwini and Timnit Gebru published “Gender Shades”, an audit of three commercial gender-classification systems from Microsoft, Face++ and IBM. They built their own balanced test set of 1,270 people, because the standard datasets were overwhelmingly lighter-skinned (79.6% of one benchmark and 86.2% of another). They misclassified darker-skinned women with error rates of up to 34.7%, against a maximum of 0.8% for lighter-skinned men. For the darkest-skinned women, error rates in two systems reached 46.5% and 46.8% — little better than guessing.

NIST confirmed the pattern a year later, at a much larger scale: its 2019 test examined 189 algorithms from 99 developers, using 18.27 million images of 8.49 million people. Most had higher false-positive rates for Asian and African American faces than for white faces — sometimes by a factor of 10 to 100. For one-to-many matching, used to search a database for a suspect, false positives were highest for African American women — a false positive there can mean being wrongly flagged as a suspect.
Healthcare: sicker patients scored as lower risk
In 2019, Ziad Obermeyer and colleagues published a study in Science of a widely used algorithm that decides which US patients get extra care, using future healthcare costs as a stand-in for health needs — an approach typical of the industry. At a given risk score, Black patients were considerably sicker than White patients. Fixing the bias would have raised the share of Black patients identified for extra help from 17.7% to 46.5%. The cause was the proxy: less is spent caring for Black patients at the same level of need, so the algorithm read lower spending as better health. Reformulating it to predict health needs directly removed much of the bias.
Types of AI bias at a glance
| Type of bias | What it means | Real example | What would help |
|---|---|---|---|
| Training data bias | Data reflects past discrimination or omits groups | Amazon’s hiring tool learned from a decade of mostly male CVs | Audit and rebalance training data |
| Proxy bias | The system uses a stand-in tracking a protected characteristic | The healthcare algorithm used cost as a proxy for illness | Predict the real outcome, not a stand-in |
| Measurement bias | Success is measured so unfairness stays hidden | COMPAS looked accurate overall while its errors fell unevenly | Report error rates separately by group |
| Testing bias | Testing covers too narrow a group | Face recognition tested mostly on lighter-skinned men | Test on balanced datasets |
| Interaction bias | User behaviour reinforces the system’s skew | A chatbot learning from clicks can amplify stereotypes | Monitor live outputs and let users flag problems |
How to spot AI bias: a practical checklist
Run through these questions whenever you use or are judged by an AI system.

- Who was it tested on? Ask for accuracy broken down by gender, age and ethnicity — a headline figure can hide a 30-point gap.
- What is it actually predicting? Check whether it predicts the real outcome or a proxy. Proxies are where bias hides.
- Do its mistakes fall unevenly? Find out whether one group is more often wrongly rejected or overlooked. Uneven errors are the clearest signal of bias.
- What data was it trained on? Ask where the training data came from and whom it represents.
- Can anyone explain a decision? If a system cannot say why it rejected someone, you cannot check it for bias.
- Is there a human in the loop, and can the decision be challenged? A person should review high-stakes decisions, and there should be a clear appeal route.
The same habits apply to tools you choose at work. Before pasting anything sensitive into a chatbot, read how to use AI safely at work without sharing private data.
What can actually be done about it?
Bias cannot be fixed by one tweak, and anyone promising a “bias-free” AI system is selling something. It can, however, be reduced.
For organisations using AI: test on balanced data before deployment, and keep testing after launch. Report error rates by group, not just overall accuracy. And write the rules down: a short, clear AI policy for a small business saying which tools may be used, for which decisions, and who checks the output.
For regulators: the EU AI Act, in force since August 2024, treats some uses — including certain hiring and law-enforcement systems — as high risk, with requirements for testing, documentation and human oversight. Regulation will not remove bias, but it sets a floor: systems that affect people’s lives must be tested before they are trusted.
For everyone else: be sceptical of confident outputs. A score, ranking or shortlist is a suggestion dressed up as a measurement — treat it as a starting point.
Frequently asked questions
What is AI bias in simple terms?
AI bias is when an AI system treats some groups unfairly, usually because it learned from skewed data or was given a badly chosen goal.
Is AI bias deliberate?
Almost never. In the cases above, nobody programmed the system to discriminate — the bias came from historical data, proxies such as healthcare costs, and testing that missed whole groups.
Can AI bias be removed completely?
No — but it can be reduced a great deal, with better training data, honest choice of what to predict, testing broken down by group, and human review. The goal is a system whose errors are known, measured and small enough to manage.
How do I know if an AI tool is biased?
Look for the warning signs above: no accuracy breakdown by group, no explanation for decisions, no way to appeal, and a vendor that will not say what the system was trained on.
What is the difference between AI bias and AI hallucination?
Bias is a systematic skew in how a system treats people — its errors follow a pattern. Hallucination is when a chatbot invents facts that sound plausible but are false. Both are reasons to check AI output first; see what is AI hallucination and how to check AI answers.
Who is responsible when biased AI harms someone?
Responsibility sits with the people who chose, deployed and relied on the system. Employers and vendors are expected to test for bias and provide a way to challenge decisions. Blaming the algorithm is not a defence.
The bottom line
AI bias is what happens when a system learns our history too well: Amazon’s hiring tool, COMPAS, face recognition and a healthcare algorithm all learned the skews in their data. None set out to discriminate — each one did, because the data, the goal or the testing carried a skew. That is also the hopeful part: bias enters through data, goals and testing, so it can be found the same way. Ask who the system was tested on, what it is really predicting, and whether its mistakes fall unevenly. Never let a score be the final word on a decision that changes someone’s life.
Start this week: pick one AI tool you use and ask its provider for accuracy figures broken down by group. One question, asked early, is worth more than a long apology after a biased decision has already been made.


