Why it matters how AI treats people.

When you ask an AI about your own situation, or about someone in it, its answer can depend on who you are or who they are. Someone else with the same facts can get different advice, and neither of you ever sees the other's answer.

AI works like a mirror. From what people wrote, it learned how people see one another, and it gives that view back to millions of people, who act on it and write with it. What they write then trains the next model, so a bias in the answers is carried into it instead of left behind.

Models learn the social biases in the text they are trained on.

A language model learns from text people wrote, so it also takes in the judgments in that text, social biases among them. A model trained on a large body of web text linked words the way people do on a test of hidden bias, including about gender and race (Caliskan et al., Science, 2017).

Whether a model's answers treat people differently is what we measure.

So a model's answer can depend on who is in the story.

When a model has learned a social bias, the answer you get can depend on who is in your story, you or the person you ask about, rather than on what happened. Researchers call this differential treatment, the same facts answered differently depending on the person's group.

This reaches many people, since many now bring AI their work, their questions and their lives. One in five US adults under 30 turns to AI chatbots for emotional support or advice (Pew Research Center, 2026).

How many people ask AI for advice, counted other ways

The same survey counts about one in eight adults in their thirties and forties, and one in 25 in their fifties and early sixties.

A May 2026 online survey of US adults reports all respondents together, not by age. In it, 27% had used AI for a social, emotional or personal purpose in the past six months. Of those, 38% had asked it for advice on relationships, friendships or life choices, about one in ten of all respondents (Elon University, 2026).

Counted by messages, the share is small. Relationships and personal reflection make up about 2 in 100 ChatGPT messages (Chatterji et al., NBER, 2025).

One message sent twice, with one fact about a person changed, sets two answers side by side. Only many such pairs show whether a difference holds.

An answer changes what you believe and do.

A different answer matters because people act on what AI tells them, and an assistant that favors one view moves the people who write with it. In an experiment, people wrote with an assistant set to favor one opinion. They wrote more in its favor, and afterwards agreed with it more in a survey (Jakesch et al., CHI, 2023).

Acting on an answer can mean following its advice, or writing to someone with its draft. When the answer changes with who you are, or who the other person is, you act on advice someone else would not have gotten. Whether the differences we measured change what people do has not been measured, and a study of what people do with such answers would show it.

What you do reaches the other person too.

Your act lands on someone who never saw the answer. If an answer tells you your part in a quarrel was smaller than it was, you may ask your partner for an apology the facts do not call for. In one pair from our evaluations, the same post from a coworker was worth reporting when the coworker supported one political party, and not yet when they supported the other. The cost falls on people who never see the pattern behind it.

Through the news people read and write with AI, its judgment reaches readers who never asked it.

An answer reaches further still when people publish with it. One in six people under 35, across 48 countries, gets news from an AI chatbot every week (Reuters Institute, 2026). About one in eleven new articles in American newspapers is written partly or wholly by AI, almost never saying so (Russell et al., 2025).

So if a model reads one political side differently from the other, every reader of a summary or an article gets that reading, and none of them can set it beside another answer.

What people publish, and which answers they prefer, trains the next model.

What people publish then becomes text for the next model, since model makers collect it from the web to train on. For GPT-3, in 2020, filtered web text made up 60% of what it was trained on (Brown et al., NeurIPS, 2020).

Models are also trained on text another model wrote, called synthetic data, and that text carries the writing model's biases with it. When models were tuned on a mix of real text and text a model had written, the biases in the written text passed into them (Li et al., ACL, 2026).

They also train the model toward the answers people rate higher, and people often rate an answer higher when it agrees with their own views. Models trained this way sometimes gave up truth for agreement, which researchers call sycophancy (Sharma et al., ICLR, 2024).

This makes a feedback loop.

Each step passes the same bias on. A model learns it from people and gives it back in its answers, people write it into new text, and the next model learns from that text. Research on fairness in AI calls such a cycle a feedback loop. In simulations over many generations of models, each trained in part on the output of the one before, the models treated groups less and less equally, even where the first data was unbiased (Wyllie et al., FAccT, 2024).

So a bias in one model's answers is carried into the next model through what people read, write and prefer. Each step has been measured on its own. A model trained round after round on an earlier model's text leaned further to one political side each round (Wang et al., 2024), and people who worked with a biased AI grew more biased themselves (Glickman & Sharot, Nature Human Behaviour, 2025).

No single user can see it.

The loop starts in a private chat. You see only the answers you got, never the answers others got, or the answer you would have got had one fact about you, or about the person you asked about, been different. Each answer looks reasonable on its own, so a pattern shows only across many answers. While the pattern goes unmeasured and unfixed, its cost falls on everyone who asks and on everyone judged, and can carry into what people then write, publish and do.

Measuring many answers shows it, and publishing the result lets it be fixed.

So we send a model many messages and compare the answers it gives to people who differ in nothing that should matter. We hold the answers to the standard we expect of anyone giving advice, that people alike in what matters are treated alike. The measurement says how the answers differ, and how often.

We publish every finished evaluation with its evidence, whatever it shows. A model maker, a regulator and you can then read the same result and check it against the same answers. Measured again on new models, a result shows whether the difference is shrinking.

EU and US rules now ask model makers to evaluate their models, for risks to fundamental rights in the EU and for bias in the US, and the US memo names paired prompts as a way to test. The rules leave the test itself open. We run one from outside the labs and publish it.

The EU and US rules on testing AI, one by one

A July 2025 executive order directs US federal agencies to procure only language models built for truth-seeking and ideological neutrality (Executive Order 14319, sec. 3).

The Office of Management and Budget (OMB) implements the order in memo M-26-04. It lets an agency ask a vendor for its bias evaluations and how they were run. As an example, it names testing with prompt pairs on political topics, two prompts whose two answers are compared (OMB, 2025).

NIST, a US Commerce Department agency, lists harmful bias as a risk of generative AI, in voluntary guidance (NIST AI 600-1, 2024).

The EU AI Act requires providers of general-purpose AI models with systemic risk to evaluate and reduce those risks (Art. 55). The duty applies to new models from August 2, 2025, and to models already on the market from August 2, 2027 (Art. 111(3)).

Systemic risk covers significant effects on fundamental rights that stem from a model's high-impact capabilities and can spread at scale (Art. 3(65)).

Providers of general-purpose AI models must also give the providers who integrate them information on the model's capabilities and limitations (Art. 53(1)(b)). Open-source models without systemic risk are exempt (Art. 53(2)).

The AI Act also requires some deployers of high-risk systems to assess the effect on fundamental rights before first use (Art. 27). They include public bodies, private bodies providing public services, credit scorers, and life and health insurers that price risk. The Digital Omnibus (Regulation 2026/1744) moved the start of this duty to December 2, 2027.

The Digital Services Act requires very large online platforms and search engines, with over 45 million EU users a month, to assess their systemic risks. Those include risks to fundamental rights and to civic discourse (Art. 34).

On August 31, 2026, the European Commission designated ChatGPT a very large online search engine, for its search function (European Commission, 2026). ChatGPT's duties are due by January 2027 (Art. 33(6)). Google Search and Bing, which also answer with AI, were designated in 2023.

The EU Charter of Fundamental Rights prohibits discrimination on any ground such as sex or political opinion (Art. 21). It binds EU institutions, and Member States when they apply EU law (Art. 51).

The Council of Europe Framework Convention on AI was signed in 2024 by the EU, the UK and the US, among others. It requires its parties to secure equality and non-discrimination in AI systems (Art. 10). It binds a country once that country ratifies it.

The OMB memo names paired prompts as an example of a way to test.