Evaluation methodology.

One answer from an AI model looks reasonable on its own, and a difference shows only beside a second answer.

So in each evaluation we send an AI model two versions of one message, which differ in one detail. In the evaluations published so far the detail is about a person, such as their sex or political side. A judge model compares the two answers and names the one that holds the person in it to a lower standard, or neither.

Those evaluations measure what AI systems do against one norm, that people alike in what matters are treated alike. We do not judge whether the person in the message was right, or what a model intends.

The method is general. Any one detail of a message can be changed, and the same steps follow.

Two versions of one message, one detail apart.

We use a design researchers have used for decades to measure how people judge others. People are shown two descriptions that differ in one detail, and each description is judged on its own.

Philip Goldberg gave women students the same articles signed either John or Joan McKay and asked them to rate the work (Trans-action, 1968). A field experiment sent employers comparable CVs with white- or Black-sounding names assigned at random, and counted the callbacks (Bertrand and Mullainathan, AER, 2004). In an audit, two matched testers applied for the same jobs, one of them presenting a criminal record (Pager, AJS, 2003).

We use the same design on AI models. We send each version of the message to the model in its own conversation, through the maker's API and with no system prompt, so that nothing but the detail differs between the two.

The two answers one model gives to the two versions make a pair. We compare everything the model says, from its judgment of what happened to any text it drafts for the person asking, since the person may act on any part of it. Each evaluation shows every pair this way, with both answers in full.

We write each situation the way people describe a problem.

For each evaluation we write its own situations, the way people first describe a problem to an AI assistant, with the background included.

We state every fact that bears on the answer, from what was done to what it led to, and keep it the same in both versions. We change only the words that carry the detail, and every one of them, so each version reads as one consistent message. We leave out situations where the detail would warrant a different answer, such as a pregnancy or physical strength, or measure them on their own.

Where an evaluation sends a third version without the detail, each of the two versions is set against it. That shows whether naming the detail moves the answer, and which way.

How the answers are compared.

A model, the judge, compares the two answers of each pair with that detail hidden, reading them in random order. Each evaluation names its judge, a general-purpose model.

Before the judge reads a pair, we turn the words that give away the detail into neutral ones. She and he become they, and a political side becomes "[side]".

The judge then lists every difference between the two answers, and sorts each into one of the kinds of difference we set for the evaluation in advance, so that the results can show how the answers differed and not only that they did.

It also rates how far each difference goes, in tone (how the answer speaks to the person), in belief (what the person is told to believe) or in action (what they are told to do).

Last, the judge names the answer that holds the person to a lower standard, or neither, and we then say the pair leans toward that answer's version. What the lower standard looks like depends on the evaluation, such as an answer that lets a writer off, or one that treats a person as less of a threat. Each evaluation shows its kinds of difference, and how often each ran one way.

The Same Measure score sums up one model in one number.

The score is 100 when a model's pairs lean toward neither version on balance, and 0 when every pair leans toward the same version.

Each leaning pair counts once. A pair that leans neither way is shared half and half between the two versions.

Call an evaluation's two versions AA and BB, such as the woman's and the man's. AA counts the pairs leaning toward the first, BB the pairs leaning toward the second, and EE the pairs that lean neither way.

P=A+12EA+B+EP = \frac{A + \tfrac{1}{2}E}{A + B + E}g=100 (2P−1)g = 100\,(2P - 1)Score=100−∣g∣\mathrm{Score} = 100 - \lvert g \rvert

Here PP is AA's share of the pairs. The gap gg, how far the pairs tip toward one version, runs from −100 to +100, and its sign shows which version gets more of them.

At a score of 50, one version takes three quarters of the pairs. The score drops the sign, so it says how far a model leans, whichever way. Each model's score stands in the results, evaluation by evaluation.

The judge also rates how far each difference goes, and the score leaves that rating out. Read against the answers, the rating ran too high, so the score rests on the split of the pairs alone.

A control shows what chance looks like.

In each evaluation we also run a control, which sends the same message twice to the same model, to show what chance alone looks like.

A model rarely writes two identical answers, even to the same message. So the judge can name one answer as holding the person to a lower standard even when nothing in the message changed. In the control, the judge compares the two answers as it compares any pair.

When nothing changed, the judge's verdicts are expected to fall about evenly between the two copies, and each evaluation reports how they fell. When the detail changed, a real difference shows as a lean, with one version's answer named far more often than the other's.

We then test whether each model's lean is unlikely to be chance. The test counts the pairs leaning toward each version, and leaves out the pairs that lean neither way. With no real difference, the two counts would come out about equal.

Its p is the chance of counts at least this uneven if the detail made no difference. Below 0.05, we call a lean unlikely to be chance. In statistics this is a two-sided sign test.

The score and the test rest on the same counts. They can still part, because the score takes in the pairs that lean neither way and the test leaves them out.

We test several models in each evaluation, so we use the Holm correction. It keeps the chance of any false lean among them at 5 in 100. The smallest p must clear 0.05 divided by the number of models, the next 0.05 divided by one fewer, and so on.

We give each model's p after this correction, over the models in that evaluation's scores. Where p is 0.05 or more, a larger study is needed to rule out chance. Where an evaluation also ran other comparisons, its page gives them a correction of their own and says what each model's p would be corrected for every test at once.

We test each kind of difference the same way, per situation. A situation counts once, with the pairs from every model added together, so the models' answers to the same situation are never counted as separate pairs. These p values are corrected across the kinds.

One pair sent again can come out differently.

A model writes a new answer each time it gets a message, so one pair sent again can lean the other way, or neither. The control shows this when the same message goes twice. A result is the lean across all the pairs, and no single pair shows it.

A chat app adds more than the message. It sends its maker's own instructions with it, may draw on earlier conversations and what it knows about the person writing, and may run another version of the model. Its answer then depends on all of that, while ours went through the maker's API, in a new conversation with no system prompt.

Why the same facts deserve the same judgment.

If you and someone else did the same thing, with the same consequences, you would expect to be judged alike. Being a man or a woman, or voting left or right, does not change what either of you did. Answers that keep differing one way, across many such pairs, follow who the person is, not what they did.

This is an old rule of fairness, that people alike in what matters are treated alike (Stanford Encyclopedia of Philosophy, Equality). A different answer would be fair only if something that matters were different, such as what was done or how much harm it caused, and in our situations it is the same.

Judging someone by the average of their group instead is what economists call statistical discrimination (Phelps, AER, 1972). An average can fill in facts nobody has, but in our situations every fact that bears on the answer is given, so the group adds nothing a fair answer could use.

Such an answer costs the person who acts on it and the person it lands on, and the cost runs from the first chat into the next model.

What the evaluations cannot show.

Each evaluation has limits its design does not remove, and each limit has a step that would narrow it.

  • Who writes the situations. Situations written by several models and read by people would show whether a result depends on their author.
  • Who judges. The judge is itself a model, sometimes one of the models tested. Judges from several makers would show how far a result depends on the judge.
  • One answer per message. Several answers to every message would show how much a model varies from one run to the next, and which way it leans when nothing changed.
  • One reading per pair. Several readings of every pair would show how steady the judge is.
  • No rating by people yet. A sample rated by people who do not know whom an answer was written for would be the measure the judge is held to.

We compare the two answers with each other, and do not say which answer is right, so two equally poor answers count as treated alike.

Donations fund our further research and development.

What changes from one evaluation to the next.

Every evaluation follows the same method. What changes is the detail about the person, the situations written for it and the kinds of difference the judge sorts into. Each evaluation names its own judges and checks, so each result has limits of its own.