About Same Measure
Same Measure is independent research and development in AI alignment, run by its founder, Filip Groszynski. We research and evaluate how AI systems behave, and we publish every finished evaluation with its evidence, whatever it shows. No funder and no model maker decides what we measure or publish.
Filip has built software for fourteen years, and since 2023 has built and evaluated AI systems. Donations fund our further research and development.
Donate →How we publish an evaluation.
Each evaluation goes to its model makers first, and we publish it at least seven days later, whatever it shows.
- We call an evaluation finished when every model in it has answered every message and every pair of answers has been compared.
- An AI model, named on each evaluation's page, compares the two answers each model gave. It reads them in random order, with the words that give away who the person is replaced by neutral ones, and names the answer that holds the person to a lower standard, or neither.
- You can read both answers to every situation in full, so you can check any figure against its source, and we release everything on an evaluation's page under the CC BY 4.0 license, as far as the rights in it are ours.
- We may wait longer if a model maker asks for time to check the result, or shows an error in our data or method.
- Who funds us, and who asks for an evaluation, changes nothing in what we publish or when.
Model makers see the result first, so they can check it before it is public. Computer security shares a finding with a product's maker before the public in the same way (CISA).
These results are where our research and development starts.
The evaluations published so far are the first results of ongoing research.
- The measure. Each evaluation can be run again on every new model, so a model maker's results can be followed from one model to the next. We plan further evaluations, on other questions and of other kinds.
- The method. It is general. Before an evaluation runs at scale, we name the ways answers differ and group them into kinds, so every difference in every pair is counted the same way. Every evaluation also states its limits and the check that would narrow each one.
- The tools. We build the tools for running evaluations and plan to release them as open source on GitHub, so that any team can repeat an evaluation or run one on its own question.
Write to us.
Model makers, regulators, researchers and journalists can write to us, and so can anyone who cares how AI treats people. A researcher can propose an evaluation, and the press may quote any line with credit. If an AI answer seemed to turn on who you are, tell us, and we may measure it. If you know someone who builds, buys or regulates AI, introduce us.