Engineering notes

Measurement, post-mortems and the potholes, in public

A company that claims an accuracy figure without its basis is not a number should have somewhere it publishes that basis. This is where we write how we measure, what we got wrong, and why a corrected basis sometimes leaves us looking worse.

Measurement2026-09-09
What goes here, and what does not
An opening note. Only things that can be reproduced — every number with its basis, and every correction stated along with whether it flattered us or not.
2 min read

Run it on the same ruler first. Everything else comes after.

If you have an eval set, we plug into it. If you do not, we use a shared third-party harness where neither the judge nor the answer prompt is ours to choose. Only a result both sides can reproduce is worth comparing.

Start building Book a joint evaluation

Measurement, post-mortems and the potholes, in public

What goes here, and what does not

Run it on the same ruler first. Everything else comes after.