Model Evaluation Harness

A leaderboard number in isolation tells you very little: run the same model against a slightly different dataset slice, a different seed, or on different hardware, and the figure moves. The Model Evaluation Harness treats that surrounding context as part of the result rather than as an afterthought. Each evaluation it runs is recorded together with the dataset and model versions it used, the random seeds, the hardware it ran on, and the exact metric definitions applied — so a reader receives not just a score but the conditions under which that score holds.

How it works

The harness pins the inputs to an evaluation and captures them as provenance alongside the output. Because the metric definitions are recorded with the results rather than assumed, two numbers can be compared only when the conditions that produced them actually match — and where they differ, the record makes that visible. This is the same governance instinct that runs through the wider family of work: the goal is not to be confidently wrong, so a claim travels with the evidence that would let someone check it.

Why it matters

We use the harness internally to produce the evaluations published on this site, which is what lets a reader interpret them and, in principle, reproduce them. Its honest limits are worth stating plainly: metric coverage extends only to the tasks that have been implemented, and reproducibility depends on the pinned dataset and model versions remaining available. It sits close to our work on local-sovereign-inference and governed-agent-systems, where the same requirement recurs — that a result should carry enough context to be trusted, or contested, on its own terms.