MODEL Benchmark

Why this benchmark exists

Most benchmarks give every model one score and stop there. That single average hides the two things that matter most: whether a model spends its own compute wisely, and where it is confidently wrong.

ModelBench measures both directly, in one currency — prediction error, how surprised a model is by what actually came next.

Agency — given control over its own compute, does a model allocate it where it pays off? A higher agency score means the model's own choices beat a flat, even allocation of the same budget.

Blind spots — where does a model confidently predict the wrong thing? These aren't random errors. They cluster into patterns, and a leaderboard that only reports an average never shows them.

Scores here are facts. How they're computed is our method — and that part stays ours.

Glossary

• Agency score (nats) — how much better a model's self-chosen compute allocation predicts text than a uniform allocation of the same budget. Higher is better.
• Self PPL vs uniform PPL — perplexity when the model picks its own depth vs when every token gets the same depth. The gap is the value of choice.
• Depth vs difficulty — whether the model spends more compute on tokens it's most wrong about. Negative is expected: it avoids wasting compute on tokens no amount of compute would fix.
• Depth vs marginal benefit — whether the model spends compute where it actually reduces error. Positive means it targets what's fixable.
• Blind spots — positions where the model is both wrong and confident: the silent failures that matter.
• Blind spot % — the fraction of scored positions that are blind spots.
• Concentration — how clustered the blind spots are. High means a few systematic patterns; low means scattered noise.