How benchmark scores work
The directory shows two numbers Priors calculates itself. Neither is a model’s result: they describe the benchmark. Leaderboard scores stay exactly as published.
Quality
A score from 0 to 100 for how useful a benchmark is for telling current models apart. It’s a weighted sum of five parts, each also 0 to 100:
| Part | Weight | What it measures |
|---|---|---|
| Headroom and spread | 30% | Can it still tell top models apart? 75% is the room left above the top quarter of models, 25% is the gap between best and worst. A board with under 5 points between them caps at 35. Scores with no fixed scale, such as Elo, get a neutral 50. |
| Recency | 20% | Full credit for 90 days after the benchmark's release, then falling to zero at two years. Neutral 50 if we haven't verified the release date. |
| Objectivity | 20% | A judgment by Priors, not a measurement. Some benchmarks have their own hand-set value. The rest default by grading method: 80 for percent scores, 65 for other scores, 45 for Elo, 40 for human or LLM judges. |
| Breadth | 15% | How many models it has measured, with full credit at 40. |
| Frontier coverage | 15% | The share of today's 10 default lineup models it has measured. |
Then points are added or taken away, and the total stays between 0 and 100:
- +15 for the benchmarks Priors prefers: DeepSWE v1.1, Terminal-Bench v4-AA, ARC-AGI-3, SimpleBench, BullshitBench v2, HLE-Diamond, AA-Omniscience v1, FrontierMath Tier 4 (v2), ProgramBench v1 Fully Resolved.
- −25 for a few older versions with a clear replacement.
- −20 if its results were last updated before 2026-09-22, when the current frontier models began to ship.
Quality doesn’t prove a benchmark is valid or free of training contamination, and part of it is our judgment. The same score guides the benchmarks suggested in the ranking flow and the builder.
Recommended
The directory’s default order: quality, lifted by how many published Priors pages give the benchmark weight.
quality + 12 × log2(1 + pages using it)
One page adds 12 points and three add 24. Each doubling adds another 12, so the first few pages count most.
Cost per task
Cost is what a model actually spent on a benchmark’s tasks, never its token price alone. By default it is then priced the way subscribers pay: measured cost times the model’s share of API price on a subscription plan.
Each model uses its median plan until you pick one for its lab in a cost chart’s settings, and the share assumes the plan’s full allowance is used. A new OpenAI model with no plan data yet is estimated from Codex’s published limits. A model with no plan data at all stays at API price. Plan data comes from Real API Pricing.
The Pricing setting in a cost chart switches the whole chart to API prices.
A model a provider serves free today costs nothing on subscription pricing. Turn off Count free tiers in a cost chart’s settings to price it on its paid plans instead, or at API price if it has none.
Speed tiers, under a cost chart’s settings, run the same model faster or cheaper. Cost moves by the tier’s price. Under subscription pricing, a tier a plan covers spends its allowance faster: Fast on OpenAI’s plans, and Ultrafast, which only ChatGPT Pro 500 has. Flex and Claude Opus Fast are outside every plan, so they cost what the API charges, even on a chart priced by subscription.
Speed and time per task
Output speed comes from Artificial Analysis. Since its Performance Benchmarking v3.0.0 it covers a request’s full duration, time to first token included, averages its workloads over the past 5 days, and counts a request that fails on the endpoint’s side as 0. It used to be the median of one prompt size, so values before Nov 4, 2026 are not comparable with later ones.
Artificial Analysis’ time per task is computed, not timed: output tokens per Intelligence Index task divided by output speed. It leaves out the time tools take to run. Other sources’ time per task, such as DeepSWE’s, is the measured length of a run.
A speed tier’s speed is measured against the same model’s standard tier, by Artificial Analysis where it benchmarks the tier and otherwise from OpenRouter’s real traffic. It is measured on one effort and applied to every effort, since a model’s output speed barely moves with effort. The two sources can disagree widely, and each mark says which it uses.
The other columns
- Top model: the best score the benchmark reports, older models included.
- Models: how many the publisher has measured. Priors’ estimates don’t count.
- Mixes: how many published Priors pages give it weight.