Contents · 1 / 8
JackBench
What it is
JackBench is an evidence-driven evaluation system for measuring the complete model, provider and harness combination rather than the model alone. A harness is the scaffolding an agent works inside: the prompts, tools, context handling and control loop wrapped around the model. JackBench compares frontier and near-frontier models across coding-agent harnesses, reasoning configurations, skills and controlled harness modifications.
The working hypothesis behind it: many apparent model failures are really interactions between model behaviour, provider routing, harness design, context handling and evaluation design. JackBench exists to test that claim rather than assume it.
The system under test
The same model can be routed through different providers, run with different reasoning settings, and sat inside different harnesses. Each of those choices changes the results. Treating the whole stack as the unit is what lets JackBench say which part moved a number.
Model
The part everyone already measures: the weights and their training. The same weights can behave very differently depending on everything below.
Provider route
The same open-weight model is served by many providers, at different quantisations, sampling defaults, context limits and reliability. Two routes to "the same model" can return different work, so JackBench records the requested and observed identity of every route.
Reasoning config
Thinking on or off, and how much, trades quality against cost and time. So do sampling settings: temperature and its relatives are set and recorded per attempt, because a model at temperature 0 and the same model sampling freely are different systems. Either way, a model with reasoning enabled is a different system from the same model without it, and they get compared as such.
Harness
The scaffolding the agent works inside: prompts, tools, context handling, the control loop. The published 19.4-point gap in the local baseline is this variable moving on its own, with everything else held still.
Treatment
A deliberate, controlled modification: a skill, a prompt pack, a harness change. Treatments are tested one at a time against an untreated control, because bundled changes make it impossible to say what did what.
From attempt to evidence
What gets measured
Every system under test gets measured along five dimensions. Each holds more than one number: solve rates and blind-judged scores both live under quality, tokens and time both under efficiency.
Did the work pass? Solve rates on identical task sets, compared like for like, with blind-judged scores where pass/fail can't capture the difference.
Does it pass repeatedly? The same system, asked again, should not be a coin flip.
Tokens and time per outcome. A right answer that took three times longer is a different result.
Real provider spend per attempt, from the receipts. Success rate alone hides it.
How it fails: model, provider, harness or evaluator. Naming the failing part is most of the value.
Blind judging
Judges have favourites. A human recogniser knows a model's style; an LLM judge goes further and tends to prefer work that sounds like its own. Left unblinded, a scoreboard quietly measures familiarity instead of quality.
So judgement, where it's needed at all, is pairwise and blind. A three-judge panel drawn from three different model families receives two candidates' work and a frozen rubric, and nothing else. Candidate order comes from a sealed randomisation seed, a slice of comparisons is repeated in reverse order as an audit, and the key joining blind comparisons back to real attempts stays locked away until every verdict is in.
Two aligned votes establish a verdict. Ties, parser failures and unresolved disagreement don't get rounded up; they're recorded as withheld. And deterministic evidence always outranks the panel: if the checks contradict a judge, the checks win.
What the rubric actually asks is criteria-first and about the work, not the writing. The distinctions the panel is qualified on: did the work do exactly what was asked; did it fail in a subtle, semantic way while looking right; did it overbuild beyond the ask; did it stop early; did it claim verification it never ran; did it step outside the boundaries it was given. The panel never grades taste or human preference.
The judges are part of the system under test too. The panel had to pass 336 known-label predictions, across both candidate orders and repeated rubric permutations, before qualifying; the full exam and its results are written up in the qualification note. Evaluator failure is a tracked failure class, and a judge that can't be shown to score a dimension reliably doesn't get to score it at all.
Why it matters
One published example. The same local model, on the same 144 tasks, moved 19.4 points depending on which harness it sat in. Nineteen points is the sort of gap people normally blame on the model.
A leaderboard that only measured the model would have missed the entire effect. And solve rate is only one of the five columns: in the hosted baseline, K3 succeeded at near-identical rates in two harnesses whilst spending 5.62× more in one of them, so the two set-ups only look equivalent until the bill arrives.
Figure 2 as a table
| Solve rate | Pi 59.7% (43/72) | OpenCode 61.1% (44/72) |
| Cost, relative | Pi 1× | OpenCode 5.62× |
| Median duration, relative | Pi 1× | OpenCode 3.22× |
The practical reading, if you run models locally: the model is only half the decision. The scaffolding around it is measurable, and sometimes it moves the result more than the choice of model does.
Evidence discipline
Grading is deterministic wherever a check can be automated, and blind wherever it can't: a judge sees the work, never which model or harness produced it, so nothing rides on a brand name. Historical protocols, receipts and analyses are preserved byte-identical; corrections are recorded alongside them rather than over them. Task sets and graders stay private, because anything published ends up in training data and that would invalidate future runs. What does get published is the aggregate data: every study ships its results as CSV.
Runs are mirrored to Braintrust for observability, metadata only: no raw content leaves the sealed receipts, and the receipts, not the dashboard, hold analytical authority. The mirror is for watching experiments move and reconciling that every scheduled attempt actually happened.
Results
JackBench is one programme containing separate studies, and the research page catalogues them with their evidential status. Three have public notes so far: the First Pi–OpenCode Baseline (published, a pooled null with a large efficiency asymmetry), the Local Qwen 3.6 Harness Baseline (a 19.4-point harness gap at zero provider cost), and the Seven-Model Pi–OpenCode Baseline (one Holm-surviving effect, no universal harness winner). In-progress studies are listed there too, labelled as exactly that.