Contents · 1 / 7
Local Qwen 3.6 Harness Baseline: One Harness Swap Moved the Same Model 19.4 Points
Most harness research happens on hosted frontier models, which leaves anyone running a small local model guessing whether the choice matters for them. This study measured it directly. Qwen 3.6 27B ran the same 144 tasks in Pi and OpenCode on the studio's own hardware, with deterministic grading and zero provider cost. The harness moved the result by 19.4 points: 103/144 in Pi against 75/144 in OpenCode (95% CI 10.4 to 28.5, exact McNemar p = 6.2e-05), with Claude Code descriptive at 102/144. Same model, same tasks, a fifth of the benchmark apart. The takeaway: for local deployments the harness is a first-class decision, as consequential as the model itself, and testing yours costs electricity rather than provider bills. This is a standalone study inside the JackBench programme; its hosted-model counterpart is the Seven-Model Pi–OpenCode Baseline, and the two are never pooled.
The question
The registered research question: under a frozen local inference stack, how do Qwen generation and harness choice affect agent outcomes? This note answers the harness half for one generation. The generation half waits on the Qwen 3.8 study, which stopped at a mechanics failure and is listed as incomplete on the research page.
Method
The model ran the same 144 tasks in each harness, 720 attempts in all. Grading is deterministic: a task passes its checks or it doesn't, and every attempt writes a sealed receipt at completion. The analysis plan was registered before the runs. The confirmatory contrast is Pi against OpenCode; anything outside it, including the Claude Code arm, is labelled descriptive.1
Every attempt writes a sealed receipt. The table below comes from those receipts, not from notes taken along the way.
The result
Qwen 3.6 27B runs on hardware the studio owns, so the provider cost for this study was zero.2 The same model, on the same tasks, produced very different results depending on the harness it sat in.
| Harness | Solved | Rate | Δ vs OpenCode | p |
|---|---|---|---|---|
| Pi | 103/144 | 71.5% | +19.4 | 6.2e-05 |
| OpenCode | 75/144 | 52.1% | ref | ref |
| Claude Code (descriptive) | 102/144 | 70.8% | · | · |
Exact McNemar, paired. 95% CI on the gap: 10.4–28.5. Claude Code sat outside the registered comparison set; its rate is descriptive. Provider cost: 0 USD.
Download the results (CSV)Figure 1 as a table
| Pi | 71.5% |
| Claude Code (descriptive) | 70.8% |
| OpenCode | 52.1% |
The gap is 19.4 points and the interval doesn't get near zero. For a model that costs nothing to run, that's a lot of performance to gain or throw away on harness choice alone.
Corrections and limitations
Known limits: one local model, one task set, and a local-only boundary. Hosted models are a separate study with its own note; comparing across the two requires an explicit bridge that hasn't been built, so don't.
Cite this
Tyler, J. (2026). Local Qwen 3.6 Harness Baseline. Cold Anvil Studios. https://coldanvil.com/research/local-qwen-36-harness-baseline/
@misc{tyler2026localqwen,
author = {Tyler, Jack},
title = {Local Qwen 3.6 Harness Baseline},
year = {2026},
publisher = {Cold Anvil Studios},
howpublished = {\url{https://coldanvil.com/research/local-qwen-36-harness-baseline/}}
}