Cold Anvil Studios
Specialist AI harnesses, built on measurement rather than hype.
Vibes and Hype: a match made in hell
Peter Steinberger, who builds with AI as seriously as anyone, has called "vibe coding" a slur: the term makes skilled AI-assisted building sound trivial, and he has a point. My objection is narrower, and it's about the literal practice. Letting AI build products and write code on vibes alone is a mistake. Human engineers don't ship unmeasured, unreviewed work, so why would we allow AI to? It needs to be measured, secure and quality controlled. That's not vibes, that's computer science.
And vibe coding is only one part of the problem. The other part needs a name, so I'm coining one: Hype Harnessing is piecing together a stack from whatever harness, tool or skill was hot on YouTube or Reddit this week, without checking whether it improves the output. When it's bad enough to notice the degradation, it's really bad. When it feels good, it's probably better, but how can you be sure? The middle zone is the worst: if you can't feel the vibe in either direction, have you improved your stack, or silently introduced a regression for the whole system?
That is what Cold Anvil Studios is for.
The bet
The model is only part of the result, and that isn't just my finding. Princeton's SWE-agent work showed the same GPT-4 roughly tripling its success rate on real GitHub issues when the interface around it was redesigned, with the model untouched. MetaGPT, a Chinese-led ICLR oral paper, showed structured workflows producing markedly more coherent working software than loosely chaining the same models together. The studio's own research points the same way: the same local model moved 19.4 points between two harnesses with nothing else changed.
The direction of the effect is the catch. Princeton's "AI Agents That Matter" found that complex agent stacks often fail to beat simple, cheaper baselines once cost is counted properly. The harness moves the result, and without measuring you don't know which way.
So the bet is specialist harnesses: built for one job, on the best-measured combination of model, provider, harness and treatment, giving better bang for the buck than pointing a general-purpose frontier tool at everything.
How it works
The evaluation system. It measures which combinations actually perform, and every number keeps its receipt. The studies land on the research page.
The harness that runs the studio's own agents day to day. It gets built on what the measurements say rather than what the timeline recommends.
Specialist tools for consumer and business use. Four are in the backlog as concepts, and the projects page calls them that.
Research feeds the harness, the harness builds the products, and the products raise the next questions worth measuring.
Using this work
My first month of running JackBench came to 760 USD. It was worth it for me, to know I was building performant systems, but I'm sharing the information in the hope it saves someone else the same cost. If the research is useful to you, there's a donation link. You're welcome to donate anything or nothing, recurring or one-off. You're also welcome to use the research itself: it's licensed CC BY 4.0, so link and credit me when you do. The aggregate results ship as CSV with each study. Task sets and graders stay private, because anything published ends up in training data and that would invalidate future runs.