About

Cold Anvil Studios is run by one person today and that's me, Jack Tyler.

I've always been an enthusiast of technology. As a young boy I'd sit under my bed designing laptops, PDAs, phones and cameras, before most of the world had access to them. Then it was building computers, running bulletin board systems, and writing automation scripts for online games like RuneScape.

School was never my forte, so I'm largely self-educated and light on qualifications. Things worked out in the end: these days I lead the Consumer and B2B Identity and Access Management product team for a very large music streaming company. I've been working in tech since 2010, and doing product management, whether it was in my job title or not, since 2013. LinkedIn has the details, if you want them.

Like many people, I got into AI around GPT-4. As it made its way further into the workplace, I decided I didn't want to be a leader who tells people to use a thing without understanding it. My naturally obsessive personality took it from there: a large local AI fleet, and far too much money spent on Claude, Codex and hosted-provider API calls.

How Cold Anvil happened

Cold Anvil started as a specific product, and a hobby I took rather seriously from the start. I won't go too deep into it, but a colleague once asked "so it's a Lovable clone?", which probably gives away the general premise. (I still think there's a moat in it, and it's still in the backlog.) That first build was my first taste of engineering a harness from scratch, and I learned a lot. I wouldn't call it vibe-coded: I interrogated Claude's approach and methodology the way a real engineer would, and caught a few code errors and security issues doing it. But I never ran a single eval on the whole thing, so there were definitely vibes involved somewhere.

Those learnings went into The Forge, my personal development studio: multiple agents, orchestrated by Annie, the primary agent who runs the show. The aim is to open-source it when it's ready. And to stop building off vibes, I built my own benchmark, tailored largely against my own use cases. It's called JackBench. The AI called it that, not me.

There are also four business ideas in the backlog, some more progressed than others. They're on the projects page; keep an eye out. I'm hoping the benchmarks, the research and the products come in handy for a few people. Honestly though, this is the most excited I've been about learning in almost a decade, and that's enough for me. I wouldn't mind it becoming my full-time job.

In my spare time I'm a documentary photographer, mostly of urban environments. Photography taught me that constraint makes way for creativity, and I've applied that to how I use AI: getting the most out of small open-weight models through training and harness engineering. There's more on that in A photography lesson, that translates to AI.

Using my work

My first month of running JackBench came to 760 USD. It was worth it for me, to know I was building performant systems, but I'm sharing the information in the hope it saves someone else the same cost. If the research is useful to you, there's a donation link. You're welcome to donate anything or nothing, recurring or one-off. You're also welcome to use the research itself: it's licensed CC BY 4.0, so link and credit me when you do. The aggregate results ship as CSV with each study. Task sets and graders stay private, because anything published ends up in training data and that would invalidate future runs.