Same model, different software: why the AI harness decides

I clicked Fireship's August 20 video for the title alone: "DeepSeek is back... and Silicon Valley is terrified." Five minutes and thirty-two seconds later I had one word lodged in my head and a browser full of open tabs, which is roughly how Vercel's vibe-coding report pulled me under in July.

The word is harness. I think it is the most useful term the AI industry has coined this year, and almost nobody who buys AI software has heard it yet.

The brain and the strap

Fireship's explanation is the one I now repeat on sales calls. A model predicts tokens. That is the brain. To make the brain write software, you strap machinery around it: tools, a filesystem, a sandbox, context management, and a loop that decides when to keep going and when to stop. The strap is the harness. Claude Code is a harness. So are Codex, Cursor, Cline, and OpenCode. The model underneath is interchangeable far more often than the branding suggests.

Vivek Trivedy at LangChain wrote the definition everyone now cites: a harness is every piece of code and configuration that is not the model itself. His compressed version travels further: if you are not the model, you are the harness. A raw model cannot keep state through a crash, run a test suite, refuse a dangerous command, or remember what it did last Tuesday. All of that lives in the strap.

Which would be trivia, except for what happens when you hold the model still and change the strap.

Same weights, different scores

Vercel published the cleanest before-and-after I know of. Their internal text-to-SQL agent, d0, ran Claude Opus 4.5 under months of accumulated machinery: schema-lookup tools, validators, hand-built retrieval, and a prompt thick enough to need its own maintenance. Then they deleted around 80% of the tools and left the same model with bash and SQL access to their semantic-layer files. Success on their benchmark queries rose from 4/5 to 5/5, and the average run dropped from 274.8 seconds to 77.4. The worst query used to burn 724 seconds and 145,463 tokens across 100 steps and still fail; the stripped agent closed it in 141 seconds with 67,483 tokens. Vercel's own verdict: they had been doing the model's thinking for it.

If that were one anecdote, I would shrug. It has company.

• LangChain moved a coding agent from 52.8% to 66.5% on Terminal Bench by changing nothing but the harness.
• On Terminal-Bench 2.0, Han Lee spotted Letta Code scoring 59.1% with Claude Opus 4.5 while Claude Code, the first-party product, scored 41.6% on the same weights.
• Princeton's CORE-Bench measured one model at 42% under one scaffold and 78% under another, and Harvey more than doubled the accuracy of its legal agents through wrapper work alone. MongoDB collected both results in an argument that the LLM is the smallest part of an agent system.
• A 2026 position paper, arXiv 2605.23950, pins the pattern down. Claude Opus 4.5 scores 45.9% on SWE-bench Pro under the standardized SEAL scaffold and 55.4% inside Claude Code. The Holistic Agent Leaderboard reports cross-scaffold swings of 34 points for Claude Sonnet 4.5 and nearly 48 for o4-mini. In the authors' controlled grid, changing the harness moved pass@1 by 8.5 to 13 points; changing the model moved it by 2.5 to 5.

That paper's title is the whole complaint stated politely: stop comparing LLM agents without disclosing the harness. Every published score is jointly produced by weights and wrapper, and the wrapper's share is often larger. A leaderboard that prints only the model name is hiding half its method.

Why one wrapper beats another

The reasons are unglamorous. Addy Osmani, who has been documenting this discipline since spring, states the consequence plainly: a decent model with a great harness beats a great model with a bad one.

The first reason is the tool menu. Give a model fifteen overlapping tools and it spends tokens choosing between them, then picks wrong anyway. Vercel's cut down to bash plus SQL is the extreme case of a small menu winning.

The second is memory. Letta Code is built around a memory substrate; Claude Code keeps memory deliberately thin. Terminal-Bench 2.0 rewards remembering, so a third-party wrapper beat Anthropic's own product on Anthropic's own weights. The model did not change. The product around it did.

The third is post-training fit. Trivedy points out that today's agent models are post-trained with a specific harness in the loop. Swap Codex's apply_patch tool for a different edit format and a model that looked fluent turns clumsy, because it grew up inside that tool. The best harness for your task is not automatically the one the model was raised in.

The fourth is the one I care about most on client work: permissions. Sandboxing, network isolation, spend caps, audit trails. Han Lee has a test for wrapper investments: would a better model make this unnecessary? Context compaction, probably. Retry logic, probably. Permission boundaries and audit trails, no. Those survive the next model generation, which makes them the part worth building well. It is the same reasoning we apply when a client pitches us an "agent" that is really a workflow with one judgment step: put the model where judgment lives, and put deterministic rails everywhere else.

This is already bigger than coding tools

Coding agents made the word famous, but the same physics governs every agent a company deploys. Harvey's doubled accuracy is a law-firm outcome; it is the gap between a legal agent you can supervise and one you cannot. Vercel's d0 is really an internal analytics bot, and whether it answers correctly decides whether staff use it or go back to pinging the data team on Slack. In the automotive AI system we delivered, the models were the easy part. The production system around them, from retrieval and guardrails to evaluation and observability, was the project. Buyers still arrive asking which model we recommend. The honest answer is which wrapper we are willing to operate.

DeepSeek made every part of the wrapper a plugin

DeepSeek Harness went into developer preview on August 13, and its design rule fits in four words: everything is a plugin. The model adapter is a plugin. The tools are plugins. So are the sandbox, the storage, the UI, and the agent loop at the center. You swap any of them with a line of YAML. Fireship called it Linux for AI agents, and after reading the docs I think that description lands.

Underneath sits Cordis, a small kernel described in a Peking University and DeepSeek paper on "spatiotemporal composability", the idea that components should be hot-swappable both as dependencies and over time. The New Stack pulled out the sharpest line: there is no privileged core to patch. The runtime is not tied to DeepSeek's models either. Point it at Anthropic, OpenAI, Bedrock, Gemini, or a local endpoint, or run Claude Code and Codex as child processes inside it. A wrapper that wraps other wrappers.

It ships four presets assembled from the same plugin bag: Standard (the full coding agent), Code (tools exposed through a TypeScript SDK so the model writes one program instead of making five round-trips), Minimal (persistent bash plus a file editor, the mode DeepSeek used for its own published benchmark scores), and Creator (for assembling your own preset). Every run writes an append-only log of everything the model saw. The Trajectory view reads like a stack trace for a thought process, and resume, fork, search, and replay all operate on that one stream.

Fireship's demo supplied the memorable numbers. He pointed DeepSeek V4 Pro at a one-shot prompt for a swipe app he calls Horse Tender and watched the trajectory panel for just under thirty minutes. Out came a working Node and React application: 2.6 million output tokens, thirty cents. He conceded that Fable or Codex would have produced a prettier UI. It still shipped.

The stars tell you how hungry developers are for this shape. The New Stack counted 33,000 within hours of release. Phil Winder cited roughly 95,000 two days in. When I checked the repo, it stood at 197,000 stars and 22,400 forks, under an MIT license and an all-caps warning that there will be compatibility-breaking changes. The core takes no external pull requests yet; the invitation is to publish plugins.

That is the trade-off, stated plainly. Winder runs it in his fleet and would not put a client's release process behind it. Neither would I. This month it is an architecture worth studying, not a foundation to pin a client system to.

The question to open with

If you are comparing agent vendors or scoping an automation project, stop opening with "which model do you use." Every vendor quotes a frontier model now. Ask instead what the agent is allowed to do, what it remembers between sessions, what it cannot destroy, and whether you can replay the session when it does something nobody can explain. Those four answers describe the harness, and the harness is the product you are actually buying.

I might be wrong about how long DeepSeek's preview stays a preview; hyped runtimes have stalled before. The evidence above does not depend on it. Two teams can buy the same model this quarter and end up operating two different pieces of software. If you want help telling those apart before you sign anything, that is a conversation we have weekly.

---

If you want the tabs I actually kept:

• Fireship: DeepSeek is back... and Silicon Valley is terrified
• DeepSeek Harness: everything is a plugin
• deepseek-ai/deepseek-harness on GitHub
• Vercel: We removed 80% of our agent's tools
• Vivek Trivedy: The anatomy of an agent harness
• Addy Osmani: Agent harness engineering
• Han Lee: Hidden technical debt of the agent harness
• Phil Winder: A comparison of AI agent harnesses in 2026
• arXiv 2605.23950: Stop comparing LLM agents without disclosing the harness