Agent = Model x Harness: Your Eval Layer Is Part of the Agent, Not a Tool Beside It
There's a formula I keep coming back to when people ask why their slick demo agent falls apart in production:Agent = Model × HarnessThe model is the raw reasoning — Claude, GPT, whatever. It's swappable, and. . .
There's a formula I keep coming back to when people ask why their slick demo agent falls apart in production:Agent = Model × HarnessThe model is the raw reasoning — Claude, GPT, whatever. It's swappable, and it's getting better on a curve you don't control. The harness is everything else: the goals, the loops, the tools, the scheduler, the retry logic. Most of the engineering that matters lives in the harness, not the model. But here's the part most teams get wrong. They define the harness asthe plumbing to run the model— goals + loops + tools — and then boltevalsandobservabilityon the side as external QA. Things you pointatthe agent, after the fact, from outside. That's the mistake. Your eval layer and your trace layer are inside the harness. They're not tools beside the agent; they're the half of the agent that makes it a closed loop instead of an open one. Harness = goals + loops + tools + lens + evals. The first three let the agentact. The last two let itknow whether the action was any good— which is the only thing that turns an agent thatrunsinto an agent thatimproves. Let me make that concrete, because I learned it the unglamorous way: by watching a scheduled agent crash and discovering exactly which parts of my harness saved me. Open loop vs closed loopAnopen-loopagent acts and moves on. It writes the file, hits the API, ships the commit — and nobody, including the agent itself, knows if the outcome was correct. You find out when a human notices something's broken. Most "autonomous" agents in the wild are open-loop. They're impressive right up until they silently do the wrong thing for three days. Aclosed-loopagent has a feedback path:act → observe → evaluate → correct → act betterEnter fullscreen modeExit fullscreen modeThe two pieces that close it are thelens(observability — every model call and tool step, resolved inputs, raw outputs, so a run is never a black box) and theevals(judgment — the output scored against a standard: deterministic checks, contract validation, model-as-judge). Lens tells youwhat the agent did; evals tell youwhether it was good. You need both, wired into the harness itself — not run by hand once a week by a human who remembers to check. The incident that taught me thisI run a fleet of scheduled agents — background workers on cron, each doing a focused job on a timer, each in an isolated session the main process can't watch live. One of them crashed mid-run. In an open-loop setup, that's a silent disaster: the worker dies, produces nothing, leaves no trace. You discover it days later when the output that should exist doesn't. (If you've run background agents, you know this exact pain — the ones that "go dark" for a week before anyone notices. )Here's what happened instead, because the lens and eval layers were part of the harness:1. The lens meant the failure left a record. Every worker writes its transcriptstub-first: the moment it starts, before doing any real work, it writes a record withOutcome: IN-PROGRESS. So even a worker that dies one second later has left a footprint. The crash wasn't invisible — there was a transcript sitting there, frozen mid-run, saying "I started and never finished. "2. The evals meant the failure got caught — loudly, and repeatedly. Worker transcripts are validated against a versioned contract. A stub stuck atIN-PROGRESSis fine in normal mode (it might still be running), but under the "finished" check it becomes anerror:# Normal: an unfinished stub is acceptableagent-eval validate transcripts/# Finished mode: a lingering IN-PROGRESS is a hard erroragent-eval validate transcripts/--finishedEnter fullscreen modeExit fullscreen modeSo the dead run didn't just leave a record — it gotscored as invalid, and kept showing up in the gate's invalid list every single run until someone dealt with it. The eval layer refused to let the failure quietly blend into the background. That's the whole point. The lens made the failure observable; the evals made it un-ignorable. Neither is something I went and ran by hand — they're standing parts of the harness that turned a crash from a black hole into a tracked defect. Closing the correction stepDetecting the failure is observe + evaluate. The last piece iscorrect. The naive fix is "add afinallyblock so the worker finalizes its own transcript on crash. " That doesn't work, and the reason is instructive:a crashed isolated session is gone. The process that would run thefinallyis the thing that died. You can't ask a dead worker to clean up after itself. So the correction step has to liveoutsidethe worker, as its own small loop. I wrote a sweeper: a scheduled janitor that scans for transcripts stuck atIN-PROGRESSwhose run isprovably over— older than a threshold set safely past the longest possible run time, so it can never race a worker that's still legitimately going — and finalizes them tofail. The logic is deliberately boring:for each transcript stub still marked IN-PROGRESS: if age > MAX_RUN_TIME + safety_margin: # the run is provably dead rewrite Outcome -> "fail (auto-finalized: abandoned mid-run)"Enter fullscreen modeExit fullscreen modeIdempotent, never deletes anything, runs on a timer. The first time it ran it finalized a backlog of long-dead stubs and dropped the gate's invalid count in finished-mode to just the known, expected exceptions. Now it keeps the loop closed automatically: a worker can still crash, but its corpse gets cleaned up within the hour without me touching anything. That'sact → observe → evaluate → correct, fully wired — and three of those four steps are the lens + eval layers doing their job. Self-correcting ≠ self-improving (and why that matters)Here's where I have to be honest, because this is the part it's tempting to oversell. What I built is aself-correctingharness. The loop closes: the system detects deviations from its own standard and repairs them without a human in the loop. That's real, and it's the floor you want under any fleet of autonomous agents. But self-correcting is notself-improving. Self-correcting means the systemholds the line— it keeps itself at the standard. Self-improving would mean the systemmoves the line upon its own: noticing a check has false positives and tightening its own rubric, noticing a worker keeps regressing and adjusting that worker's instructions. My harness doesn't do that. Istill author every improvement. The loops run themselves; the loops were designed by a human. And honestly — that's the right place to stop, for now. A harness that rewrites its own workers and its own eval criteria unsupervised is a different and much riskier thing, precisely becausethe evals would be grading work the same system authored. The moment the thing being judged and the thing doing the judging are the same closed system with no external anchor, "it passes its own evals" stops meaning very much. Self-correctionwithin a human-set contractis the sound version. Self-modificationof the contract itselfis where you want hard guardrails and a human gate before you flip it on. So the honest formula, fully expanded:Agent = Model × Harness, whereHarness = goals + loops + tools + lens + evals— and it's the lens + evals that make the agentself-correcting. Getting from there toself-improvingis a real step further, and it's one you should take deliberately, with a human holding the gate. The takeawayIf you're building agents and your evals and traces are something you runoccasionally, from outside, you don't have a closed loop — you have an open-loop agent with some QA scripts nearby. The upgrade isn't more tooling. It's a reframe:Lens and evals belong inside the harness, running on every action, not on demand. The lens makes failures observable; the evals make them un-ignorable— that's what converts a crash from an invisible black hole into a tracked defect. Closing the correction step often means a separate loop, because the thing that failed can't always clean up after itself. Self-correcting is the floor; self-improving is a further, deliberate step— keep a human on the gate before the system grades its own homework. Agent = Model × Harness. The model is improving on its own. Whether youragentimproves is a question about your harness — and specifically about whether you've wired the loop closed. The two layers in this post map to two tools I use as a single unit. **agent-eval* is the eval framework — it scores and gates an agent'soutput(contract validation, drift, hallucination, staleness); thevalidate --finishedcheck in the incident above is its work. AgentLensis the trace layer — it captureshow the agent got there(every model and tool step, resolved inputs, raw outputs) so the eval signal is actually debuggable. agent-eval tells you something broke; AgentLens tells you why. They ship together on purpose: two halves of one feedback loop — which is exactly the argument of this post. *