The Rubric Line That Sent Me Down a Three-Layer Debugging Rabbit Hole
Two days before my hackathon deadline, I re-read the judging criteria forthe track I was submitting to and found a sentence I'd apparently skippedthe first time through:". . . a measurable efficiency gain over. . .
Two days before my hackathon deadline, I re-read the judging criteria forthe track I was submitting to and found a sentence I'd apparently skippedthe first time through:". . . a measurable efficiency gain over single-agent baselines. "My project, RedCouncil, is an adversarial multi-agent system — fivespecialized AI agents (Growth, Risk, Legal, TechDebt, Customer) debate abusiness decision, cross-examine each other, and a Synthesizer produces aseverity-scored report. 68 tests passing, auth, storage, deployment, alldone. I had never once measured whether it actually beat a single modelcall. My first instinct was to bail. My second, better instinct was to ask: howfast can I actually get real evidence for this, instead of assuming it'seither fine or hopeless?Building the eval, fastThe fastest honest version of this isn't a new subsystem — it's reusingeverything you already have. Same scenarios, same model, one new promptthat asks a single Qwen call to do what all five agents do combined, and apurely deterministic scoring function — no LLM judge, just counting and setmath. Two numbers mattered most:Cross-agent conflicts surfaced. A single model call hasno mechanismto represent disagreement between specialist viewpoints. This number is 0 for the baseline by construction, which makes it a free, structurally guaranteed data point. High-severity findings caught, compared head-to-head on the same 1–10 scale, same model, so the comparison isolates the architecture, not model size. I ran it. Numbers came back. They looked good. I started building theslide deck. Then I actually read the results file instead of just the summary, andfound the first problem. Bug 1: the metric that was empty by designOne of the fields I'd built,domains_missed_by_baseline, was empty inevery single scenario — which I initially read as "RedCouncil never missesanything the baseline catches. " It took a second look to realize the fieldwasalwaysgoing to be empty, because of how I'd defined it: it onlychecked domains present in RedCouncil's output but absent from thebaseline's. Since the baseline prompt forced one finding per domain everytime, that set difference was structurally guaranteed to be emptyregardless of how RedCouncil actually performed. Not a finding — anartifact of how I'd defined the metric. Worse, when I checked thereversedirection, the one that actuallymattered, I found RedCouncil was missing an entire domain — Growth — in 3of 5 scenarios. My own multi-agent debate system was covering less groundthan the single dumb baseline it was supposed to beat. Bug 2: the retry that gave up too earlyRoot-causing this with real trace data (not speculation) showed atwo-part failure: the coverage checker only flagged a domain as "missing"if its findings scored above a severity threshold — so a legitimatelylower-severity Growth finding could slip through undetected. And when aretrydidget triggered, the acceptance logic only kept the retry if itstrictly improved on the previous attempt. If the model's retry attemptfailed to recover the missing domain, the system gave up and kept thebroken result — because "no better than before" and "still broken" lookedidentical to the acceptance check. Fix: lower the threshold so every domain gets checked regardless ofseverity, and give the retry loop several bounded attempts instead of one,tracking convergence properly instead of a single strict comparison. I re-ran the comparison. Growth showed up in all 5 scenarios. I updated thedeck. I thought I was done. Bug 3: zero, againThen I actually opened the live product UI to record demo footage, and the"conflicts surfaced" counter read0— on a scenario that, reading thetranscript, clearly had agents contradicting each other's core claims inplain English. Round one of debugging this found a genuine plumbing bug: one part of thecode wrote a result to a field calledconflicts, another part read from afield calledconflict_count. Nobody had ever written to that second key. Classic. Fixed it, redeployed, tested again. Still zero. Round two went deeper, and this is the part I actually learned somethingfrom: the conflict detector was gating candidate disagreements through alexical-overlap check before it would even evaluate whether they opposedeach other. My Growth agent argues in terms ofsavings, cost, ROI. MyCustomer agent argues in terms ofchurn, friction, retention. Zeroshared vocabulary — so the gate filtered the pair out before the actualstance-comparison logic ever ran, even though the underlying disagreementwas completely real. This is the kind of bug that's obvious once you see it and invisible untilyou do:specialists with genuinely different mandates use genuinelydifferent words by design. A similarity gate built for a system whereeveryone talks about the same thing is exactly the wrong tool for a systemwhose entire premise is that the agentsdon'tthink alike. The fix: stop inferring disagreement from vocabulary overlap. Trust thesignal the agents themselves already produce when they explicitly rebuteach other during cross-examination. Bug 4: the one hiding inside the fixTrusting each agent's own rebuttal signal introduced a new failure modealmost immediately: rebuttals are oftenmutual. If Growth rebuts Risk andRisk rebuts Growth in the same exchange, and each rebuttal becomes its ownconflict entry, you double-count every disagreement that goes both ways. Caught this one before it shipped, with a simple fix: dedupe on theunordered agent pair before counting, not the directional edge. What actually changed by the endConflict detection went from a broken0, to a plumbing fix that stillread0, to a real but partial fix, to a detector that could finally seedisagreements that don't share vocabulary — at which point the trueconflict count roughly doubled from what the vocabulary-limited version hadfound. Every one of those jumps looked like "the fix worked" right up untilthe next layer surfaced. What I'd actually tell someone building an eval harnessA metric that's always the same value isn't a metric — check what it's structurally capable of showing before you trust it. Mine was empty-by-definition for an entire category of finding. If your system's whole premise is that components think differently, don't build detection logic that assumes they'll describe things similarly. That's not an edge case, it's the design working as intended running into a detector that assumes the opposite. Fixing your eval can be as much engineering work as fixing your product— arguably more, because a broken product fails loudly and a broken eval fails silently, as a confident, wrong number. Non-determinism compounds across every layer you don't pin down. Between run-to-run model variance and the layers of retry logic in the system itself, "the number" was never really one number — it was a distribution I was sampling from once and calling final. Worth deciding on purpose whether that's good enough for your use case, rather than discovering it by accident three reruns in. RedCouncil runs on Qwen Cloud (Qwen-Max) end to end — all five debateagents plus the Synthesizer. If you're building anything where multiplemodel calls are supposed to disagree with each other on purpose, budgetreal time for your evaluation code, not just your product code. Mineneeded just as much debugging.