Workflow Series (05): Evaluation Framework — Three-Layer Testing and Trace Tracking
Why Workflows Need a Dedicated Evaluation FrameworkTraditional software testing covers code correctness. Workflows add two layers of uncertainty:LLM output is non-deterministic: the same input can produce. . .
Why Workflows Need a Dedicated Evaluation FrameworkTraditional software testing covers code correctness. Workflows add two layers of uncertainty:LLM output is non-deterministic: the same input can produce different results across runsCross-step dependencies: a Phase 3 problem may only surface at Phase 7, making the debugging chain longWithout an evaluation framework, every workflow change requires a full end-to-end run: slow, expensive, incomplete coverage. Three-layer testing decomposes the problem. Three-Layer Evaluation StructureLayer 3: End-to-end tests (Workflow level) Full pipeline from trigger to completion Test cases: eval/cases. yaml Metrics: completion rate, Phase 4 avg rounds, gate trigger rate Layer 2: Integration tests (Phase level) Cross-step data flow is correctly passed Cross-phase routing logic fires correctly Layer 1: Unit tests (Step level) Each subagent's output matches its output contract No real LLM calls — validates JSON schema onlyEnter fullscreen modeExit fullscreen modeTest priority:Layer 1 should be the most numerous and fastest — catches contract violations in seconds. Layer 3 is the slowest and most expensive — run it only when changes affect the main pipeline. Layer 1: Step-Level Unit TestsUnit tests verify that subagent output files match the declared schema. No real LLM calls needed. # tests/unit/test_phase3_output. pyimportjsonfrompathlibimportPathdeftest_analysis_output_schema():"""Phase 3 output must conform to analysis_final. json schema"""output=json. loads(Path("test_fixtures/phase3/analysis_final. json"). read_text())assert"passed"inoutputassertisinstance(output["passed"],bool)assert"confidence"inoutputassert0. 0= 0. 95)-phase_4:done (first candidate passes)-phase_5:done-phase_6:done-phase_7:doneexpected_metrics:e2e_success:truephase4_rounds:1gates_triggered:[]-id:WF-E2E-002name:Low confidence path (Gate A triggered)input:jira_key:AE-MOCK-002bug_description:"Intermittentcrash,noreproduciblesteps"expected_flow:-phase_3:done (confidence 70% = fully automated completions / total triggers Phase 4 average rounds 80% = fraction of workflows where at least 1 candidate passed (below 80%: root cause analysis quality or fix strategy needs work) Gate trigger rate None:# Workflow-level trace (top layer)trace=langfuse. trace(name=f"wf-bug-e2e:{jira_key}",input={"jira_key":jira_key},metadata={"workflow_version":"1. 3. 0"})forphase_idinget_pending_phases():# Phase-level spanspan=trace. span(name=phase_id,input={"context":get_phase_context(phase_id)})result=execute_phase(phase_id)span. end(output={"status":result["status"],"passed":result["passed"]},level="DEFAULT"ifresult["passed"]else"WARNING")ifgate_triggered:trace. event(name="human_gate_A",metadata={"triggered_by":"low_confidence","value":confidence})Enter fullscreen modeExit fullscreen modeWhat Trace AnswersHow long did each phase take? → span start/end timestamps Which phase consumed the most tokens? → span usage field What was the raw error when a subagent failed? → span output. error field Is Phase 3 confidence within a healthy range across runs? → span output. confidence, aggregated across multiple tracesEnter fullscreen modeExit fullscreen modeNo more guessing execution order or digging through files. Design ChecklistUnit tests (Layer 1)[ ] Every subagent output has a schema validation test[ ] Fixtures cover both success and failure paths[ ] No real LLM calls — use saved real outputs as fixturesIntegration tests (Layer 2)[ ] Each phase's output fields align with the next phase's context_inputs[ ] All routing conditions (high/mid/low confidence, timeout, failure) have test coverage[ ] Routing logic is implemented as a pure function, runnable in millisecondsEnd-to-end tests (Layer 3)[ ] eval/cases. yaml covers happy path, low-confidence path, fix-failure path[ ] 4 core metrics have defined thresholds[ ] Baseline delta comparison runs before every release; threshold violations block releaseTrace tracking[ ] Every workflow run has a top-level trace[ ] Every phase has a span recording input, output, and latency[ ] Human gate triggers are recorded as events with reason metadataSummaryThree layers, three speeds: Layer 1 validates contracts with fixtures in seconds, Layer 2 tests data flow and routing in seconds, Layer 3 runs the full pipeline in minutes — the first two catch most problems before Layer 3 runsMetric baselines are release gates: if end-to-end completion rate, Phase 4 rounds, candidate pass rate, or gate trigger rate crosses a threshold, the change needs investigationTrace turns black boxes into queryable records: no more guessing execution order or digging through files — search the Langfuse trace for the run and read the spanCheck outPrimeSkills— a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works. Find more useful knowledge and interesting products on myHomepage