FAILURE MODES

Your agent finished the task. It still failed.

A new benchmark ran 3,420 agent runs across 95 expert tasks, 4 harnesses, and 9 models. 31.1% of runs finished the task while breaking at least one harness rule. The answer was right. The run was not.

2026-10-29

Your agent finished the task. It still failed.

A new benchmark, TraceTrial-Bench, ran 3,420 agent runs across 95 expert tasks, 4 harnesses, and 9 models. 31.1% of runs finished the task while breaking at least one harness rule. The answer was right. The run was not.

The 38-second reel: your agent finished the task. It still failed.

The number

31.1% of runs achieved task success while failing at least one harness criterion. That is the paper’s figure, from 3,420 runs across 95 expert tasks, 4 harnesses, and 9 models. Harness scores across the benchmark ranged from 59.8% to 72.0%.

The seven harness components are context, execution, governance, lifecycle, observability, tools, and verification. The reel names four of them: context, governance, observability, and verification.

The gap

Most benchmarks grade the answer. Almost none grade the harness. That follows the paper’s own framing: benchmarks focus on model capability and overall performance, with limited attention to harness evaluation.

If your eval only checks the output, a run that broke a governance rule, lost context, or skipped verification still counts as a pass. The output looked right, so nothing flagged it. The run itself was the failure, and nobody was watching the run.

The fix

Score the run, not just the result. An eval that checks the output and the harness is the only one that catches a run that finished the task while breaking a rule. Check context, governance, observability, and verification on every run, not just whether the final answer came back right.

What we did not verify

These are the paper’s numbers, not our test. We did not re-run TraceTrial-Bench, and we did not test any harness ourselves. The benchmark is “Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution” (arXiv 2610.32285, Oct 2026). The 31.1% figure and the 59.8% to 72.0% harness score range come from the paper. The voiceover in the reel is AI-generated. Watch the reel on Threads.

This post started as an Instagram post →

Discussion

Talk it through

Argue with us on Instagram.