HeyHomees Lab Notes · No. 10 · TraceLite AI · Published October 27, 2026
By HeyHomees Newsroom ·
In 30 seconds
- Runs 95 mechanism-focused tasks across 4 harnesses and 9 models, totaling 3,420 Docker runs.
- Macro-averaged harness scores range from 59.8% to 72.0%, a 12.2-point span.
- 31.1% of runs succeed at the task despite failing at least one harness criterion; only 8.4% pass every criterion.
What is it?
TraceTrial-Bench is a benchmark that evaluates AI agent harness reliability by organizing harness behavior into seven components (context, execution, governance, lifecycle, observability, tools, verification) and testing them with 95 expert-level, mechanism-oriented tasks.
How does it work?
The framework runs each task with a model plus a harness, captures execution traces via a Trajectory Tracing module, gathers evidence from the agent, harness, and environment with a Multi-source Evidence Acquisition module, and then attributes failures using a Failure Attribution module that separates agent performance from harness reliability.
How well does it do?
Across 4 harnesses, 9 models, and 95 tasks, the evaluation of 3,420 runs yields macro-averaged harness scores from 59.8% to 72.0%; Claude Code achieves the highest Agent Pass rate (44.0%) and OpenClaw the highest Harness Macro (72.0%). Only 8.4% of runs pass every criterion in both layers, while 31.1% succeed despite at least one harness failure.
What does it need to run?
The benchmark requires Python ≥ 3.11, Docker, and a Linux-style shell (or WSL on Windows). Users must provide a model.toml file with provider endpoint, model identifier, and API key.
By the numbers
- Macro-averaged harness score range: 59.8% to 72.0%
- Runs succeeding despite harness failure: 31.1%
- Runs passing every criterion: 8.4%
- Total runs evaluated: 3,420
The honest read
The reported macro-averaged harness scores (59.8%–72.0%) and pass rates are self-reported by the TraceLite AI authors based on their own benchmark runs. The figures show how often harness criteria are met, but they do not measure absolute task quality or real-world impact beyond the 95 curated tasks. The benchmark also notes that a sizable share of runs (31.1%) succeed even when a harness fails, indicating that the harness score alone does not fully predict task success.
What we don't know yet
- Exact download size of the benchmark repository
- The specific hardware (CPU/GPU) used for the reported runs
- Whether the benchmark includes any proprietary models or only open-source ones
Sources
- Paper (arXiv abstract) (primary)
- Code, GitHub
Researched and drafted with AI from the sources above; every figure is checked against them by code before it is published, and a line that fails is dropped.