---
title: "AI Coding Agent Benchmark: A Production Checklist"
description: "Audit coding-agent benchmarks for private tasks, repeat runs, harness controls, exploits, and cost per solved task."
source: "https://agntc.work/blog/ai-coding-agent-benchmark/"
pubDate: "2026-09-22T00:00:00.000Z"
category: "observability-evals"
author: "Erez Eden"
proficiencyLevel: "Advanced"
tags: ["AI Coding Agent Benchmark","Coding Agents","Agent Evaluation","SWE-bench"]
---
# Executive Summary
An AI coding agent benchmark is credible for a production decision only when its tasks resist memorization, its harness and network access are controlled, results repeat across runs, exploits are audited, and cost is reported per solved task. This runnable checker rejects reports that omit those controls.

# Key Takeaways
- Treat model, harness, tools, and network policy as one evaluated system; a model score alone does not describe the deployed agent.
- Use private or post-cutoff tasks, repeat each task, and audit for benchmark shortcuts before trusting a resolution rate.
- Compare cost per solved task, not cost per run, because cheap failures are not useful production work.

---

An AI coding agent benchmark is credible for a production decision only when its tasks resist memorization, its harness and network access are controlled, results repeat across runs, exploits are audited, and cost is reported per solved task. The runnable checker on this page applies a proposed screening policy to benchmark reports. It is an operational starting point, not a universal standard.


## What the checker tests


The checker accepts a benchmark report only when it declares six controls. These are deliberately strict defaults for an internal shortlist, not empirical laws:


| Control | Minimum accepted evidence |
|---|---|
| Task source | Private or post-cutoff tasks |
| Repetitions | At least three runs as a low-cost variance screen |
| Harness | Name and version |
| Network | Blocked or allowlisted to limit answer retrieval |
| Exploit audit | Explicitly completed |
| Cost | Cost per solved task reported |


I ran two fixtures. The production-style report passed. A public, single-run report with an open network and no exploit audit failed all relevant checks.


The cutoff of three runs is my proposed floor. It is lower than the eight runs per task reported by Real-SWE and the at-least-five runs per configuration in GitHub's evaluation. Three runs can reveal obvious instability; it cannot establish a narrow confidence interval. The network rule is also a policy choice: an open network can let an agent retrieve upstream code or known solutions, which the exploit study identifies as a shortcut. A benchmark can still test open-network agents, but this gate will not admit it as clean evidence of unaided repository problem solving.


```json
{
  "production_suite": { "decision": "accept", "failures": [] },