An AI coding agent benchmark is credible for a production decision only when its tasks resist memorization, its harness and network access are controlled, results repeat across runs, exploits are audited, and cost is reported per solved task. The runnable checker on this page applies a proposed screening policy to benchmark reports. It is an operational starting point, not a universal standard.
What the checker tests
The checker accepts a benchmark report only when it declares six controls. These are deliberately strict defaults for an internal shortlist, not empirical laws:
| Control | Minimum accepted evidence |
|---|---|
| Task source | Private or post-cutoff tasks |
| Repetitions | At least three runs as a low-cost variance screen |
| Harness | Name and version |
| Network | Blocked or allowlisted to limit answer retrieval |
| Exploit audit | Explicitly completed |
| Cost | Cost per solved task reported |
I ran two fixtures. The production-style report passed. A public, single-run report with an open network and no exploit audit failed all relevant checks.
The cutoff of three runs is my proposed floor. It is lower than the eight runs per task reported by Real-SWE and the at-least-five runs per configuration in GitHub’s evaluation. Three runs can reveal obvious instability; it cannot establish a narrow confidence interval. The network rule is also a policy choice: an open network can let an agent retrieve upstream code or known solutions, which the exploit study identifies as a shortcut. A benchmark can still test open-network agents, but this gate will not admit it as clean evidence of unaided repository problem solving.
{
"production_suite": { "decision": "accept", "failures": [] },