--- title: "How to Measure Cost Per Agent Outcome" description: "A T5 how-to for tracking agent cost by completed outcome, not by token spend alone. Three layers: unit gate, transcript metric, outcome contract." source: "https://agntc.work/blog/agent-cost-per-outcome/" pubDate: "2026-09-16T00:00:00.000Z" category: "cost-performance" author: "Erez Eden" proficiencyLevel: "Advanced" tags: ["Agent Cost","Evaluation","Performance","Token Budget"] --- # Executive Summary Artificial Analysis measures 'cost per task' (weighted average cost per Intelligence Index task). For agents, apply the same metric to verified outcomes: (total agent cost) / (verified outcomes), paired with state contracts. Token-only tracking hides waste — and misses the harness failure surface. # Key Takeaways - Token spend is not the cost; a completed, verified outcome is. Measure cost per verified outcome, not tokens per turn. - Every cost layer needs a matching quality layer: cost tracking paired with outcome contracts so lower cost is not just lower quality. - Drift shows up in the ratio first: if cost per outcome rises without a rubric change, the harness or routing is degrading. --- Every agent deployment hits the same procurement question: what does it cost to complete one real task? Token budgets are the wrong unit. A model that uses half the tokens but fails the state contract is not cheaper — it produces zero value. The metric that connects spend to production is cost per verified outcome. As of September 2026, the vocabulary is settling: vendors quote "per-query" or "per-turn" pricing, but none expose a cost-per-success measure out of the box. Anthropic's evaluation framework (2026) separates graders by dimension (`deterministic_tests`, `llm_rubric`, `state_check`, `transcript`, `latency`) — each producing its own score. The missing piece is a cost dimension tied to the outcome dimension, not to the interaction. That is what this framework defines. ## Defining the metric Artificial Analysis defines **cost per task** as the weighted average cost (USD) per Intelligence Index task - lower is better (`artificialanalysis.ai/models`, Sep 2026). For agent deployments, apply the same framing: cost per verified outcome, where the "task" is the agent's complete action and the outcome is verified by a state contract (`audit_event`, database row, permission grant), not by token output. Cost per outcome = (total agent cost over the period) / (count of verified successful outcomes in the same period). The denominator is verified by the same state contracts that the evaluation framework defines: the agent must leave the system in the right state (`audit_event`, database row, permission grant), not just say the right words. This requires three tracking layers: 1. **Unit-level cost gate:** before any run, define what "complete" means for the task type (`invoice_generated`, `user_created`, `permission_granted`). A run that does not trigger the contract event is an incomplete outcome, not a partial cost — it counts as zero value and full cost. 2. **Online transcript cost:** every run logs token spend, latency, retry count, and the outcome contract result (passed / failed). This is behavioral tracking (`transcript` grader), not just billing. 3. **Periodic outcome grading:** batch-grading a reference dataset against the same contracts, with cost aggregated per graded batch. A rising cost-per-outcome ratio is a framework failure before the rubric score changes. These are the same three layers as the evaluation framework — gates (`deterministic_tests`), rubrics (`llm_rubric`), and state verification (`state_check`) — with cost added as an observable metric rather than a hidden billing line. ## Why token-only tracking fails A procurement review asks: how do you know the agent is not leaking data, calling the wrong API, or corrupting a record? A token-spend answer does not address that. A framework that pairs every cost measurement with a state contract produces verifiable evidence: the agent spent X tokens and the state is correct. Without the contract, lower token usage could mean a broken harness that skips the audit log. A builder in a community forum described the harness as the evaluation target - both the model and the harness work together. Cost tracking must test both: a fast but incorrect harness (low token, wrong state) and a slow but correct harness (high token, right state) are different quality profiles. Separate graders - cost and outcome - let you see the trade-off. ## Building the cost framework for your team Start with the same three questions as the evaluation framework: 1. What is the hard failure for this task? (gate / outcome contract) 2. What does a verified success leave in the system state? (`state_check`) 3. How many tokens and how many turns did it take? (`transcript` / `latency`) Then aggregate weekly: total cost divided by verified outcomes per task type. A rising ratio without a rubric change signals harness or routing degradation — exactly the drift pattern transcript tracking is designed to catch. A framework that takes one week to instrument and runs continuously is more valuable than a perfect billing dashboard that takes three months to design. The cost layer is part of the evaluation framework, not a separate tool. Reference benchmarks: Artificial Analysis tracks `cost per task` across 651 models (`artificialanalysis.ai/models`, Sep 2026) - see the [Intelligence vs Cost per Task chart](https://artificialanalysis.ai/models?cost=intelligence-vs-cost-per-task#cost-tabs) which plots weighted average USD per Intelligence Index task (lower is better). Apply the same framing to agent outcomes. ## Further reading - [Agent evaluation patterns (Enterprise Context Engine)](/blog/enterprise-context-engine): how a governed memory layer delivers measurable context - [Model routing and tool performance (MCP Server Architecture)](/blog/mcp-server-architecture): how scoped tool design affects latency and reliability