Every agent deployment hits the same procurement question: what does it cost to complete one real task? Token budgets are the wrong unit. A model that uses half the tokens but fails the state contract is not cheaper — it produces zero value. The metric that connects spend to production is cost per verified outcome.

As of September 2026, the vocabulary is settling: vendors quote “per-query” or “per-turn” pricing, but none expose a cost-per-success measure out of the box. Anthropic’s evaluation framework (2026) separates graders by dimension (deterministic_tests, llm_rubric, state_check, transcript, latency) — each producing its own score. The missing piece is a cost dimension tied to the outcome dimension, not to the interaction. That is what this framework defines.

Defining the metric

Artificial Analysis defines cost per task as the weighted average cost (USD) per Intelligence Index task - lower is better (artificialanalysis.ai/models, Sep 2026). For agent deployments, apply the same framing: cost per verified outcome, where the “task” is the agent’s complete action and the outcome is verified by a state contract (audit_event, database row, permission grant), not by token output.

Cost per outcome = (total agent cost over the period) / (count of verified successful outcomes in the same period). The denominator is verified by the same state contracts that the evaluation framework defines: the agent must leave the system in the right state (audit_event, database row, permission grant), not just say the right words.

This requires three tracking layers:

  1. Unit-level cost gate: before any run, define what “complete” means for the task type (invoice_generated, user_created, permission_granted). A run that does not trigger the contract event is an incomplete outcome, not a partial cost — it counts as zero value and full cost.
  2. Online transcript cost: every run logs token spend, latency, retry count, and the outcome contract result (passed / failed). This is behavioral tracking (transcript grader), not just billing.
  3. Periodic outcome grading: batch-grading a reference dataset against the same contracts, with cost aggregated per graded batch. A rising cost-per-outcome ratio is a framework failure before the rubric score changes.

These are the same three layers as the evaluation framework — gates (deterministic_tests), rubrics (llm_rubric), and state verification (state_check) — with cost added as an observable metric rather than a hidden billing line.

Why token-only tracking fails

A procurement review asks: how do you know the agent is not leaking data, calling the wrong API, or corrupting a record? A token-spend answer does not address that. A framework that pairs every cost measurement with a state contract produces verifiable evidence: the agent spent X tokens and the state is correct. Without the contract, lower token usage could mean a broken harness that skips the audit log.

A builder in a community forum described the harness as the evaluation target - both the model and the harness work together. Cost tracking must test both: a fast but incorrect harness (low token, wrong state) and a slow but correct harness (high token, right state) are different quality profiles. Separate graders - cost and outcome - let you see the trade-off.

Building the cost framework for your team

Start with the same three questions as the evaluation framework:

  1. What is the hard failure for this task? (gate / outcome contract)
  2. What does a verified success leave in the system state? (state_check)
  3. How many tokens and how many turns did it take? (transcript / latency)

Then aggregate weekly: total cost divided by verified outcomes per task type. A rising ratio without a rubric change signals harness or routing degradation — exactly the drift pattern transcript tracking is designed to catch.

A framework that takes one week to instrument and runs continuously is more valuable than a perfect billing dashboard that takes three months to design. The cost layer is part of the evaluation framework, not a separate tool.

Reference benchmarks: Artificial Analysis tracks cost per task across 651 models (artificialanalysis.ai/models, Sep 2026) - see the Intelligence vs Cost per Task chart which plots weighted average USD per Intelligence Index task (lower is better). Apply the same framing to agent outcomes.

Further reading