What an AI Evaluation Run Really Costs

An AI evaluation run has two model bills: generating the application’s answers and judging those answers. In an illustrative test of 200 cases, three variants, two trials and two model judgments per answer, the model subtotal is $6.72 at the published Claude rates used below. Re-scoring the saved answers after a rubric change is $0.72 more; regenerating one changed variant and scoring its new answers is $2.24 more. These figures are arithmetic examples, not a test we ran or a quote for your application. Platform charges, retries, storage and human review would add to them.

The first decision is therefore what changed. A new scoring rule can usually be applied to recorded answers. A new application prompt, model, tool or retrieval corpus may require fresh answers to test the changed application. Alibaba Cloud’s Model Evaluation guide explicitly separates inference charges from judge charges and describes reusing saved inference result sets. The worksheet below makes that reuse decision visible before calculating a budget.

Count answers and judgments separately

For a simple offline, pointwise evaluation, define N as the number of cases, V as the number of application variants and R as independently generated trials per case and variant. New application outputs equal G = N × V × R. If each output receives J separate model judge calls, judgments equal Q = G × J. These equations describe this design, not every evaluation system. A pairwise comparison may send two answers to one judge call; several scores may also come from one call. Inspect the evaluator’s actual requests before turning a metric count into a call count.

Record input and billed output tokens separately for both kinds of call. The model subtotal is generation input tokens × generation input rate + generation output tokens × generation output rate + judge input tokens × judge input rate + judge output tokens × judge output rate, with rates converted to the same token unit. The Anthropic pricing page lists standard per-million-token input and output rates separately. It also lists other price categories, including prompt caching. Use the category your requests actually incur.

There are other meters. A hosted evaluation platform may charge for scores, traces, storage or seats; the Pydantic comparison shows that these units differ among tools. Human labeling requires reviewer time. Retrieval, tools, embeddings, storage and failed or retried calls may create additional charges. If you do not know those inputs, leave them as not measured rather than entering zero. Our guide to monitoring an AI agent covers the separate production telemetry budget; a trace fee should not be hidden in a token subtotal.

Choose a scoring method for each question. Alibaba documents model numerical and classification scoring, string matching, text similarity and manual classification. Its string and similarity methods do not call a judge model, though application inference still costs money; manual review has its own labor cost. A deterministic check can establish whether JSON parses or a required field exists. It cannot, by itself, decide whether an open-ended answer is faithful or useful. A model judge can handle a narrower semantic question, but its answers need checking against human labels on representative cases. The Pydantic evaluation guide makes that distinction between code checks, model judges and human labels. No single agreement percentage is a universal pass mark.

Decide what changed before rerunning

Saved outputs let you avoid paying for the same generation twice, but only when they still answer the question you are testing. Keep the original outputs immutable and record the versions behind them. This is a proposed decision procedure, not a feature claim about every evaluation platform.

Change since the recorded run	New application outputs?	What to score	What the result can show
Scoring rubric or judge changes; application and cases stay fixed	No	Re-score all outputs that must be compared under the new rule	A scoring difference on the old answers, not a new application result
Application prompt, model, tool behavior or retrieval snapshot changes	Yes, for the affected variant and cases	Score its new outputs under the same judge used for the comparison	How the changed application answers those cases
New cases are added; old cases and application stay fixed	Yes, for the added cases	Score new outputs; retain comparable old records	Expanded dataset coverage, with the original subset still visible
More independent trials are requested	Yes, for those trials	Score each newly generated output	Variation across fresh generations, if trial settings are recorded

If the judge changes while you compare a new application variant with old ones, re-score the old and new outputs under that same judge. Otherwise the difference could come from the scoring rule. If the retrieval index changed, a saved answer describes the old index; re-scoring it does not test the new index. The same applies to changed tools or model behavior. If a provider does not offer a frozen model version, record the identifier and run time it does expose, and describe the remaining comparability limit.

Alibaba’s guide supports choosing existing inference result sets as evaluation data. It does not imply that re-scoring those results validates a later version of your application. The Splunk workshop discusses sampling and specialized evaluation models as cost alternatives; neither removes the need to decide which application outputs are current.

Price three actions on one example dataset

Here is a fully illustrative workload, chosen to expose the arithmetic. Assume 200 cases, three variants and two independent trials: 200 × 3 × 2 = 1,200 answers. Assume two separate judge calls for every answer: 1,200 × 2 = 2,400 judgments. These are invented workload inputs, not measured usage or a recommended test size.

For each application answer, assume 1,000 billed input tokens and 300 billed output tokens from Claude Sonnet 5.5. Anthropic’s published standard rates are $2 per million input tokens and $10 per million output tokens. The example generation cost is (1,000 × $2 + 300 × $10) / 1,000,000 = $0.005 per answer.

For each judgment, assume 2,000 billed input tokens and 200 billed output tokens from Claude Haiku 5.5, with a prompt under 100,000 tokens. The same official page lists $0.10 per million input tokens in that prompt range and $0.50 per million output tokens. The example judge cost is (2,000 × $0.10 + 200 × $0.50) / 1,000,000 = $0.0003 per judgment. Choosing a cheaper model is a price assumption, not evidence that it judges this task accurately.

Action on the same 200 cases	New answers	New judge calls	Generation cost	Judge cost	Additional model subtotal
Full run of three variants	1,200	2,400	$6.00	$0.72	$6.72
Changed rubric; re-score saved answers	0	2,400	$0 new	$0.72	$0.72
One variant changed; regenerate and score it	400	800	$2.00	$0.24	$2.24

The second and third rows require the prior answers to exist. Their earlier generation charge is already incurred, not erased. The third row holds the dataset, judge and two unaffected variants constant. If the judge changes too, add the cost of scoring every output needed for a fair comparison. If only some cases changed, count those affected cases instead of charging the whole variant. This is why a version-aware ledger matters more than one generic cost-per-run number.

This table excludes prompt-cache pricing, batch discounts, long-context tiers, reasoning or tool charges, retries, platform fees, storage and human work. It assumes every request succeeds and that the stated token counts are billed exactly as entered. Actual token counts depend on your prompts and outputs. Put your measured token counts and current rates into the AI API cost calculator, then add the platform and people rows yourself. The calculator estimates model tokens; it does not decide whether an old answer remains a valid test.

Keep a ledger that makes reuse defensible

A useful run record joins each case to its output and score. Keep a case ID and dataset version; application prompt and configuration version; model identifier and, where available, exact version; retrieval and tool snapshot; trial ID; generated answer or a secure reference to it and its hash; judge model and rubric version; timestamps; and billed input and output tokens for both calls. Add any platform score or event count, storage period, retry count and human minutes that your bill uses. Protect stored inputs and answers according to your data rules.

The hash and versions make a later question answerable: is this the same output scored under a new rule, or a new output from a changed application? A changed verdict on an identical answer is a scoring change. An improvement claim needs comparable cases, a consistent evaluator and new outputs from the changed application. The ledger is our proposed workflow; your platform may expose some fields and require you to store others.

Run frequency belongs in the same sheet. A single $6.72 model subtotal is not a monthly budget. If you run that identical full example once a day for 30 days, the illustrative token subtotal is 30 × $6.72 = $201.60, assuming the same tokens and rates every time. If most daily changes affect only one variant, count those actual changes instead. Pull requests, scheduled jobs and production sampling can each trigger different numbers of runs. The Splunk workshop illustrates why evaluation frequency and sampled volume matter; its vendor savings claims are not inputs to this example.

Cut cost without losing the question

Start by defining the failure you want to catch. Use a deterministic check where an exact answer exists. Reserve a model judge for questions that need one, and inspect whether multiple metrics can be returned in one call before assuming one call per metric. Keep human review for calibration and ambiguous or consequential cases. Alibaba recommends a small initial validation of evaluation dimensions and representative data before scaling the run; its cost guidance also describes rule-based scoring and saved inference results.

Compare a cheaper judge with human-labeled examples from the task’s relevant slices, including difficult cases. Decide in advance which mistakes are tolerable for that use, then inspect disagreements. Reducing judge cost is useful only if the resulting score still answers the intended question. A local model moves expense to hardware and operations; it does not make evaluation free. Our fine-tuning cost guide is a separate budget when training and serving enter the decision.

Reuse complete saved answers for re-scoring when the application has not changed. Provider prompt caching is a different mechanism: it can reduce charges for eligible repeated prompt prefixes under a provider’s rules, but it does not preserve an application answer for later evaluation. Likewise, replaying one saved answer is not a fresh stochastic trial. If batching suits your deadline, check the maker’s current eligibility and price before adding a discount. The API pricing comparison explains why token categories and conditions matter. The AI model tracker can help you note the model release behind a run; it cannot substitute for a provider’s exact version record.

The budget decision is simple once the records exist: list the cases and variants affected by a change, count new generations and judge calls, multiply their measured token categories by current official rates, then add platform, storage and human costs. Keep the old answer set when it is evidence for the old system. Generate again when the question is about the new one.

Sources

• Anthropic Claude pricing — standard Sonnet 5.5 and Haiku 5.5 token rates and pricing categories.
• Alibaba Cloud Model Evaluation guide — scoring methods, inference and judge fee separation, and saved inference result sets.
• Pydantic’s evaluation tool comparison — offline evaluation workflow, scoring methods and different platform meters.
• Splunk evaluation-cost workshop — discussion of evaluation scale, sampling and alternative judge models.

All pages read on November 11, 2026.

Free toolWhat will 10,000 chats a month cost?Estimate my bill