A model can cost less per request and still cost more for every task it completes. In the example above, two models each get 100 tasks. The current model costs $9 in total and completes 90 tasks. The candidate costs $8 but completes 70. The cost per successful task rises from $0.10 to about $0.114, even before anyone fixes the failures. These are illustrative numbers, with retries included in each model's total cost.
Decide what counts as success
Use the product's final state. For an assistant that saves a draft, reopen the record through the normal read path and check the stored fields. A confident confirmation or a successful API status leaves the task unverified until you read the result. For extraction, compare fields against the source and mark missing evidence as unknown. Decide what failure blocks release before you run either model.
Use the same tasks for both versions
Pin the model version, prompt, tool permissions, retry policy and time budget. Keep the surrounding configuration fixed when the goal is to isolate the model. For each case, record the expected result, both outcomes, latency, retries and total cost. Include common requests and cases where mistakes matter, such as ambiguous inputs, missing fields or a tool timeout. Use authorized, sanitized inputs in a sandbox if a tool can change state.
Read the pairs individually. If the candidate fixes five cases but breaks three that the current model handles, inspect those three before celebrating the average. Compare latency on the same inputs. Calculate total model and tool cost across all attempts divided by the number of successfully completed tasks, and keep the completion count beside the ratio. If no task succeeds, report that the ratio is undefined.
Roll out with a rollback condition
Keep the proven model selectable. Test the candidate offline first, or use shadow traffic with side effects disabled where the data and permissions allow. Start with a limited rollout only when the evaluation passes. Write down which critical failure, completion drop or latency change should restore the previous version, and check that the rollback path works before using it. An outage fallback can keep the app available during a provider failure. Quality monitoring still needs its own checks when the candidate answers normally.
Why this comparison matters
In the November 2026 AI Gateway Production Index, Amelia Charles and Harpreet Arora report that more than half of October's language-model requests on Vercel AI Gateway went to versions less than 60 days old on that gateway. The report measures aggregate usage on one service. It gives teams a reason to make evaluation repeatable, while the results for a particular workload still have to come from its own tests. Vercel's model-switching exercise also recommends testing representative prompts or shadow traffic before replacing the primary model. For a reusable evidence checklist, see the AI-generated code acceptance guide.