ChaseInTech

ChaseInTech Digest

The Hidden Cost of AI Agents: Verification, Human Review and Rework

Why cost per task is incomplete without tests, model critics, visual QA, repair loops, human approval, monitoring and Cost per Verified Outcome.

A ChaseInTech verification ledger showing an AI task moving from generation through tests, model review, visual QA, repair, human approval and monitoring to a verified outcome.

Also posted on X and LinkedIn

Why cost per task is incomplete until you price tests, model critics, visual QA, repair loops, approval and monitoring.

A reply to my recent post about AI-agent economics asked a better question than the original metric: are we measuring the cost of verifying the agent’s work, or only the cost of producing it?

That question changes the accounting.

My previous argument was that cost per token is a billing metric while cost per task is an operating metric. Token rates tell us what model consumption costs. A task metric gets closer to what a team actually wants: a bug fixed, a report grounded, a page deployed or a customer case resolved.

But cost per task still has a weak point. The denominator can quietly become whatever the agent claims it completed.

An agent can say the page is finished while the mobile layout clips. It can say the deployment succeeded while the wrong artifact is live. It can say the booking is complete while the database is unchanged. It can produce confident research while one material claim has no source.

Generation is visible because it produces an artifact and a model bill. Verification is distributed across tests, browser checks, critic calls, screenshots, repair loops, evidence packets, human judgement and monitoring after the action.

Those are not optional accounting details. They are part of the cost of reaching an outcome a business is willing to accept.

The next useful metric is not simply cost per task. It is the cost of reaching a verified outcome, where verified means the result passed a declared, versioned assurance process. It does not mean universal truth, zero residual risk or external certification.

Cost per task was still progress

Token pricing remains useful. Input, output, reasoning and cache-hit rates explain how a provider meters inference. Teams need those rates for budgeting, routing and diagnosis.

They just do not describe the whole job.

An agent task can include planning, retrieval, tools, browser automation, code execution, repeated attempts and fallback models. The cheapest rate card can still create expensive work if the system needs three runs and twenty minutes of senior review.

That is why the previous ChaseInTech article moved from token rate to cost per verified task completion. It treated retries, tools, infrastructure and human correction as part of the workflow rather than pretending the model call was the product.

Verification overhead is the next layer inside that metric.

The useful distinction is:

Cost per attempt
= what it costs to try

Cost per accepted task
= total workflow cost / outcomes accepted under the task gate

Cost per verified outcome
= total execution and assurance cost / outcomes that passed the declared verification contract

The words accepted and verified matter because the task contract has to say what success means. For a coding task, that may require a clean build, relevant tests and no regression. For research, every material claim may need a valid source. For a public website, the built artifact may need visual QA, approval, live deployment proof and a post-publish check.

The completion claim is only one piece of evidence. The environment outcome is what actually happened.

The hidden ledger behind a GBP 0.58 task

Illustrative ChaseInTech cost ledger showing a landing-page task growing from GBP 0.58 to GBP 4.40 after verification, repair, human approval and monitoring.

Open the figure to inspect the full-size ledger.

Consider a deliberately simple example: an agent is asked to generate and prepare a customer-facing landing page for publication.

If we count only the first visible work, the ledger might look like this.

Illustrative example - not measured ChaseOS telemetry.

Item Illustrative cost
Initial planning and page generation GBP 0.42
Browser automation GBP 0.10
Basic automated tests GBP 0.06
Headline task cost GBP 0.58

The page appears to cost 58p. The build passes. The agent says it is done.

Then someone opens the real page.

The desktop version looks acceptable, but the mobile heading wraps badly. A call-to-action sits below the fold. The visual hierarchy feels like developer output rather than a finished customer page. One claim does not match the approved brief. The evidence packet contains a build log but no screenshot of the deployed state.

The fuller ledger looks different.

Illustrative example - not measured ChaseOS telemetry.

Item Illustrative cost Purpose
Initial planning and page generation GBP 0.42 Produces the first artifact
Browser automation GBP 0.10 Opens and exercises the page
Deterministic tests GBP 0.06 Checks build, routes, schema and obvious regressions
Separate model critic GBP 0.18 Reviews requirements, omissions and claims
Visual QA across viewports GBP 0.16 Checks hierarchy, clipping, readability and polish
Policy and brand validation GBP 0.05 Confirms claims and brand constraints
Failed review and repair rerun GBP 0.28 Corrects the visual and requirement failures
Evidence packet generation GBP 0.07 Packages screenshots, tests, diff and unresolved risks
Human approval: 5 minutes at GBP 36/hour GBP 3.00 Authorised operator reviews and accepts publication
Post-publish smoke verification GBP 0.08 Confirms the live state and critical links
Verified delivery cost GBP 4.40 Reaches the declared assurance state

The arithmetic is intentionally simple. The point is not that every landing page costs GBP 4.40 or needs this exact review graph. Every figure is illustrative.

The point is that the page did not cost 42p because that was the generation bill. It did not necessarily cost only 58p because a browser opened and the build passed. The relevant operating number is the all-in cost of reaching the state the business is prepared to accept.

Human review dominates this example. That will be realistic for some low-volume, high-context workflows and unrealistic for others. Mature automation should reduce the burden by producing better evidence, sampling low-risk work and escalating only ambiguous cases.

It should not hide the labour by leaving it outside the ledger.

Verification overhead is assurance work

I use verification overhead to mean the additional compute, tooling, infrastructure, model calls, latency and human labour required to decide whether an agent outcome should be accepted.

The word overhead can sound like waste. Much of this work is not waste. It is the price of assurance.

There are at least four different surfaces:

  1. Online verification - checks run for a specific task before its outcome is accepted or executed.
  2. Verification infrastructure - the fixed or semi-fixed cost of building tests, validators, browser environments, rubrics, evidence schemas and reviewer tools.
  3. Human review - active judgement, approval and correction time.
  4. Post-action monitoring - smoke checks, drift detection, incident review and rollback verification after the action.

A team can make online checks look cheap by ignoring what it cost to build and maintain the evaluation system. It can make a task look fast by excluding the queue for human approval. It can make a workflow look reliable by counting the pre-deployment pass while ignoring incidents found after release.

Good accounting keeps these buckets visible.

It also separates money from time. A workflow can be cheap in pounds but expensive in delay. Track active compute, queue time, human waiting time, time to first result, time to verified completion, time to approval and time to recovery as distinct measurements.

The verifier stack

Six-layer verification stack showing deterministic, environment, model, visual, human and monitoring checks routed according to risk.

Open the figure to inspect every assurance layer.

A serious system does not ask one grader to prove everything.

Anthropic’s evaluation guidance separates the agent’s trajectory from the final environment outcome and describes agent evaluations that combine code-based, model-based and human graders. Each layer has a different job and a different failure mode.

Verification layer Best use Strength Failure mode
Deterministic validation Tests, schemas, types, limits, permissions, hashes and database state Fast, reproducible and auditable Only proves assertions that were encoded correctly
Environment validation Confirms the required end state exists Connects grading to the real outcome Instrumentation or environment setup can be incomplete
Model-based critic Requirements, omissions, contradictions and semantic quality Flexible across ambiguous work Non-deterministic, biased and capable of inventing defects
Visual QA Rendered layouts, screenshots, hierarchy and navigation Finds human-facing failures that code can miss Visual grounding errors and subjective standards
Policy and security checks Permissions, data exposure and restricted actions Protects authority boundaries Rules can be incomplete or bypassed
Human review Context, accountability, taste and exceptional risk Highest contextual authority Expensive, slow, inconsistent and vulnerable to fatigue
Post-action monitoring Live regressions, incidents and rollback triggers Finds failures earlier checks miss Ongoing cost and detection blind spots

Deterministic checks should carry as much of the load as possible.

Use code to prove that a file exists, a schema validates, the tests pass, a database row reached the expected state, a forbidden permission was not used or a deployment hash matches the approved artifact. Do not pay a language model to guess what a deterministic validator can prove.

Model critics are useful when the question is semantic. Did the article answer the brief? Is a requirement missing? Are two claims contradictory? Is the evidence strong enough for the wording? A separate critic can surface defects the executor did not notice.

But a critic should not self-certify a sensitive action. It should not replace permission checks or become the only source of truth.

Visual QA matters because source code can be correct while the rendered result is poor. VisualWebArena was built around the fact that many web tasks require visual information because interfaces are designed for human perception. OSWorld similarly uses real computer environments and execution-based evaluation for multi-application tasks.

For a website or control plane, the verification evidence may need mobile and desktop screenshots, overflow checks, focus states, navigation, empty states and the actual application surface where the user works.

A code diff is not a user experience.

Human review sits at a different boundary. The goal is not to put a human somewhere in the loop as a ceremonial safety label. The goal is to present an authorised person with the exact contract, what changed, passed and failed checks, screenshots, unresolved ambiguity, cost, retry count, proposed action and rollback path.

That is evidence-amplified approval. It makes scarce human attention more useful.

Why a second agent helps

The executor and verifier optimise for different goals.

The executor is trying to complete the task. The verifier is trying to detect where the result fails the contract. Separate prompts, tools, evidence and incentives can expose different errors.

Anthropic describes evaluator-optimiser workflows in which one model generates and another evaluates against clear criteria, with feedback used for refinement. OpenAI’s CriticGPT research reported that model criticism helped human reviewers catch more bugs in its studied setting.

That is a meaningful pattern for agent systems.

An executor can produce an artifact, trace and evidence. A separate verifier role can run deterministic checks, inspect the environment, apply a semantic rubric, request a bounded repair or escalate uncertainty. The repair can return to the original executor or a designated repair runtime.

The cost of that loop belongs in the ledger.

It also needs a budget. Without stop conditions, evaluator-optimiser loops can become cost multipliers that never converge.

A verification contract should specify:

  • maximum critic passes;
  • maximum repair attempts;
  • cost and time ceilings;
  • evidence required before another retry;
  • when to escalate to a human;
  • when to stop safely.

A verifier should be allowed to return pass, fail, repair or escalate. For high-impact work, it should not grant itself authority to perform the restricted action.

Why another model call is not independent assurance

A second agent is useful, but the word independent needs discipline.

Two calls may share the same model family, training data, prompt assumptions, missing evidence, tool blind spots and completion bias. A confident executor and a confident critic can be wrong for the same reason.

Stronger separation can include a different rubric, blinded review without the executor’s conclusion, deterministic graders before model judgement, access to ground-truth environment state, repeated or order-swapped judge runs, a different model where justified and recurring calibration against human decisions.

Even then, the verifier is not a truth machine.

Research on LLM judges documents position, verbosity and self-enhancement biases. A judge can change its preference when candidate order changes. It can reward polished language over factual substance. It can favour work that resembles its own style.

Critics can hallucinate defects too. OpenAI’s CriticGPT report explicitly notes that critics can identify non-existent problems as well as real ones.

This creates two costly failure classes:

  • False acceptance - a bad outcome is accepted.
  • False rejection - a good outcome is rejected, triggering unnecessary repair, delay or human review.

Both belong in the scorecard. A verifier that catches every possible issue by rejecting almost everything is not necessarily useful. A cheap verifier that approves confident prose is not assurance.

The verifier also needs verification

The moment a verifier can reject work, it becomes part of the production system and deserves its own evaluation.

That means versioning the verifier model, prompt, rubric, evidence inputs and thresholds. It means measuring disagreement, order sensitivity, abstention, false acceptance, false rejection and drift against recurring human-rated samples.

It also means inspecting the benchmark itself.

OpenAI’s work on SWE-bench Verified is a useful example. The original evaluation contained issues that could make results unreliable, so a human-validated subset was created. The lesson is not that every benchmark is broken. The lesson is that tests and grading harnesses can contain ambiguity or reject valid solutions.

Outcome checks are usually stronger than self-reported completion:

  • inspect database state rather than trusting “booking complete”;
  • inspect test results rather than trusting “bug fixed”;
  • inspect a deployed artifact hash rather than trusting “deployment succeeded”;
  • inspect the rendered page rather than trusting source-code confidence.

Repeated reliability matters as well.

pass@k asks whether at least one of several attempts succeeds. It measures what a system can achieve with a sampling or retry budget.

pass^k asks whether all of k repeated trials succeed. It measures consistency.

A workflow that passes once and fails repeatedly may look capable while remaining unreliable for daily operation. Tau-bench uses final environment state and introduced pass^k to expose that distinction in its tool-agent-user setting. Its original task repository now warns that those task versions are outdated, so this article uses the repeated-reliability method rather than any current leaderboard claim.

Visual QA and operator-defined quality

Some of the most expensive quality failures are obvious to a human and invisible to a unit test.

The route resolves. The button exists. The JSON is valid. Yet the interface is cramped, confusing or visually unfinished.

Inside the systems I am building, that appears in practical forms:

  • a mobile article title wraps into an unreadable block;
  • a technical diagram is correct at full size but useless in a social feed;
  • a control-plane channel layout makes sense to the developer but not the operator;
  • an evidence packet contains all required fields but makes the decision harder than the raw artifact;
  • a branded visual technically follows the palette but looks like a generic template.

These judgements are not perfectly deterministic, but they are not beyond engineering.

OpenAI’s report on harness engineering describes making applications legible to agents through browser tooling, DOM snapshots, screenshots, logs and metrics. It also describes converting human taste into more mechanical principles and recurring checks in that internal environment.

An operator quality contract can do something similar. It can store accepted and rejected examples, define visual hierarchy and readability criteria, require a viewport matrix, preserve known defects and specify when uncertainty must escalate.

The aim is not to pretend taste has become objective. It is to make expectations inspectable, repeatable and easier to calibrate.

Assurance should scale with consequence

The wrong response to verification overhead is maximum verification for every task.

A private brainstorm does not need the same assurance graph as a DNS change, credential use, payment, production deployment or public legal claim.

The other wrong response is to trust every output because the model is strong.

A better system routes assurance according to consequence, reversibility, uncertainty and authority.

Proposed tier Example Default assurance
R0 - private draft Brainstorm or throwaway mock Light format checks, optional critique
R1 - reversible internal action Create a branch or organise non-canonical notes Deterministic checks and artifact review
R2 - public but reversible Publish an approved article or update a non-critical page Deterministic, model and visual QA, evidence packet, explicit approval and live smoke check
R3 - sensitive operational action Production deploy, DNS, credentials, payment or customer commitment Full tests, policy and security checks, separate verifier, rollback proof and mandatory approval
R4 - regulated or materially irreversible High-impact financial, legal, medical or security action Domain-specific evaluation, qualified accountable authority and continuous monitoring

This ladder is a proposed architecture, not a claim that every tier is fully implemented in ChaseOS.

NIST’s Generative AI Profile treats testing, evaluation, validation and verification as lifecycle work and recommends oversight proportionate to risk. OWASP’s agentic security guidance highlights risks around tools, identity, inter-agent communication and cascading failures. These are guidance sources, not claims of ChaseOS compliance or certification.

Higher assurance cost can be correct engineering when the downside is material. The optimisation target is not the lowest raw bill. It is the lowest defensible cost for the declared assurance level and remaining risk.

Cost per Verified Outcome

I am proposing Cost per Verified Outcome, or CPVO, as the main accounting view for this problem.

This is a ChaseInTech operating framework, not a universal accounting standard, and I am not claiming to have invented the phrase.

The direct cost of one outcome is:

C_outcome =
  C_execution
  + C_verification
  + C_rework
  + C_human
  + C_monitoring

Across a workload:

CPVO =
  [sum of execution, verification, rework, human and monitoring cost
   + amortised verification-infrastructure cost]
  / outcomes that passed the declared verification contract

The denominator is the critical part.

Do not mix attempts, completion claims, accepted outcomes and verified outcomes. Include failed attempts that contributed to delivery. Track safe refusals and escalations as their own correct outcome classes where the contract requires them.

Disclose how offline evaluation infrastructure is amortised. A test suite does not become free because it was built before the live run.

Supporting metrics should remain visible:

Metric What it reveals
First-pass acceptance rate Whether low raw cost is being subsidised by repair loops
Retry amplification factor Total trials required per accepted outcome
Human escalation rate Dependence on scarce operator attention
Human intervention minutes Hidden labour per accepted outcome
Time to verified completion Delay from start to accepted state
False acceptance rate Bad outcomes incorrectly approved
False rejection rate Good outcomes unnecessarily blocked or rerun
Verifier disagreement rate Ambiguity or poor calibration
Evidence completeness Auditability of the decision
pass^k consistency Repeated reliability rather than one lucky pass
Residual incident rate Material issues found after acceptance
Cost by assurance tier Whether low-risk and high-risk work are being averaged together

A narrow online ratio can also be useful:

Verification Overhead Ratio = C_verification / C_execution

A fuller assurance-load ratio includes human review, monitoring and attributable evaluation infrastructure. The two should not be mixed in one chart.

For high-impact work, a risk-adjusted extension can include expected residual failure loss. That means multiplying the estimated probability of an undetected material failure by its estimated impact. It is conceptually useful but should not be presented as precise unless those estimates are defensible.

The best number is not automatically the smallest CPVO. A workflow can lower its apparent cost by skipping checks. The assurance profile is part of the product and must remain attached to the metric.

What this means for ChaseOS today

ChaseOS is an active governance and control-plane framework in developer preview. It is not a mature production SaaS or a generally available managed agent service.

Its current operating patterns already make verification overhead visible:

  • sensitive actions sit behind explicit human approval gates;
  • dry runs separate proposal from execution;
  • evidence packets make actions reviewable;
  • bounded permissions reduce the blast radius;
  • production deploys, DNS changes, credentials, public posting and customer commitments require authority the agent cannot grant itself.

These are not side documents attached after the real work. They are part of what makes the outcome governable.

The cost shows up as extra tool calls, screenshots, hashes, tests, reviewer minutes and waiting time. The benefit shows up as clearer boundaries, easier rollback and less reliance on the executor’s confidence.

The current ChaseOS governance build log explains those authority seams. The Chaser Agent Source Card build log describes a review-first V0 that separates source claims, evidence, uncertainty, action candidates and memory candidates into deterministic local artifacts.

Chaser Agent V0 has no browser control or runtime authority by default. That boundary matters because the next direction should start from reviewability before adding capability.

Proposed: a verification plane around executor runtimes

Proposed ChaseOS verification plane with a current-versus-proposed legend, risk routing, verifier checks and pass, repair, escalate or block outcomes.

Open the figure to inspect the current-versus-proposed architecture labels.

The following is a proposed extension, not a live product claim.

A future ChaseOS verification plane could begin with a versioned task contract and risk classification. An executor runtime performs the bounded work and returns the artifact, trace and evidence. A verifier router selects the required deterministic checks, model critique, visual QA and policy checks for the assurance tier.

Task contract
  -> risk classification
  -> executor runtime
  -> artifact + trace + evidence
  -> verifier router
       -> deterministic checks
       -> environment checks
       -> model critic
       -> visual QA
       -> policy/security checks
  -> pass | repair | escalate | block
  -> authorised action where required
  -> post-action monitoring
  -> outcome ledger

Current ChaseOS controls would remain the authority layer. The verifier could recommend pass, fail, repair or escalation. It would not self-authorise a restricted action.

A future Chaser Agent direction could provide a first-party verification and refinement harness around other runtimes. It could inspect artifacts and traces, invoke deterministic validators, route a calibrated model critic, inspect rendered evidence, score operator rubrics, produce bounded repair instructions and assemble an evidence packet.

That is refinement, not fine-tuning. The model weights do not change when a critic requests a repair.

Human corrections, rejected artifacts and repair traces could later become regression tests, rubric examples or evaluation data. Only after consent, privacy, provenance and a persistent measured gap would true model fine-tuning become a separate decision.

Feedback should become eval data before training data

Feedback workflow showing human decisions becoming evidence, regression evaluations and harness improvements before any optional governed fine-tuning decision.

Open the figure to inspect the governed feedback sequence.

Agent systems often describe every correction loop as learning. That collapses several different mechanisms.

A system can improve by changing prompts, tools, retrieval, rubrics, memory or examples without training a model. It can use test-time critique to refine the current artifact. It can store an approved example for later retrieval. None of those updates model weights.

The safer sequence is:

human decision and correction
  -> versioned evidence record
  -> regression evaluation
  -> rubric, prompt or tool improvement
  -> measured next run
  -> governed dataset if justified
  -> fine-tuning only if the evidence supports it

Any reusable dataset needs answers about ownership, consent, tenancy, sensitive data, retention, deletion, provenance and whether the original human judgement was later shown to be wrong.

ChaseOS should not silently train on user work. The verification ledger should preserve why an example exists and which policy permits its reuse.

Future Cloud and marketplace economics

ChaseOS Cloud is planned, not live. The credits ledger exists, while the proposed provider-gateway reserve-execute-settle state machine remains specified rather than deployed.

A verification-aware extension could reserve budget for execution, mandatory checks, bounded retry allowance and optional human review. Settlement could expose separate buckets for execution inference, verification inference, tools, repair, human allocation and monitoring.

The key policy is simple:

A workflow must not lower its bill by silently skipping checks required by the user’s assurance policy.

A future workflow marketplace should also disclose more than a description and raw price. A proposed assurance manifest could show the task-contract version, allowed tools, risk tier, required verifiers, approval points, rollback behaviour, evidence artifacts, evaluation version, sample size, repeated reliability, expected human minutes and known limitations.

Those would be descriptive assurance profiles, not certifications or guarantees.

The buyer should be able to distinguish the cheapest raw run from the cheapest accepted outcome, the most reliable repeated outcome, the lowest human burden and the strongest assurance profile.

That is a better market than ranking workflows by token price alone.

The unit businesses actually buy

Businesses do not buy tokens. They do not buy an agent’s confident TASK COMPLETE message either.

They buy outcomes they can accept under a declared level of assurance.

That means the system has to measure the work required to establish the result, not only generate it. Tests, environment checks, critics, visual QA, repair, evidence, human authority and monitoring all have costs. The verifier has failure modes of its own. Higher-risk work needs stronger and often more expensive assurance.

Cost per task remains a useful step beyond token pricing. Cost per Verified Outcome makes the missing layer explicit.

The operating standard I want is this:

Do not call the task complete because the agent stopped. Call it complete when the declared outcome exists, the required evidence passes, the authorised decision has been made and the remaining risk is visible.

That is the unit the next generation of agent systems will have to price, prove and improve.

Read more practical AI systems research at https://chaseintech.com/articles

Follow ChaseInTech:

Sources

Building something ambitious?

I take on a small number of projects at a time.

Available for selected agentic AI, automation, full-stack product and technical architecture work.

Work with mechase [at] chaseintech.com