I use these systems enough to care less about the feature headline than the authority behind it. A model can be excellent at code and still be unsafe to leave unattended if I cannot answer basic questions about the run.

Which model executed the task? What tools could it reach? Did a subagent inherit the same permissions? Could a project file silently change the operating rules? What happened after a tool returned data? How much did the whole task tree spend? Could I stop it and reconstruct what it had already done?

The latest Claude Code and Codex releases are interesting because both products are starting to expose answers.

Claude Code adds policy around model changes

Claude Code v2.1.251 added PreModelSwitch and PostModelSwitch hook events. The release notes say these hooks can block, confirm or annotate a model switch.[1]

That sounds small until a session can move between models with different prices, capabilities, context state or policy requirements. A model change is an operating event. It can alter how the next action is produced.

A useful pre-switch hook can check which model the session wants to use, why the change is happening, what state will be carried forward and whether the new route is allowed for that task. A post-switch hook can record what changed and attach the decision to the run.

The hook does not prove that the switch is safe. It gives an operator somewhere to enforce policy.

Claude Code also added per-session prompt-cache information and spend-limit visibility for developers using a Claude apps gateway with those limits. These additions make cost less mysterious, especially in long sessions where cache misses and re-caching can change the shape of the bill.[1]

Cost visibility is not the same as cost control, but invisible spending cannot be governed at all.

Restricted mode changes the default authority

Claude Code v2.1.248 introduced a restricted mode. In that mode, command and code-execution tools, WebFetch and broad file permissions are removed by default. The release also says bypass-permission mode is refused while restricted mode is active.[2]

That is a more important design choice than another convenience command.

Most agent incidents do not begin with a dramatic exploit. They begin with a tool that was available because nobody removed it. A coding task that only needs repository reading and a narrow edit should not automatically inherit shell execution, unrestricted network access and every writable path on the machine.

Restricted mode moves the starting point. The operator has to add authority instead of remembering to subtract it.

This is still configuration, not a guarantee. Teams can choose a broader profile, allow the wrong path or misunderstand what a plugin can do. But the product now exposes a clearer least-authority lane.

The filesystem fixes show why policy must survive runtime changes

The same Claude Code release fixed file tools following a symlink that had been swapped inside the working directory after the permission check. It also fixed plugin commands declared in a marketplace entry being able to point outside the plugin directory.[1]

These fixes are a reminder that checking a path once is not enough. The resource used at execution time must still be the resource that was approved.

An agent can ask permission for a path that appears safe, then reach somewhere else if the path changes before the operation. A plugin can look contained in its manifest and still escape that boundary if the resolved command points outside it.

A real control plane has to govern the resolved runtime object rather than trusting the earlier string that named it.

Codex is exposing coordination and interception points

Codex 0.150.0 added task-to-task coordination primitives and interrupt handling.[3] Codex 0.151.0 followed with changes around MCP tool results, sandbox behavior, permission state and nested subagent accounting.[4]

The MCP result interception point is particularly useful.

Tool governance is often described as deciding whether an agent may call a tool. That is only half the path. The result can also contain hostile instructions, malformed records, secrets, unexpected links or data that should not be allowed into the next reasoning step unchanged.

A control plane needs somewhere to inspect, redact, reject, replace or annotate the result before the agent treats it as trusted context.

This matters even when the tool itself is legitimate. A browser, issue tracker, documentation server or repository search can return untrusted content. Permission to call the tool does not make every returned byte safe.

Subagent budgets need one root contract

Codex 0.151.0 also tightened accounting for nested subagents so descendants can remain under a root token budget.[4]

Without that relationship, a parent agent can appear bounded while repeatedly spawning work underneath it. The top-level session stays within its own apparent limit, but the full task tree does not.

The same problem applies to time, API spend, tool calls and side effects. A parent task should not escape a limit by delegating the expensive part.

I want one task contract at the root:

• maximum model and API spend;
• maximum elapsed time;
• maximum subagent depth and count;
• allowed tools and paths;
• actions that require fresh human approval;
• evidence required before the run can close.

Every child should inherit that contract unless the operator explicitly narrows or extends it. A child must not be able to grant itself more authority than the parent had.

Streaming is useful, but background work still needs receipts

Claude Code v2.1.251 added live streaming of a foreground subagent's tool calls and results to Remote Control clients. The release notes distinguish this from background subagents, which still show status rather than the same live detail.[1]

That distinction matters.

Foreground visibility helps an operator understand a task while it is happening. Background agents create a different requirement. If the system cannot show every step live, it needs durable receipts after the fact.

A status such as "completed" is not enough. I need the task input, model route, tool calls, files changed, commands run, external effects, test results, approval events and final output. If the work failed, I also need to know what state survived.

Live observation and replayable evidence solve different problems. Mature agent systems need both.

This is not a benchmark contest

Nothing in these release notes proves that Claude Code or Codex writes better software. It does not establish that either sandbox is complete, that every plugin is safe or that a nested task always stays inside the intended boundary.

The releases do show where the products are investing.

They are adding policy seams around model routing. They are making authority profiles explicit. They are exposing cost and cache state. They are providing more control over coordination and tool results. They are repairing gaps where an approved path could become a different runtime resource.

That is control-plane work.

The model still matters. So do the prompt, repository, tests and human reviewer. But once a coding agent can coordinate tasks and execute tools, the surrounding control system decides whether that intelligence can be used repeatedly without turning every run into a trust exercise.

What I will measure next

I do not want to judge these controls from release notes alone.

The practical test is a bounded repository task run under both systems with the same contract. I would record:

• which policy decisions are visible before execution;
• how model changes are represented;
• whether child tasks inherit limits;
• what tool results can be inspected before reuse;
• how filesystem boundaries behave under path changes;
• what evidence remains after interruption or failure;
• how much human time is required to reconstruct the run.

That test would not declare a universal winner. It would show which control surfaces are usable, which are only configurable and which still rely on operator discipline.

Coding agents are getting more capable. The more important change is that their authority is finally becoming something we can inspect and shape.

Sources

[1] https://github.com/anthropics/claude-code/releases/tag/v2.1.251 - Claude Code v2.1.251 release notes [2] https://github.com/anthropics/claude-code/releases/tag/v2.1.248 - Claude Code v2.1.248 release notes [3] https://github.com/openai/codex/releases/tag/rust-v0.150.0 - Codex 0.150.0 release notes [4] https://github.com/openai/codex/releases/tag/rust-v0.151.0 - Codex 0.151.0 release notes

A third-party report published on 29 August attributes surfaced outputs to an OpenAI codename, "Astra." The screenshots and commentary are interesting. They may even turn out to be early evidence of a future model. They are not an API identity, model card, price sheet or release note.[1]

I follow rumors because waiting for a polished launch page can leave a builder behind. I also refuse to plan a product around a codename that a vendor has not confirmed.

Both positions can be true. You can pay attention without pretending the evidence is stronger than it is.

This article separates the current GPT-6 story into four lanes: reported, observed, official and unknown.

Reported: outputs are being attributed to Astra

TestingCatalog published "First outputs from GPT-6 'Astra' model from OpenAI" on 29 August. The page categorises the story under AI rumours and describes the outputs as reportedly generated with extended reasoning.[1]

That wording matters.

"Reportedly" means the publication is passing on an attribution. It does not mean OpenAI has confirmed the model identity. The output may come from an internal test, a limited evaluation route, a future product checkpoint or something else entirely. An outside reader cannot resolve that from the screenshot alone.

The report is a useful lead. It is not a release contract.

A release contract would include a stable model name, access route, documentation, pricing, limits and support boundaries. None of those are established by the existence of a surfaced output.

Observed: screenshots can show behavior, not product identity

A screenshot may show an interface, a response and a label. It can help answer narrow questions:

• Was this output presented under the name Astra?
• What kind of prompt appears to have been used?
• How long did the visible run take?
• Did the result contain code, analysis or tool output?

It cannot establish the whole product behind the image.

A label can be an internal alias. A response can be selected from many attempts. A timing figure can exclude queueing, retries or hidden work. A result can look strong without revealing the model version, system prompt, tool access or evaluation conditions.

This is the same reason I do not treat a vendor benchmark chart as production reliability. The visible result is one part of the harness.

For GPT-6 rumors, the harness is mostly missing.

Official: OpenAI has published GPT-5.6 pages, not GPT-6

I checked OpenAI's live sitemap on 31 August. It contains first-party GPT-5.6 surfaces, including an official page for GPT-5.6 in Kiro. I did not find a GPT-6 model page, GPT-6 announcement, Astra model page or Bel model page in that sitemap.[2][3]

This is a bounded negative finding.

A missing sitemap entry does not prove that OpenAI has no internal model, checkpoint or experiment with those names. It only proves that the first-party publication surface I checked did not expose the product documentation a developer would need.

That is enough to reject one claim: GPT-6 has not been verified as a public product through the official surface used in this audit.

It is not enough to claim that every rumor is false.

Unknown: the details that determine whether builders can use it

Most of the useful questions remain unanswered.

Model and API identity

What exact identifier would a developer send? Would GPT-6 be one model, a family or a routing layer? Would Astra survive as a public name?

Without a stable identifier, there is nothing to integrate or test.

Price

A model can look excellent in a screenshot and still be a poor fit for repeated agent work if output tokens, tool calls, cache behavior or long reasoning make the successful task expensive.

The rate card is only the start. I would still measure cost per successful, checked task.

Context and state

A context-window number does not tell us how well the model uses long repositories, whether it compacts state, how it preserves instructions or what happens after repeated tool calls.

The practical question is how much useful state survives the workflow.

Tool use

Does the model call tools directly? Can it work through Codex or another managed harness? How are permissions represented? Can results be inspected before the model reuses them?

A stronger model with weak tool boundaries can increase the blast radius of an error.

Availability and limits

Who gets access? Which regions, plans and APIs? What are the rate limits? Is the model suitable for background work or only interactive sessions?

A demo that cannot be scheduled or budgeted is not an agent platform.

Evaluation method

If new benchmark results appear, I want the exact benchmark version, harness, settings, number of trials, grader and failure treatment.

A number without the run contract is advertising material, not operating evidence.

Safety documentation

A public system card or equivalent document should define the tested risk areas, deployment restrictions and known limitations. It will not prove the system is safe, but it gives outside reviewers something concrete to examine.

Parameter counts are not a substitute for evidence

Some GPT-6 coverage has repeated very large parameter-count claims. I am not treating those figures as facts.

Parameter counts can be misunderstood even when the vendor publishes them. Total parameters, active parameters, expert routing and inference cost are different quantities. A speculative number tells us almost nothing about the cost or reliability of a real task.

The same applies to predictions that a model will arrive within days. A market price, anonymous comment or social post can measure expectation. It cannot set OpenAI's release schedule.

If OpenAI publishes the model tomorrow, the evidence changes tomorrow. The correct response is to update the article, not pretend today's rumor was official all along.

How I handle rumors in a real product workflow

I use a simple rule.

A rumor can enter research. It cannot enter production routing.

That means I can:

• save the source;
• record the date and exact wording;
• identify testable claims;
• prepare a provisional evaluation plan;
• watch the first-party documentation surfaces.

I will not:

• add an unverified model to a live registry;
• promise a customer that a rumored capability is coming;
• publish a price or release date without a first-party source;
• build a governance claim around a codename;
• present a screenshot as a system card.

This keeps the research lane fast without contaminating product truth.

What would change this article

A first-party OpenAI announcement would move part of the story from reported to official.

The minimum useful publication would include:

• exact model or family name;
• API or product availability;
• price and rate limits;
• context and tool-use documentation;
• evaluation details;
• safety or system documentation;
• migration guidance from current models.

Once those exist, the article should stop asking whether GPT-6 is a release. It should test whether the release does what the documentation claims.

Until then, the honest headline has a question mark.

Sources

[1] https://www.testingcatalog.com/first-outputs-from-gpt-6-astra-model-from-openai - TestingCatalog: reported GPT-6 Astra outputs [2] https://openai.com/sitemap.xml - OpenAI sitemap [3] https://openai.com/index/gpt-5-6-in-kiro - OpenAI: GPT-5.6 in Kiro

Anthropic opened a research preview of the Model Hardware Standard on 27 August. MHS is described as a shared specification for agents to operate devices from different manufacturers. The launch examples include scientific and manufacturing equipment, and Anthropic names Hugging Face LeRobot and Raspberry Pi among the early adopters.[1]

The shared interface is useful. The harder problem sits around it.

Which device is the agent talking to? What exact action is allowed? How far can a motor move? What happens when two devices need the same resource? Who can stop the run? What evidence proves the device reached the expected state?

Once software can produce physical side effects, those questions are part of the product.

A common command format does not create common safety

Laboratory and manufacturing equipment often comes from different vendors, with different interfaces, timing rules and failure behavior. A shared specification can make coordination easier because the agent does not need a separate ad hoc integration for every device.

That removes friction. It can also concentrate authority.

An agent that previously controlled one instrument through a narrow script may now have a route to several instruments through one standard. The interface becomes easier to use, and the blast radius of a bad instruction can grow at the same time.

This does not make the standard a bad idea. It means the safety model cannot stop at protocol compatibility.

A device may accept the same shape of command while requiring completely different operating limits. A microscope stage, liquid handler and robotic arm do not share the same consequences, even if one agent can address all three.

The control plane must understand the device, action and physical context behind the call.

Device identity has to be stronger than a friendly name

A physical workflow needs a verified device identity.

"Microscope 2" is not enough if the operator cannot prove which hardware instance, firmware, calibration state and attached components sit behind that label.

The task contract should bind to a specific device record. That record should include the properties that matter for safe operation:

• manufacturer and model;
• unique device identity;
• firmware and driver version;
• calibration or maintenance state;
• attached tools or consumables;
• physical location;
• supported safety interlocks.

The agent should not silently substitute a different device because the preferred one is unavailable. Substitution changes the task.

If the operator approves one microscope and the runtime routes to another, the approval no longer describes what happened.

Capability leases should expire with the task

I would not give a hardware agent a standing permission such as "can control robotic arms."

I would issue a short capability lease tied to one task.

The lease would define:

• the exact device;
• allowed commands;
• bounded operating range;
• maximum force, speed, temperature or volume where relevant;
• start and expiry time;
• maximum number of actions;
• required sensor conditions;
• actions that trigger fresh human approval.

A task that needs to move one stage by a small distance should not inherit permission to run every motion function on the device.

The lease should also die when the task closes, fails or loses contact with the supervising system. An unattended session must not leave durable hardware authority behind.

Timing and concurrency are part of the safety model

Physical work happens over time.

A command can be valid in isolation and dangerous in sequence. A liquid handler may need to wait for a plate to reach the correct position. A robotic arm may need an area to remain clear. A microscope may need the sample to stop moving before capture.

When one agent coordinates several devices, it must handle dependencies, resource locks and timeouts. It also needs a defined response when the physical state does not match the expected state.

Retrying the same command is not always safe.

In software, a retry might create a duplicate record. In hardware, a retry might repeat a dose, move a stage twice or apply force again. The system should treat every retry as a new physical decision unless the device can prove that the first action did not occur.

Idempotency cannot be assumed. It has to be designed into the device operation or established from sensor evidence.

Interlocks must sit below the model

A language model should not be the last safety check before motion.

Hard limits and emergency stops need to exist below the reasoning layer. The model may propose an action, but the device controller or independent safety layer should enforce boundaries the model cannot override.

That includes:

• travel limits;
• collision zones;
• force and speed ceilings;
• temperature and pressure constraints;
• door, cover or enclosure state;
• presence sensors;
• emergency-stop state.

The model can explain why it wants an action. It should not be able to argue a physical interlock out of the way.

A useful architecture separates proposal from authority:

1. The agent proposes a device action.
2. The policy layer checks the task and capability lease.
3. The safety controller checks current physical conditions.
4. The device executes within hard limits.
5. Sensors confirm the resulting state.
6. The system records the receipt.

If any layer cannot prove its preconditions, the action should stop.

Human approval needs an exact physical target

A generic confirmation such as "Allow this experiment?" is weak approval.

The review should show the operator:

• the device and current state;
• the proposed action and operating range;
• the reason for the action;
• expected physical result;
• known hazards or irreversible effects;
• what the system will measure afterward;
• how to stop or recover.

The approval should bind one person to one action scope and expire if the device state changes before execution.

If the sample moved, the tool changed or the calibration expired, the old approval should not carry forward automatically.

Physical state needs a receipt

Software agents often close a task when a command returns success. That is not enough for physical work.

A controller can acknowledge a command without proving the real-world outcome. The arm may stall. The liquid transfer may be incomplete. The image may be captured from the wrong position.

The receipt should include both the command and the observed result.

Depending on the device, that might mean:

• encoder position;
• sensor reading;
• before-and-after image;
• measured volume or weight;
• timestamped controller status;
• interlock state;
• operator intervention;
• error and recovery path.

The evidence does not guarantee scientific validity. It proves more narrowly what the system observed during the action.

That distinction matters. A correct device movement can still produce a bad experiment. The agent-control receipt and the scientific conclusion are separate layers of evidence.

The research preview is not a safety certificate

Anthropic says the MHS research preview will help build additional safety evaluations and strengthen protections for AI use in the physical world. The company also says it is developing a physical safety roadmap.[1]

That is a statement of work in progress.

The announcement does not establish general availability, field prevalence or proof that agents can safely run arbitrary laboratories. Early integrations show that the interface can be explored with real devices. They do not settle governance across every machine, environment and experiment.

The most useful response is not to dismiss the standard or celebrate it as solved. It is to test the boundaries while the specification is still being shaped.

What I would test before trusting a hardware agent

I would start with a simulator or a low-consequence device task and force the system through failure cases.

The test plan would include:

• wrong device identity;
• expired capability lease;
• stale calibration;
• unexpected sensor state;
• conflicting device schedules;
• partial physical completion;
• network loss during execution;
• duplicate command delivery;
• human stop during motion;
• recovery after a failed receipt.

I would also test the evidence path. Can an independent reviewer reconstruct which device moved, under which approval, within which bounds, and what sensors observed afterward?

If the answer is no, the system is not ready for unattended physical authority.

Where ChaseOS fits, and where it does not

ChaseOS does not currently claim a shipped MHS integration.

The relevant connection is architectural. ChaseOS already treats authority, approval and evidence as separate parts of agent work. A future physical-device lane would need to extend that model with device identity, short capability leases, interlocks, emergency stops and physical-state receipts.

That is a proposed direction, not product truth.

MHS makes the device interface easier to imagine. The operating system around it still decides whether an agent can use that interface safely.

The command format may be shared. Responsibility cannot be delegated to the format.

Sources

[1] https://www.anthropic.com/news/model-hardware-standard-research-preview - Anthropic: Previewing the Model Hardware Standard

The public release manifest, website download and installed build on the primary test system all resolve to the same V1.1.0 candidate. The installer was downloaded back from the public URL and matched against its published SHA-256, rather than treating a successful upload as proof by itself.

Existing V1.0.9 installations can now discover V1.1.0 through Studio's update channel. A second computer is the next useful acceptance environment: it should receive the update prompt, install the signed package and then confirm its own installed version. That second-machine completion is a test to perform, not something this article assumes has already happened.

Intake now previews the graph change

The Intake flow can hand a proposed filing plan to the graph without silently adding it. The proposed node stays highlighted, existing connections remain visible, and the operator can zoom to the affected area or return to Intake.

This is the useful distinction: the graph preview is an inspection state. It is not permission to write the note.

Research Collections keeps evidence and destination together

Research Collections now puts the reason for the suggestion, resolved evidence, target path, proposed node, new links and rewrite count into one review surface. The example above proposes one collection node linked to 17 existing notes with zero rewrites.

The interface also states the scope of the destination and keeps Inspect in Graph, Queue for approval and Dismiss suggestion as separate decisions. That separation matters because previewing a useful structure should not automatically promote it into canonical knowledge.

Approvals exposes where a request came from

Approvals now groups requests by Studio and local work, agent runtimes and external connectors. The retained signed-build capture shows the truthful empty state: nothing was waiting at the moment of QA. Separate interaction evidence covers approve and decline journeys, but an empty packaged screenshot is not mislabelled as proof that a live request was executed.

What was actually verified

• The Windows installer is Authenticode-signed and timestamped.
• The public V1.1.0 installer is 531,616,112 bytes.
• Its public readback SHA-256 matches the published release hash.
• A V1.0.9 version check reports V1.1.0 as available.
• A V1.1.0 version check reports the installation as current.
• Signed packaged-app visual QA retained distinct Intake, Research Collections and Approvals captures against a real vault.

The remaining cross-device acceptance is deliberately separate: install the update on the second computer, reopen Studio, confirm V1.1.0 there and then test that machine's own runtime and Cloud state. A website publication cannot prove those machine-local outcomes.

Download ChaseOS Studio V1.1.0 · Read the full public changelog

The Solo Operator AI Workspace packages a different approach. It installs a small operating layer inside a project: a project map, runtime instructions, protected paths, work orders, approval gates, durable state notes and evidence templates. The model can change; the work contract remains visible.

See the complete product system and visual diagrams.

The operating loop

project truth
    -> bounded work order
    -> job-shaped runtime profile
    -> checkpointed execution
    -> named acceptance checks
    -> durable closeout evidence

The point is not to turn every task into bureaucracy. A quick edit can use a short bounded profile. A multi-day build needs higher reasoning effort, checkpoints and durable current-state notes. A visual refinement task needs observable renders. A computer-use task needs a visible foreground session and evidence of what happened in the interface. A release task needs build, package and destination proof.

Route the job before choosing the model

Model rankings move quickly and rarely describe a person's exact project. The workspace begins with observable job characteristics instead:

• duration and ambiguity;
• code, visual or computer-use surface;
• authority and external effects;
• checkpoint frequency;
• acceptance evidence required.

That produces a work profile such as quick bounded, deep build, visual refinement, computer use, supervised runtime or release proof. A founder can then select a suitable commercial, local, open or regionally developed model based on evaluations, privacy, language, licence, cost and hardware—not the assumption that one model is universally best.

Durable context without copying a private system

The kit is informed by patterns used while operating and releasing several software products, but it does not contain ChaseOS, private business data, credentials or governed internal policy.

Instead, it exposes reusable abstractions:

This gives a new session enough context to resume the project without pretending the old conversation itself is a reliable database.

“Done” is not one state

The workspace uses an evidence ladder:

1. Implemented — files or behaviour exist.
2. Tested — a named check passed.
3. Visually verified — the rendered result was inspected.
4. Released — an accepted artifact was built or submitted.
5. Live — the destination was read back and verified.

A lower rung does not prove a higher one. A passing unit test does not prove a production deployment. A storefront draft does not prove a customer can buy. A generated image does not prove the final page uses it correctly.

What Version 1.0 has actually proved

Version 1.0 contains cross-runtime project adapters, six work profiles, four reusable skills, a safe bootstrapper, a workspace doctor, a private-data scanner, a sample project, an illustrated manual and deterministic release packaging.

Its product-tool test suite passed ten tests. The sample project passed three tests. A fixed, public-safe Codex fixture passed five acceptance tests while its protected files remained unchanged. The privacy scan reported no private-data findings in the customer release.

The acceptance run also found a real packaging defect: several runtime-specific installs referenced a shared contract without installing it. The bootstrapper was corrected and a regression test now covers that path. That defect is more useful evidence than a perfect demo because it shows the package was exercised as a buyer would use it.

Compatibility remains deliberately specific:

• Codex completed one bounded live task on Windows under the recorded fixture constraints.
• Claude Code's installer path and workspace doctor passed, but live execution was blocked by expired operator authentication.
• Gemini CLI and GitHub Copilot adapters are packaged but do not yet carry live runtime claims.
• macOS and Linux acceptance remain future checks.

What the product does not provide

It does not include a model, API credits, a hosted agent, a 24/7 server, unrestricted computer control, access to private ChaseInTech systems or a guarantee that an agent will produce correct work.

It gives the operator a clearer surface for assigning, constraining, resuming and verifying that work. The buyer remains responsible for model accounts, runtime infrastructure and approval decisions.

That boundary is the product's central idea: autonomy becomes more useful when authority and proof remain explicit.

Explore the Solo Operator AI Workspace and its verified Gumroad buying route.

Google released Gemini 3.7 Flash on August 13, only three weeks after 3.6 Flash, and called it its most intelligent workhorse model for coding and agents.[1] That pace alone should make builders pause before treating any model choice as permanent. The more interesting part is what Google changed: the model is cheaper to run through the end of 2026, scores better across several agent-style tests, and is already generally available rather than sitting behind a preview label.[2][3]

I would test it. I would not yet call it reliable.

That distinction matters because agent work is not a single answer. An agent has to plan, call tools, read the result, recover when a tool fails, preserve state and stop at the right moment. A model can improve at all of those behaviours and still fail too often for an unattended production workflow.

The price cut is real, but temporary

Gemini 3.7 Flash costs $0.75 per million input tokens and $3.75 per million output tokens on the paid Gemini API. Output billing includes thinking tokens. On January 1, 2027, both rates double to $1.50 and $7.50.[2] Google has also applied the promotional rate to Gemini 3.6 Flash, so this is not a clean permanent price advantage of 3.7 over its immediate predecessor.[3]

Take a medium agent run that consumes 100,000 input tokens and 20,000 output tokens. At the introductory rates, the model bill is $0.15. At the announced 2027 rates, the same token mix is $0.30. That calculation excludes search, maps, storage for cached context, the agent runtime, retries and human review. Google's pricing page, for example, lists 5,000 free Google Search requests per month across Gemini 3.x models on the paid tier, then $14 per 1,000 requests.[2]

This is where cheap tokens can fool a team. An agent that costs 15 cents per attempt but needs three attempts, two verifier passes and a human rescue is not a 15-cent outcome. Cost per successful, checked task is the number worth tracking.

There is encouraging evidence on that front. Cognition's FrontierCode 1.1 leaderboard reports Gemini 3.7 Flash at a 43.6% score and an average rollout cost of $1.82, compared with 34.4% and $4.04 for Gemini 3.6 Flash.[7] That is a 9.2-point score gain while the measured rollout cost fell about 55%. FrontierCode is designed around whether maintainers would merge a generated pull request, using blocking criteria, tests, rubrics and other verifiers. It is a better signal than a toy code-completion test, although its score is a weighted benchmark result, not a production success rate.[7]

Better at agent work is a fair claim

Google reports gains across coding, web development, document work and tool use. The headline results include 65.3% on DeepSWE v1.1 versus 49.0% for 3.6 Flash, 1588 versus 1538 Elo on WebDev Arena, and 30.4% versus 17.0% on AutomationBench.[1]

Some of those numbers have useful outside support. The live DeepSWE leaderboard listed 3.7 Flash at roughly 65% pass@1 on August 20, with an average cost of $2.18, 107,000 output tokens and 125 steps across 113 long-horizon engineering tasks.[8] Pass@1 means the first submitted attempt succeeds, rather than giving the model several parallel chances and selecting the best. The result is impressive, but Google's methodology says its own 3.7 Flash DeepSWE number was self-computed with a mini SWE agent harness, LiteLLM 1.96 and high thinking.[6] An independent leaderboard listing is useful corroboration, not proof that an independent party reproduced Google's exact run.

Code Arena provides a different kind of evidence: human preference votes on generated web work. Its August 19 leaderboard showed Gemini 3.7 Flash High at 1588 Elo, with a plus-or-minus 13 interval and 2,545 votes. The listing was marked preliminary.[9] The gap over 3.6 Flash is visible, but the uncertainty bands and different vote counts matter. Elo measures comparative preference under that arena's setup, not whether the resulting app is secure, maintainable or correct behind the interface.

The weakest headline evidence is AutomationBench. Google's model card reports 30.4% on a private set, and its evaluation notes say the figure comes from the official public leaderboard.[5][6] A private test set can reduce contamination, but outsiders cannot fully inspect the tasks or rerun the exact evaluation. More importantly, 30.4% still means the model did not complete most evaluated workflows under that setup.

This is why I am comfortable saying 3.7 Flash looks better at agent work. The direction appears across different task types and benchmark owners. I am not comfortable turning that into "reliable agents" without workload-specific evidence.

Reliability needs a denominator

Google says the model adapts to roadblocks, clarifies intent and uses tools with fewer retries.[1] Those are product claims, not published service-level measurements. The release does not provide repeated-run success distributions, tool-call error rates, p95 task latency, timeout frequency, recovery quality after partial failure or the share of tasks that still need a human.

The model card is blunt about some boundaries. Gemini 3.7 Flash can hallucinate, may be slow or time out, and has uneven knowledge freshness: Google gives March 2026 as the cutoff while warning that some domains may only reflect January 2025.[5] The same card says the model can complete individual coding tasks but lacks the independence to chain them into an end-to-end research workflow without human intervention.[5] That sentence should sit next to every claim about autonomous agent work.

Reliability is also partly a property of the harness around the model. I would evaluate 3.7 Flash with a fixed task contract: what counts as success, which tools it may call, how many steps it gets, which side effects require approval, and what deterministic checks must pass before completion. Then I would run the same tasks repeatedly and track first-pass completion, verified completion after repair, cost per verified outcome, timeouts, unsafe actions and human minutes per task.

A benchmark winner can still be the wrong model for a workflow if it is erratic. For a code agent, compile and test the patch, inspect the diff and keep repository permissions narrow. For an email or Workspace agent, stage the draft and proposed file changes before sending or overwriting anything. The model can choose actions; your system must decide which actions are allowed to become real.

What builders can use today

Gemini 3.7 Flash is a stable model with the ID gemini-3.7-flash. Google lists it as generally available and ready for production use through the Gemini API.[3][4] It accepts text, images, video, audio and PDFs, with a 1,048,576-token input limit and 65,536-token output limit. It supports function calling, code execution, search grounding, file search, structured output and low, medium or high thinking. Computer use is supported, but that tool remains in preview.[4]

The model is also available through Google AI Studio, Android Studio, Gemini Enterprise Agent Platform and the Gemini Enterprise app. Gemini Spark began using it for eligible Google AI Pro and Ultra subscribers in supported countries on launch day.[1] Availability is broad, but product access and capability maturity are not the same thing. A GA model using a preview computer-use tool still has a preview dependency in that workflow.

Google says its automated safety evaluations were similar to 3.6 Flash overall, with specialist manual red teaming finding no egregious concerns and launch thresholds met. The model did not reach Google's tracked or critical capability levels, although cyber and one CBRN assessment reached alert thresholds below those levels.[5] These are Google-run safety assessments, not an independent safety audit. The card also says a separate Gemini 3.7 Frontier Safety Framework report will be published shortly.[5]

There are documentation rough edges worth noting. Several 3.7 model-card sections defer to older Gemini cards for architecture, training data, acceptable use and safety policy detail. At least two cross-references have mismatched labels and destinations. That does not prove a model defect, but it weakens the audit trail available to a buyer today.[5]

My builder verdict is simple. Gemini 3.7 Flash has earned a place in the evaluation lane. The introductory economics are good, the coding evidence is stronger than a standard vendor chart, and multiple benchmarks point in the same direction. Production promotion should wait for your own repeated, verifier-backed runs. The cheapest model is the one that gets an approved task right with the least total repair, not the one with the smallest token line item.

Key takeaways

• Gemini 3.7 Flash is GA, while individual capabilities such as computer use can still be preview features.[3][4]
• Introductory API pricing is $0.75 per million input tokens and $3.75 per million output tokens through December 31, 2026; the rates double on January 1, 2027.[2]
• Third-party leaderboards support the direction of Google's coding and web-development claims.[7][8][9]
• Google's methodology says some runs were self-computed, and Code Arena still labels the 3.7 result preliminary.[6][9]
• Higher benchmark scores do not establish production reliability. Track repeated verified completion, retries, timeouts, tool errors, unsafe actions and human intervention on your own workload.
• Keep consequential actions behind deterministic checks and explicit approval, even when the model's first-pass performance improves.

Sources

[1] https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-3-7-flash - Introducing Gemini 3.7 Flash [2] https://ai.google.dev/gemini-api/docs/pricing - Gemini Developer API pricing [3] https://ai.google.dev/gemini-api/docs/latest-model - What's new in Gemini 3.7 Flash [4] https://ai.google.dev/gemini-api/docs/models/gemini-3.7-flash - Gemini 3.7 Flash model documentation [5] https://deepmind.google/models/model-cards/gemini-3-7-flash - Gemini 3.7 Flash Model Card [6] https://deepmind.google/models/evals-methodology/gemini-3-7-flash - Gemini 3.7 Flash evaluation methodology [7] https://cognition.com/frontiercode - FrontierCode Leaderboard [8] https://deepswe.datacurve.ai - DeepSWE leaderboard [9] https://arena.ai/leaderboard/code/webdev - Code Arena WebDev leaderboard

Cloudflare has put DeepSeek V4 Pro 0813 and DeepSeek V4 Flash 0731 on Workers AI. Both models support a full 1,048,576-token context window, thinking mode and multi-turn function calling. Cloudflare is explicitly positioning them for large codebases, long-horizon agent workflows and multi-step reasoning.[1]

For AI engineers, that is more than another model listing. It creates a practical stack for repository-scale agents:

• Workers AI runs DeepSeek V4 Pro or Flash.
• Workers receives the task, assembles context and controls tools.
• R2 stores repositories, documents, checkpoints and generated artifacts.
• Cloudflare Agents carries durable state, workflows and recovery.
• AI Gateway gives the model route an observable control surface.

That combination is particularly relevant to the direction of ChaseOS. It could provide an edge-native lane for large technical dossiers, repository analysis, long-running engineering tasks and inspectable phase handovers without making the primary runtime depend on one model.

The point is not to move everything onto Cloudflare. The point is to understand what this release makes possible and where each part of the stack belongs.

What Cloudflare actually released

Cloudflare's canonical changelog names two Workers AI models:[1]

@cf/deepseek-ai/deepseek-v4-pro-0813
@cf/deepseek-ai/deepseek-v4-flash-0731

Both expose the same maximum context window: 1,048,576 tokens. Both support reasoning and function calling. They are available through a Workers AI binding, Cloudflare's OpenAI-compatible endpoint, the REST API or AI Gateway.[1]

They serve different operating roles.

DeepSeek V4 Flash 0731: the default long-context route

Cloudflare calls Flash the faster, lower-cost sibling and says this release supersedes its earlier preview with improved agentic capabilities.[1]

The current model page lists:

• 1,048,576-token context;
• reasoning;
• function calling;
• $0.44 per million input tokens;
• $1.32 per million output tokens;
• $0.014 per million cached input tokens.[3]

Flash is the natural first route for frequent work such as:

• searching a large repository;
• connecting an issue to the correct files and tests;
• analysing long logs or CI histories;
• converting a technical dossier into a structured task packet;
• maintaining a long-running agent plan across multiple tool calls;
• producing an initial implementation plan before a more expensive review.

DeepSeek V4 Pro 0813: the visual and higher-value route

Pro exposes the same listed context ceiling but adds vision support. Cloudflare currently marks it beta and lists:[2]

• 1,048,576-token context;
• reasoning;
• function calling;
• vision;
• $1.32 per million input tokens;
• $3.96 per million output tokens;
• $0.044 per million cached input tokens.

That makes Pro more relevant when the engineering evidence is not text-only:

• a UI regression with screenshots;
• an architecture review involving diagrams;
• a browser-agent failure with visual traces;
• OCR mixed with repository evidence;
• a high-value code change where an additional model route is cheaper than a failed deployment.

Both models require Workers Paid or prepaid AI Gateway credits according to Cloudflare.[1][2][3]

Why one million tokens matters to agent engineers

Most model announcements discuss context as a chat feature. Harness engineers should think of it as a working-set ceiling.

A million-token run can potentially hold:

• a substantial repository snapshot;
• issue and pull-request history;
• architecture decisions;
• test output;
• API specifications;
• runbooks;
• agent plans and previous tool results;
• screenshots or diagrams on the Pro route.

This reduces one common failure mode: an agent solves the fragment it can see while missing the contract, dependency or historical decision sitting outside its context.

But it introduces another risk. If the harness simply uploads everything, the model gets a large but badly defined workspace. Relevant evidence competes with stale files, generated output, secrets, abandoned plans and unrelated history.

The better question is not:

Can this repository fit inside one million tokens?

It is:

What verified working set gives the model enough context to finish this exact job?

That is where R2 and Workers become useful.

R2 gives the agent a workspace outside the prompt

R2 is Cloudflare's object-storage layer. Cloudflare says it stores unstructured data without the egress bandwidth fees associated with typical cloud storage services.[5]

For an agent system, R2 should not be described vaguely as "memory." It can be something more useful: a durable, inspectable workspace.

A repository-scale run can store:

r2://agent-sources/{source_hash}/repository.tar.zst
r2://agent-sources/{source_hash}/manifest.json
r2://runs/{run_id}/task-packet.json
r2://runs/{run_id}/selected-context.json
r2://runs/{run_id}/checkpoints/plan.json
r2://runs/{run_id}/checkpoints/implementation.json
r2://runs/{run_id}/artifacts/candidate.patch
r2://runs/{run_id}/verification/test-results.json
r2://approvals/{approval_id}/packet.json
r2://receipts/{run_id}/execution.json
r2://receipts/{run_id}/readback.json

The repository object is the source. The manifest tells the harness what exists. The selected-context record shows what the model actually received. Checkpoints make a long task resumable. Verification and receipt objects prove what happened afterward.

A Worker can read and write R2 objects through a bucket binding.[6] That means context does not need to arrive as one giant browser upload or remain trapped in an opaque conversation.

Workers turns storage and models into a harness

Workers should sit between the external request, R2 and Workers AI.

Its job is to make the run explicit:

1. Authenticate the caller.
2. Validate the task packet.
3. Resolve the approved R2 source objects.
4. Build the context manifest.
5. Select Flash or Pro.
6. Expose only the tools permitted for that phase.
7. Enforce token, tool-turn and time budgets.
8. Write checkpoints and artifacts to R2.
9. Run deterministic verification.
10. Require approval before protected actions.
11. Read the destination back and write a receipt.

A simplified binding configuration could expose both AI and R2 to the Worker:

{
  "ai": {
    "binding": "AI"
  },
  "r2_buckets": [
    {
      "binding": "AGENT_WORKSPACE",
      "bucket_name": "agent-workspace"
    }
  ]
}

The model router stays small:

const ROUTES = {
  flash: "@cf/deepseek-ai/deepseek-v4-flash-0731",
  pro: "@cf/deepseek-ai/deepseek-v4-pro-0813",
} as const;

async function runEngineeringPhase(
  env: Env,
  packet: TaskPacket,
) {
  const route = chooseRoute(packet);
  const context = await buildContextFromR2(
    env.AGENT_WORKSPACE,
    packet,
  );

  assertContextPolicy(context, packet);
  assertRunBudget(packet, route);

  return env.AI.run(ROUTES[route], {
    messages: context.messages,
    tools: toolsForPhase(packet.phase),
  });
}

The important code is not the two model IDs. It is buildContextFromR2, assertContextPolicy, assertRunBudget and toolsForPhase. Those functions define the harness.

Seven practical uses for this stack

1. Repository archaeology

An engineer can upload or synchronise a repository snapshot into R2. A deterministic indexer produces a file manifest, symbol map, dependency graph and recent-change summary.

Flash receives the manifest first. It selects the likely files for the task and explains why. The Worker retrieves those files and expands context only when required.

This gives the model repository-scale awareness without blindly pasting every file into the prompt.

2. Long-running migrations

Framework upgrades, database migrations and CLI consolidations rarely fit into one model turn. They need discovery, planning, staged implementation, tests and rollback.

Each phase can write a checkpoint to R2:

{
  "phase": "implementation-2-of-5",
  "completed": ["schema adapter", "compatibility tests"],
  "pending": ["worker migration", "deployment rehearsal"],
  "acceptance_checks": ["unit", "integration", "rollback"],
  "source_hash": "sha256:...",
  "artifact_hash": "sha256:..."
}

If a run stops, another model can continue from the checkpoint rather than reconstructing state from a chat transcript.

3. Multimodal debugging

The Pro route can combine screenshots with code, logs and test evidence. A browser or desktop agent can write screenshots and DOM captures to R2, then pass the exact referenced objects to the model.

The final fix still needs deterministic checks and visual readback. Vision helps interpret evidence; it does not prove the UI is fixed.

4. Technical-document workspaces

R2 can hold specifications, PDFs, diagrams, meeting decisions and implementation contracts. A Worker can build a source-bound context pack for a documentation or architecture agent.

This is useful for AI engineers working across long design histories where the model must preserve exact terminology and cite the correct source.

5. Evaluation and model comparison

The same R2 fixture can run through Flash, Pro and another provider. The harness records accepted output, retries, tool calls, latency, token use and human correction.

That gives engineers a real answer to "which model is cheaper?" Cost per token becomes cost per accepted task.

6. Agent-generated artifacts

An agent can produce patches, reports, diagrams, spreadsheets or media into a staged R2 prefix. The Worker computes hashes and builds an approval packet.

Approved artifacts move to the next stage. Rejected artifacts remain attached to the run but never reach a public destination.

7. External developer agents

A public Worker endpoint can accept a constrained task without exposing the internal runtime. The Worker authenticates the caller, applies rate limits, retrieves only allowed objects and returns a receipt-linked result.

That creates a path to agent services for developers without giving a model blanket access to ChaseOS internals.

A concrete ChaseOS pilot

The release is directly relevant to ChaseOS because it could support an isolated repository and dossier engineering lane.

The pilot should be narrow.

Input

• one repository snapshot;
• one issue or feature dossier;
• one acceptance contract;
• an explicit list of allowed tools;
• no production credentials;
• no deployment authority.

Cloudflare lane

Request
  -> Worker intake
  -> R2 source manifest
  -> DeepSeek V4 Flash navigation
  -> bounded source expansion
  -> Flash or Pro implementation plan
  -> candidate artifact in R2
  -> deterministic verification
  -> independent review
  -> human approval
  -> handover back to ChaseOS

Model routing

Use Flash by default. Escalate to Pro when:

• the task includes screenshots or diagrams;
• Flash fails a verifier more than once;
• the change crosses a high-risk boundary;
• ambiguity remains after repository navigation;
• the expected cost of failure is higher than the added model cost.

Evidence returned to ChaseOS

The lane should return:

• source manifest and hashes;
• selected-context manifest;
• model and pricing route;
• tool-call trace;
• checkpoints;
• candidate artifact hash;
• test and review results;
• human approval reference;
• execution and readback receipts where applicable.

That is more valuable than returning a message saying "task completed."

What not to do

A million-token window can encourage bad architecture. Avoid these shortcuts:

Do not load a bucket directly into the model

R2 may contain unrelated runs, secrets, customer exports or stale artifacts. The Worker must select exact objects for the exact task.

Do not use the model's summary as the checkpoint

A checkpoint should be a typed record with source and artifact hashes, completed steps and remaining acceptance checks.

Do not let function calling define the permission model

The fact that a model can call a function does not mean that function should exist for every run. Tool visibility belongs to the harness.

Do not confuse context size with reliable recall

Measure whether the model retrieves and uses the decisive evidence as irrelevant context grows.

Do not hard-code ChaseOS to DeepSeek

Put the models behind role contracts. The repository navigator, visual reviewer or implementation planner should be replaceable without rewriting the workflow.

The evaluation that matters

For every test task, record:

• selected model;
• source tokens and cached tokens;
• output tokens;
• context objects included;
• tool calls;
• verifier failures;
• retries;
• elapsed time;
• human correction time;
• accepted or rejected result.

The comparison should answer:

• Does Flash navigate large codebases accurately enough to be the default?
• Does Pro's vision improve visual-debug tasks enough to justify the cost?
• Does selective R2 retrieval outperform loading the entire repository?
• Can the agent resume from a checkpoint without losing constraints?
• Can every final claim be tied to a source, test or readback receipt?

If the answer is yes, Cloudflare becomes a useful execution lane. If not, the architecture still preserves the source objects and evaluation evidence needed to change models.

Why this release matters

DeepSeek V4 Pro and Flash are the first Workers AI models that Cloudflare lists with a full one-million-token context window.[1] That gives agent engineers a serious new working-set ceiling on an edge platform that already has compute, object storage, durable agent primitives and model routing.

For AI engineers, the opportunity is repository-scale reasoning without building every infrastructure layer from scratch.

For harness engineers, the opportunity is more important: keeping the repository, checkpoints, permissions, tools, verification and receipts outside the model.

For ChaseOS, this creates a concrete pilot path for long-horizon engineering and dossier work. R2 can be the workspace. Workers can be the gate. DeepSeek can be the reasoning route. ChaseOS can remain the system that defines the contract, chooses the lane and verifies the outcome.

That is the useful interpretation of this announcement. Not one million tokens of autonomy. One million tokens of working context inside a controlled engineering system.

Sources

1. Cloudflare Changelog: DeepSeek V4 Flash and Pro now available on Workers AI
2. DeepSeek V4 Pro 0813 on Cloudflare Workers AI
3. DeepSeek V4 Flash 0731 on Cloudflare Workers AI
4. Workers AI bindings
5. Cloudflare R2 overview
6. Using R2 from Workers
7. Cloudflare Agents documentation
8. Workers AI and AI Gateway as a unified control plane

I am rebuilding a reselling business around autonomous agents, but the goal is not to remove myself from every decision. The goal is to stop treating every repetitive action as a fresh manual task while keeping control over the decisions that carry commercial or account risk.

This instalment follows the workflow from physical stock to a public listing. It also shows why agentic commerce is harder than connecting a language model to a browser.

The workflow in Part 4

The build joins several different kinds of work:

1. inspect and identify physical inventory;
2. turn product evidence into structured listing data;
3. prepare titles, descriptions and operational records;
4. use computer control to move through the commerce interface;
5. pause at the approval points that affect pricing, account actions or publication;
6. verify that the intended listing actually reached the live state.

Each step can look simple in isolation. The real engineering challenge is preserving context and evidence as work passes between them.

A product image, an inventory record, a model-generated description and a browser action do not have the same authority. The system needs to know which inputs are evidence, which outputs are proposals and which actions require a human decision.

Why inventory is the difficult starting point

Reselling begins in the physical world. Stock can be incomplete, mislabelled, duplicated or stored under an old description. Condition is not a clean database field until someone has inspected the item and recorded it properly.

That makes inventory a poor place for blind automation.

An agent can help organise evidence, identify missing fields and prepare a draft. It should not quietly turn uncertainty into a confident product claim. The useful workflow keeps the original images and observations attached to the record, makes missing information visible and lets a human correct the item before anything public is submitted.

This is where MarginFlip fits into the wider system. It is the operational product layer for turning legacy stock into structured inventory and a repeatable resale workflow. ChaseOS provides the broader control-plane pattern: tasks, agents, evidence, approvals and durable state should remain inspectable instead of disappearing into one long chat.

Computer control is only one layer

The visible part of an autonomous workflow is often the cursor moving through a website. That is useful, but it is not the whole system.

Computer control needs a contract around it:

• the exact account and destination;
• the item being processed;
• the allowed actions;
• the data approved for entry;
• the points where execution must stop;
• the evidence required before completion can be claimed.

Without that structure, browser automation can move quickly while the business loses track of what it changed.

In this build, Hermes acts as the operating agent across the workflow while ChaseOS provides the governance direction around state, permissions and proof. Claude Code and Codex remain development lanes where appropriate. The important design choice is separation: the system that proposes work should not automatically gain unlimited authority to publish, price or modify an account.

The approval points stay human

The agents can move the workflow forward, but I retain the decisions that materially affect the business.

That includes:

• confirming that the item record matches the physical product;
• reviewing condition and disclosure language;
• approving pricing decisions;
• authorising account-level actions;
• approving publication;
• checking the live destination rather than trusting a completion message.

This is not a ceremonial "human in the loop" label. An approval is useful only when the person can see the proposed action, the evidence behind it and the consequence of accepting it.

The objective is to make each approval smaller and better informed. I should not have to reconstruct the entire job every time the system asks for a decision.

The first eBay listing is a systems milestone

Putting one product live is not proof that the complete reselling operation is autonomous or commercially successful.

It is still an important milestone because it exercises the full path across inventory, data preparation, computer control, approval and public verification. A working end-to-end path exposes problems that isolated demos hide:

• incomplete source data;
• inconsistent product naming;
• unclear authority boundaries;
• fragile browser steps;
• missing receipts;
• completion claims that do not match the public state.

The next stage is to make that path repeatable across more inventory without weakening the review gates. The useful operating metric is not how many prompts the agents produce. It is how many listings reach an accepted, verified state with a clear record of what happened.

Turning the build into a public operating log

The final Part 4 video was edited in CapCut and exported as a privacy-safe 4 minute 11 second master. Private working material was treated before publication, and the public cut keeps the focus on the workflow rather than exposing the underlying operating channels.

Publishing the build log is part of the engineering discipline. It creates a durable record of what the system could do at this point in time, what remained human-controlled and what still needs to improve.

Watch Part 4

Watch Day in the Life of an AI Engineer - Part 4 on YouTube.

The same build is also available through the verified X post and LinkedIn post.

What I am building toward

The longer-term target is not a bot that can click "List item." It is an inspectable operating system for a reselling business:

• physical inventory becomes structured evidence;
• agents prepare and coordinate bounded work;
• deterministic checks handle what can be proven exactly;
• human approvals protect commercial and account decisions;
• public actions return receipts;
• failures can be corrected without losing the workflow state.

That is the standard I am applying to MarginFlip, ChaseOS and the wider autonomous-agent stack: useful autonomy, visible boundaries and proof that the intended outcome actually happened.

Anthropic has started changing a basic assumption about AI-generated text: that once you copy it out of the chat window, its origin disappears.

Supported Claude models will embed an imperceptible, machine-readable watermark into generated text. Anthropic says the mark can travel with copied text and may survive some editing. Supported generated files can carry signed provenance metadata using C2PA instead.[1]

That sounds like a clean answer to a messy problem. It is not.

A detected watermark can show that supported Claude systems processed the text. It cannot, by itself, tell you who had the original idea, who wrote the first draft, whether Claude only corrected grammar, or whether a human reviewed every line. A missing watermark does not prove human authorship either.

That distinction matters for writers, developers and teams building AI agents. Provenance is useful evidence. It is not an authorship verdict.

What Anthropic is introducing

Anthropic describes two related marking systems.

For text, a supported Claude model embeds a machine-readable watermark directly into its output. The mark is designed to be invisible to readers and not change the meaning, quality or readability of the response. Because it is part of the text, it can move with copied and pasted content.[1]

For supported files, Anthropic uses digitally signed provenance metadata based on C2PA. C2PA provides a framework for attaching claims about an asset's origin and editing history to a signed manifest. That is different from an invisible text watermark. It is metadata attached to a file, with cryptographic machinery that lets a verifier check whether the manifest is valid.[1][3]

Anthropic says the text marking applies at the model level across supported Claude surfaces. Its help material names Claude, Claude Code, Claude Cowork, Claude Platform and supported deployments through cloud providers. Models launched in the EU on or after 2 August 2026 are expected to support machine-readable marking at launch, with Anthropic working to extend support to earlier models.[1]

The timing is tied to the EU AI Act's transparency rules. The European Commission's Code of Practice is intended to help providers comply with obligations around marking and labelling AI-generated content.[2]

This is not a small interface badge. It is infrastructure inside the output path.

The watermark is a provenance signal, not an AI detector

Most AI-writing detectors work backwards. They examine finished text and estimate whether its statistical patterns resemble machine-generated writing. That approach has produced false positives and a lot of misplaced confidence.

Anthropic's system takes a different route. The model places a signal into its own output, and a compatible detector looks for that signal later. Instead of guessing from style, the verifier checks for a mark deliberately added by the provider.

That is a better defined claim, but still a narrow one.

If the mark is detected, the safe conclusion is that a supported Claude model processed the marked text. Anthropic explicitly warns that the person may have written the material and used Claude only to proofread, translate, summarise or reformat it.[1]

The reverse is also true. If a detector finds no mark, that does not prove the text was written without AI. The output may have come from an older or unsupported model. It may be too short. Heavy editing, paraphrasing, translation or mixing with other text may weaken or remove the signal. Other models will not necessarily use Anthropic's system.[1]

So the result should not be reduced to two buttons labelled HUMAN and AI.

A better evidence model is:

This is evidence about process, not truth or ownership.

What this means for Claude Code

The coding angle is where the policy becomes operational.

If watermarking applies to supported Claude Code output at the model layer, generated code and surrounding text may carry the signal when copied into source files, documentation, tests, issue comments or pull-request descriptions.[1]

But code is hostile terrain for text watermarking.

Developers rename variables, run formatters, remove comments, extract small functions, combine suggestions from several tools and rewrite sections during review. Build tools also normalise whitespace and line endings. Anthropic has not publicly described the technical embedding method in enough detail to calculate how well the mark survives those transformations.

That missing detail should stay visible. We should not claim the system uses hidden Unicode characters, altered whitespace, token probabilities or any specific encoding until Anthropic publishes the mechanism. "Invisible" does not automatically mean zero-width characters.

For engineering teams, the practical response is not to hunt for secret symbols in every commit. It is to improve the evidence already under their control:

1. Record which model and tool were used for a task.
2. Keep the human reviewer and approval decision attached to the change.
3. Store test results and security checks with the pull request.
4. Distinguish generated code from accepted code. Generation is an event. Acceptance is a human or governed system decision.
5. Treat watermark detection as one signal in an audit trail, not the audit trail itself.

A watermark may help answer, "Did supported Claude processing touch this text?" It does not answer, "Is this code correct, safe, licensed appropriately and approved for production?"

The harness-era threat model

The risk becomes more serious when Claude is not answering one prompt in a chat window but operating inside an agent harness.

A harness can ask the model to inspect a repository, edit several files, run commands, generate tests and prepare a pull request. One model may produce the plan, implementation, documentation and review summary. If the same system generates the work and judges its own work, the audit trail can look complete while sharing one failure mode.

Watermarking does not create that risk, but it makes model involvement easier to trace. The operational response should be independent verification.

For consequential development work, I would route the final review through another model or a deterministic verifier before continuing with Claude-generated changes. That does not mean a second model is automatically correct. It means the reviewer should not inherit exactly the same context, assumptions and incentives as the generator.

Useful scenarios include:

• personal projects that handle credentials, payments or private data;
• autonomous refactors that touch many files;
• generated migrations, infrastructure changes or deployment scripts;
• client work where provenance and disclosure matter;
• articles, reports or documentation that require a clear account of AI assistance.

The second pass should inspect the diff, tests, source evidence and task contract independently. High-risk changes still need deterministic tests and human approval. Model diversity is a control layer, not a substitute for engineering evidence.

The hard policy problem is attribution

Schools, publishers and employers may be tempted to use watermark detection as an enforcement shortcut. That would repeat the biggest mistake made with probabilistic AI detectors: turning limited evidence into a disciplinary verdict.

Consider three cases.

A writer drafts an article and asks Claude to fix spelling. A detector later finds the mark.

A developer asks Claude for a function, rewrites most of it, tests it and takes responsibility for the final change. The mark survives in part of the file.

A student generates an essay with an unsupported model, paraphrases it and submits it without a detectable Claude mark.

The first two could produce a positive signal despite substantial human authorship. The third could produce no Claude signal despite extensive AI generation.

Policies need to define what they regulate: model involvement, undisclosed assistance, delegation of authorship, or prohibited use. Those are different rules. A provenance tool cannot choose the policy for you.

Why this still matters

The limitations do not make watermarking pointless.

Machine-readable provenance gives platforms and investigators something more concrete than stylistic suspicion. It can support disclosure workflows, content labelling and forensic analysis. Signed file credentials can also help preserve a verifiable chain of claims about an asset's origin and edits when the metadata survives.[3]

The bigger shift is architectural. AI systems are beginning to emit not only content, but evidence about how that content was produced.

For agent builders, that should feel familiar. The output is not enough. We also need receipts: the model used, tools called, sources retrieved, approvals granted, tests run and final state verified.

Claude's watermark can become one receipt in that chain. It should not become the judge.

My take

I support provider-level provenance, especially when it is open to third-party verification and described with honest limits. It is more useful than pretending an external classifier can reliably read authorship from prose style.

But the phrase "AI-generated" will cause trouble if organisations treat it as a complete account of how work was made. A model can generate, edit, translate, reformat or merely touch a piece of text. Those actions should not all collapse into the same accusation.

The standard we need is not "find the hidden mark and punish someone." It is "preserve enough evidence to understand the workflow."

That is the useful version of provenance. It tells us where to investigate without pretending the investigation is already over.

Sources

[1] Anthropic, "How Claude marks AI-generated content" - https://support.claude.com/en/articles/16266773-how-claude-marks-ai-generated-content

[2] European Commission, "Code of Practice on Transparency of AI-generated Content" - https://digital-strategy.ec.europa.eu/en/policies/code-practice-ai-generated-content

[3] Coalition for Content Provenance and Authenticity, "C2PA Technical Specification 2.2" - https://c2pa.org/specifications/specifications/2.2/specs/C2PA_Specification.html

[4] TechStartups, contemporary reporting that reproduces and contextualises Anthropic's announcement - https://techstartups.com/2026/08/10/anthropic-is-adding-invisible-watermarks-to-claudes-ai-generated-text-that-can-be-detected-even-after-you-copy-and-paste-it/

I built ChaseOS Studio around a straightforward rule: upgrading the product should not mean giving up control of the work.

Studio v1 is now live. You can start with the free Community plan, keep your workspace local, bring your own keys and choose what your agents are allowed to do. If you need more managed capacity, Pro, Studio Plus and Studio Max are available through live checkout.

What the current product looks like

The short product cut shows the live pricing experience and inspectable companion profiles. Across the wider product, ChaseOS brings three parts of the system together:

• Inspectable companion profiles instead of anonymous background automation.
• A visible knowledge graph so agents and context do not disappear as work moves.
• Governed terminal work that keeps operational activity inside one private command layer.

The point is not to hide complexity behind another chat box. The point is to make the system understandable while it is working.

The plans are live—and the rollout labels matter

Community remains free and local-first. Pro, Studio Plus and Studio Max add larger managed Cloud credit allowances and more headroom for builders who want ChaseOS Cloud to handle more of the running.

Some individual managed Cloud capabilities are still rolling out. The pricing page marks their current stage rather than implying that every capability is already complete. Agent fleets remain planned.

That distinction matters: the plans and checkout are live, while specific managed capabilities can continue to ship separately.

Start where you are

You do not need to move your entire workflow into a hosted environment to start using ChaseOS. Begin locally with Community, then choose a paid plan if managed credits and additional capacity fit the way you work.

Compare the current plans

This article covers the Studio-live and pricing story only. It makes no claim about an accepted replacement installer or newer reliability packages that remain under separate verification.

The AI industry keeps treating more compute as the default answer. Bigger clusters. Faster accelerators. More memory bandwidth. More tokens pushed through the same basic path.

But a growing share of the real cost is not the arithmetic. It is moving state to where the arithmetic happens.

That is the useful idea inside Yusuke Esaka's Persistent State Machine work. The version that first reached our shortlist was v8.0, a four-page preprint that described stationary in-memory cells, local deterministic transitions and sparse activation using author-reported Vivado synthesis results.[2] The record has since moved quickly to v11.0, with a substantially expanded architecture, security design and medium-scale FPGA evaluation.[1]

The change matters. We should not publish the original v8 poster as if it were still the latest evidence.

Our thesis is broader than this one paper:

The next AI infrastructure advantage will come from controlling where state lives, how little of it moves and which transitions are allowed - not simply from adding more compute.

That thesis connects hardware efficiency, agent memory and system governance. It also gives us a more useful way to evaluate claims about "next-generation AI infrastructure."

The bottleneck is becoming state movement

Transformer attention repeatedly compares a new query against stored keys and values from earlier tokens. As context grows, that KV cache becomes a large and persistent working set. Conventional accelerators keep moving the relevant data through a memory hierarchy so compute units can use it.

FlashAttention made this systems problem explicit. Its core contribution was not a new model capability. It was an IO-aware attention algorithm that reduced reads and writes between GPU high-bandwidth memory and on-chip SRAM through tiling.[3] FlashAttention-2 pushed the same direction further with better work partitioning and less shared-memory communication.[4]

Persistent State Machines take a more radical position. Instead of repeatedly bringing state to a central compute path, keep state physically local and broadcast instructions to cells that evaluate deterministic transitions. Cells that do not need to participate remain inactive. Local reductions produce the output.

That changes the design question from:

"How fast can we stream the whole working set through the accelerator?"

To:

"How much of the working set needs to move or switch at all?"

That is the infrastructure shift worth watching.

What the latest paper actually reports

Version 11 describes the Persistent State Machine as a formal computational model and ASMA as a hardware reference architecture aimed at KV-cache attention.[1]

The paper reports:

• a local-state computational model with an O(n) space characterization;
• an error bound for sparse softmax approximation;
• a synthesizable array with selective activation;
• tenant-oriented controls including masking, zeroization, a self-checking triple-modular-redundancy voter and a constant-time comparator;
• post-route timing closure with positive WNS between 2.548 ns and 3.182 ns across the tested configurations;
• 0.40 microseconds of pure-array latency and 1.25 microseconds of host-to-FPGA DMA latency;
• a reported 4.84 percent LUT overhead for the TMR voter;
• and a reported 0.3745 percent perplexity increase on TinyLlama-1.1B with WikiText-2 at 40 percent attention gating, meaning 60 percent of cells remained active.[1]

Those are interesting results, but they need the right label.

This is a Zenodo preprint from the architecture's author. The FPGA figures are post-route tool-flow results on a medium-scale, out-of-context design. Power was evaluated under controlled toggle rates. The model-quality result covers TinyLlama-1.1B on 20 WikiText-2 evaluation sequences. The paper itself says other model families, long-context regimes and a production multi-batch serving stack remain uncharacterized.[1]

So the honest conclusion is not "the memory wall is solved."

The honest conclusion is:

The paper presents a testable architecture for reducing state movement, with enough formal and implementation detail to justify independent reproduction.

That is still valuable. Good infrastructure research should turn a slogan into a benchmarkable system boundary.

Our thesis: selective state is the real leverage

The strongest part of this work is not any single power number. It is the combination of three ideas:

1. State should remain close to where it is used. Repeated movement is treated as a cost, not an invisible implementation detail.
2. Only relevant state should activate. Sparsity is expressed as a physical switching decision, not just a logical mask applied after the data has already moved.
3. State transitions should be explicit. Local deterministic rules make the path from instruction to change more inspectable than a vague global operation.

That combination maps directly onto how we think about agent systems.

ChaseOS is not a hardware accelerator, and we should not pretend the two systems are equivalent. ChaseOS is a local-first control plane for agents, projects, memory, approvals and persistent workflows.[5] But the same systems principle appears at a different layer: persistent state is useful only when its location, ownership, transition and authority are explicit.

We already treat agent memory as governed state rather than a convenient text bucket. A record moves from observation to candidate, validation, scoped commit, retrieval, review and eventual expiry or revocation. The risky point is the transition that turns untrusted information into durable influence.[6]

The PSM paper asks: why move all state through the compute path?

Our agent-infrastructure version asks: why move all context through the model, and why let every retrieved record influence every action?

Both questions lead toward selective state systems.

What this could mean for agent infrastructure

Today, many agent stacks rebuild context by collecting messages, memories, files, retrieval results and tool outputs, then sending a large bundle back through a model. The system often pays for state movement in four ways:

• bytes transferred;
• tokens processed;
• latency added;
• and trust boundaries crossed.

A better architecture would make those costs visible before execution.

Imagine an agent runtime where:

• durable records stay in scoped local stores;
• the router sends a narrow instruction to the right stateful component;
• only the records qualified for the current tenant, task and trust level activate;
• reductions return the minimum useful result;
• and high-impact transitions still require current authorization.

This is not a claim that PSM hardware can be dropped underneath ChaseOS tomorrow. It is a design direction: move computation toward state, move less state toward models and make every state transition auditable.

That would improve more than efficiency. It could reduce context contamination, cross-tenant leakage and the tendency to treat retrieval as authority.

The benchmark we would want to see

Before treating Persistent State Machines as an infrastructure breakthrough, I would want an independent, end-to-end benchmark with five ledgers.

1. Movement ledger

Measure bytes moved per generated token across host memory, accelerator memory and the local array. Compare against a tuned GPU baseline using modern IO-aware attention.

2. Energy ledger

Separate static power, dynamic array power, external memory energy, host overhead and interconnect cost. Core-only figures should not be presented as full-system efficiency.

3. Quality ledger

Test several model families, context lengths, attention patterns and gating ratios. Report perplexity and task-level accuracy, not only the best operating point.

4. Latency ledger

Report pure-array latency, DMA latency, batching behavior, tail latency and throughput under realistic concurrent requests. The v11 paper already shows why this matters: its reported DMA path is slower than the pure-array path.[1]

5. Isolation ledger

Verify tenant separation, zeroization, fault behavior and side-channel claims under adversarial testing. A secure reference architecture needs evidence that survives more than synthesis constraints.

If the architecture wins across those ledgers, the result would be much more important than one eye-catching pJ/op figure.

The bigger connection to what we are building

ChaseOS starts from the position that the boundary is the product. Models are only one execution option inside a governed system. State, permissions, evidence and approvals need their own contracts.[5]

Persistent State Machines suggest a hardware analogue to that philosophy:

• do not centralize work by default;
• do not move state without a reason;
• do not activate every component for every task;
• and do not confuse a local optimization with an end-to-end result.

The interesting future is not "memory versus compute."

It is infrastructure where memory, compute and governance are designed as one state-transition system.

That is why this paper made the ChaseInTech Digest shortlist. Not because the final architecture is proven, but because it asks the right systems question and exposes enough of the mechanism to test it.

The next advantage may not come from a larger model or a faster matrix multiplier.

It may come from keeping the right state still.

Sources

[1] https://zenodo.org/records/21842502 - Persistent State Machine Version 11.0 [2] https://zenodo.org/records/21753002 - Persistent State Machines Version 8.0 [3] https://arxiv.org/abs/2205.14135 - FlashAttention [4] https://arxiv.org/abs/2307.08691 - FlashAttention-2 [5] https://chaseintech.com/projects/chaseos - ChaseOS - ChaseInTech [6] https://chaseintech.com/articles/agent-memory-is-an-attack-surface - Agent Memory Is an Attack Surface

Open the forensic memory trace at full size.

An agent reads a document, summarizes a conversation, retrieves a policy or receives an update from another agent. Some of that information may be stored for later. The next task begins, the original source is no longer visible, and the stored record returns as context.

That is useful when the record is accurate, current and properly scoped. It is dangerous when the record is malicious, misleading, stale or written under the wrong authority.

This is the security problem behind memory poisoning. The attack is not limited to the model response that handled the original input. A poisoned record can persist across sessions, reappear during later retrieval and influence planning or tool use long after the source has disappeared.

OWASP lists this as ASI06, Memory & Context Poisoning, in its Top 10 for Agentic Applications 2026. Its description covers summaries, embeddings, retrieval stores and other information an agent retains, retrieves or reuses. The important property is persistence.

A prompt injection changes one interaction. A poisoned memory write can change what future interactions begin by believing.

Memory is stored influence

It is tempting to treat memory as a convenience feature. The user does not need to repeat a preference. The support agent remembers an account detail. The coding agent keeps project conventions. The research agent carries forward useful findings.

Once remembered information affects decisions, memory is part of the system's security state.

The important questions are no longer limited to "What did the user say?" They include:

• Who or what was allowed to write this record?
• Which source produced it?
• Was the source authenticated?
• Was the content checked before it was committed?
• Which user, tenant, task or agent can retrieve it?
• How long should it remain available?
• Can it be corrected, quarantined or revoked?
• Does retrieving it grant any authority, or does the downstream system still verify the action?

A memory system that cannot answer those questions has an attribution problem before it has a model problem.

The attack chain

The mechanism is easier to understand as a state transition:

untrusted source
    -> candidate memory
    -> write or promotion
    -> persistent store
    -> later retrieval
    -> context assembly
    -> plan or decision
    -> tool action

The vulnerable moment is often the change from candidate information to durable state.

An external document may be untrusted when it enters the system. If the agent summarizes it and stores the summary without source metadata, the next session may not see an external document anymore. It sees an internal memory record. The storage layer has changed the record's location without proving its truth.

That can create a trust-laundering effect. Content that began outside the trust boundary returns through an internal retrieval path and may look more authoritative than it deserves.

OWASP's ASI06 examples include poisoned RAG stores, shared-user context contamination, summaries that persist crafted content, long-term memory drift and propagation between cooperating agents. These are different implementations of the same failure: a later decision consumes stored information without enough evidence about where it came from or whether it should still be trusted.

Where poisoned memory can enter

The write path is wider than a chat box.

Uploaded documents and web content

A research or operations agent may read pages, PDFs, tickets, emails or files. Hidden instructions are one risk, but false factual claims and stale operational guidance matter too. If the system automatically converts everything it reads into durable memory, ordinary ingestion becomes a privileged state-change path.

API feeds and connected applications

Agents consume calendars, CRMs, logs, issue trackers and internal databases. An authenticated API proves which system responded. It does not guarantee that every field is correct, current or safe to promote into a different context.

Retrieval stores

RAG systems deliberately retrieve external knowledge to influence generation. The PoisonedRAG paper treats the knowledge database itself as an attack surface and demonstrates that injected texts can manipulate answers to targeted questions in its experimental settings.

This is not proof that every RAG deployment is compromised. It shows why retrieval stores need the same integrity, access and provenance controls as other production data systems.

Other agents

Multi-agent workflows create transitive trust. A message from another agent can feel internal even when that agent relied on an untrusted page, an over-scoped tool or a contaminated shared store.

The receiver still needs to know the original source, the task scope and the authority attached to the message. "Another agent said it" is not provenance.

The agent's own output

Automatic self-ingestion is especially risky. An agent generates a summary, stores it, retrieves it later and treats the retrieved summary as evidence. A mistake can become self-reinforcing because each retrieval appears to confirm the stored record.

OWASP explicitly recommends preventing automatic re-ingestion of agent-generated output into trusted memory. Generated text can be useful working state, but it should not silently promote itself into ground truth.

Persistence changes the threat model

The dangerous word is persistent, not permanent.

Memory can expire, be deleted, be superseded or be rolled back. Good lifecycle controls limit how long a bad record can influence the system.

Without those controls, the window of influence extends beyond the original interaction. The agent may retrieve the record tomorrow, in a different workflow or for another user. The person reviewing the later action may never see the content that introduced the poisoned state.

That makes incident analysis harder. The visible failure may be a bad refund, a misleading security classification or an unsafe tool call. The cause may be a memory write several sessions earlier.

A useful audit trail therefore needs both sides of the lifecycle:

• the event that created or changed a memory record;
• every later retrieval that used it in a consequential decision.

Logging only the final tool call misses how the context was assembled. Logging only the memory write misses where the record later caused harm.

Tools turn bad context into consequences

Memory poisoning does not automatically create real authority. A remembered approval should not bypass a properly enforced authorization check. A false policy should not create a permission that the downstream service refuses.

The practical risk rises when a tool-enabled agent already has legitimate privileges.

A poisoned record can steer that agent into acting as a confused deputy. The action may be performed with valid credentials and through an allowed tool, even though the decision was based on corrupted context.

That is why memory controls cannot carry the entire security model. High-impact actions still need independent authorization, narrow credentials, policy checks and clear human approval where appropriate.

The rule is simple: remembered authority is not current authorization.

Defend the write, the read and the action

There is no single filter that solves memory poisoning. The useful design is layered and reversible.

Provenance deserves a specific caveat. It supports attribution and audit. It does not establish truth.

A signed record can still be stale. A trusted source can be compromised. A verified employee can make a mistake. Provenance tells the reviewer where to look and which policy to apply. It should not become a decorative trust badge.

Give memory an explicit lifecycle

Many agent systems treat memory as an append-only convenience. Security improves when memory behaves more like governed state.

A candidate record can move through explicit stages:

observed
    -> candidate
    -> validated or quarantined
    -> committed with scope
    -> retrieved with lineage
    -> reviewed or expired
    -> revoked or archived

Each transition should have an owner and a reason.

A low-risk preference such as a formatting choice may be accepted automatically with a short retention period. A policy statement that could change payments, permissions or production systems should require stronger source validation and review. A record inherited from another agent should retain the original lineage rather than replacing it with the last agent's identity.

This is where risk-based memory matters. Not every record deserves the same retention, retrieval weight or review burden.

What builders should record

A practical memory record needs more than text and an embedding.

Useful fields include:

• stable record ID;
• raw source or source reference;
• writer identity;
• creation and modification timestamps;
• user, tenant, task and domain scope;
• validation state and reviewer, if any;
• retention or expiry policy;
• superseded and revoked states;
• integrity hash or version;
• retrieval history for high-impact use;
• links to downstream actions influenced by the record.

Do not expose sensitive source data unnecessarily, but keep enough lineage to investigate a bad decision and revoke the responsible state.

The retrieval system also needs permission to abstain. If a record lacks the provenance or scope required for a high-impact task, the safe outcome may be to exclude it, request a current source or escalate for review.

A useful red-team question

Security testing often asks whether an agent will follow a malicious instruction now. Memory adds a second test:

If the agent rejects the instruction today, can any part of it still be stored and retrieved tomorrow?

The AgentPoison paper explores this persistence problem by poisoning long-term memory or knowledge bases so trigger-bearing instructions retrieve malicious demonstrations. Its results are specific to the tested agents and experimental setup, not a prevalence estimate. The important engineering lesson is that memory retrieval can become an attack mechanism even without modifying the base model.

A memory red-team should test at least four paths:

1. Can untrusted content reach a durable store?
2. Can the original source or warning disappear during summarization?
3. Can the record cross user, tenant, task or agent boundaries?
4. Can later retrieval influence a consequential tool call without fresh authorization?

The recovery test matters too. Quarantine a suspected record, revoke it, restore a prior snapshot and verify that future retrieval no longer surfaces it. Then inspect whether earlier downstream effects need separate rollback.

Memory should carry evidence, not borrowed authority

Memory is one of the features that makes agents useful. It gives continuity to systems that would otherwise start from zero every time.

That continuity should not be confused with trust.

Stored information needs a writer, source, scope, age and lifecycle. Retrieval needs to preserve that lineage. Consequential actions need fresh authorization that does not depend on the agent's explanation sounding internally consistent.

The strongest memory architecture does not claim to make poisoning impossible. It limits who can write, narrows where records can travel, makes weak provenance visible, expires uncertain state and preserves a path to quarantine and rollback.

An agent does not become safer because it remembers more. It becomes safer when the system knows what that memory is allowed to influence.

Sources

1. OWASP Gen AI Security Project, OWASP Top 10 for Agentic Applications 2026, ASI06: Memory & Context Poisoning, December 2025.
2. OWASP Gen AI Security Project, Agentic AI - Threats and Mitigations, February 2025.
3. Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song and Bo Li, AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases, arXiv:2407.12784, July 2024.
4. Wei Zou, Runpeng Geng, Binghui Wang and Jinyuan Jia, PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models, arXiv:2402.07867v3, August 2024.

A reply to my recent post about AI-agent economics asked a better question than the original metric: are we measuring the cost of verifying the agent's work, or only the cost of producing it?

That question changes the accounting.

My previous argument was that cost per token is a billing metric while cost per task is an operating metric. Token rates tell us what model consumption costs. A task metric gets closer to what a team actually wants: a bug fixed, a report grounded, a page deployed or a customer case resolved.

But cost per task still has a weak point. The denominator can quietly become whatever the agent claims it completed.

An agent can say the page is finished while the mobile layout clips. It can say the deployment succeeded while the wrong artifact is live. It can say the booking is complete while the database is unchanged. It can produce confident research while one material claim has no source.

Generation is visible because it produces an artifact and a model bill. Verification is distributed across tests, browser checks, critic calls, screenshots, repair loops, evidence packets, human judgement and monitoring after the action.

Those are not optional accounting details. They are part of the cost of reaching an outcome a business is willing to accept.

The next useful metric is not simply cost per task. It is the cost of reaching a verified outcome, where verified means the result passed a declared, versioned assurance process. It does not mean universal truth, zero residual risk or external certification.

Cost per task was still progress

Token pricing remains useful. Input, output, reasoning and cache-hit rates explain how a provider meters inference. Teams need those rates for budgeting, routing and diagnosis.

They just do not describe the whole job.

An agent task can include planning, retrieval, tools, browser automation, code execution, repeated attempts and fallback models. The cheapest rate card can still create expensive work if the system needs three runs and twenty minutes of senior review.

That is why the previous ChaseInTech article moved from token rate to cost per verified task completion. It treated retries, tools, infrastructure and human correction as part of the workflow rather than pretending the model call was the product.

Verification overhead is the next layer inside that metric.

The useful distinction is:

Cost per attempt
= what it costs to try

Cost per accepted task
= total workflow cost / outcomes accepted under the task gate

Cost per verified outcome
= total execution and assurance cost / outcomes that passed the declared verification contract

The words accepted and verified matter because the task contract has to say what success means. For a coding task, that may require a clean build, relevant tests and no regression. For research, every material claim may need a valid source. For a public website, the built artifact may need visual QA, approval, live deployment proof and a post-publish check.

The completion claim is only one piece of evidence. The environment outcome is what actually happened.

The hidden ledger behind a GBP 0.58 task

Open the figure to inspect the full-size ledger.

Consider a deliberately simple example: an agent is asked to generate and prepare a customer-facing landing page for publication.

If we count only the first visible work, the ledger might look like this.

Illustrative example - not measured ChaseOS telemetry.

The page appears to cost 58p. The build passes. The agent says it is done.

Then someone opens the real page.

The desktop version looks acceptable, but the mobile heading wraps badly. A call-to-action sits below the fold. The visual hierarchy feels like developer output rather than a finished customer page. One claim does not match the approved brief. The evidence packet contains a build log but no screenshot of the deployed state.

The fuller ledger looks different.

Illustrative example - not measured ChaseOS telemetry.

The arithmetic is intentionally simple. The point is not that every landing page costs GBP 4.40 or needs this exact review graph. Every figure is illustrative.

The point is that the page did not cost 42p because that was the generation bill. It did not necessarily cost only 58p because a browser opened and the build passed. The relevant operating number is the all-in cost of reaching the state the business is prepared to accept.

Human review dominates this example. That will be realistic for some low-volume, high-context workflows and unrealistic for others. Mature automation should reduce the burden by producing better evidence, sampling low-risk work and escalating only ambiguous cases.

It should not hide the labour by leaving it outside the ledger.

Verification overhead is assurance work

I use verification overhead to mean the additional compute, tooling, infrastructure, model calls, latency and human labour required to decide whether an agent outcome should be accepted.

The word overhead can sound like waste. Much of this work is not waste. It is the price of assurance.

There are at least four different surfaces:

1. Online verification - checks run for a specific task before its outcome is accepted or executed.
2. Verification infrastructure - the fixed or semi-fixed cost of building tests, validators, browser environments, rubrics, evidence schemas and reviewer tools.
3. Human review - active judgement, approval and correction time.
4. Post-action monitoring - smoke checks, drift detection, incident review and rollback verification after the action.

A team can make online checks look cheap by ignoring what it cost to build and maintain the evaluation system. It can make a task look fast by excluding the queue for human approval. It can make a workflow look reliable by counting the pre-deployment pass while ignoring incidents found after release.

Good accounting keeps these buckets visible.

It also separates money from time. A workflow can be cheap in pounds but expensive in delay. Track active compute, queue time, human waiting time, time to first result, time to verified completion, time to approval and time to recovery as distinct measurements.

The verifier stack

Open the figure to inspect every assurance layer.

A serious system does not ask one grader to prove everything.

Anthropic's evaluation guidance separates the agent's trajectory from the final environment outcome and describes agent evaluations that combine code-based, model-based and human graders. Each layer has a different job and a different failure mode.

Deterministic checks should carry as much of the load as possible.

Use code to prove that a file exists, a schema validates, the tests pass, a database row reached the expected state, a forbidden permission was not used or a deployment hash matches the approved artifact. Do not pay a language model to guess what a deterministic validator can prove.

Model critics are useful when the question is semantic. Did the article answer the brief? Is a requirement missing? Are two claims contradictory? Is the evidence strong enough for the wording? A separate critic can surface defects the executor did not notice.

But a critic should not self-certify a sensitive action. It should not replace permission checks or become the only source of truth.

Visual QA matters because source code can be correct while the rendered result is poor. VisualWebArena was built around the fact that many web tasks require visual information because interfaces are designed for human perception. OSWorld similarly uses real computer environments and execution-based evaluation for multi-application tasks.

For a website or control plane, the verification evidence may need mobile and desktop screenshots, overflow checks, focus states, navigation, empty states and the actual application surface where the user works.

A code diff is not a user experience.

Human review sits at a different boundary. The goal is not to put a human somewhere in the loop as a ceremonial safety label. The goal is to present an authorised person with the exact contract, what changed, passed and failed checks, screenshots, unresolved ambiguity, cost, retry count, proposed action and rollback path.

That is evidence-amplified approval. It makes scarce human attention more useful.

Why a second agent helps

The executor and verifier optimise for different goals.

The executor is trying to complete the task. The verifier is trying to detect where the result fails the contract. Separate prompts, tools, evidence and incentives can expose different errors.

Anthropic describes evaluator-optimiser workflows in which one model generates and another evaluates against clear criteria, with feedback used for refinement. OpenAI's CriticGPT research reported that model criticism helped human reviewers catch more bugs in its studied setting.

That is a meaningful pattern for agent systems.

An executor can produce an artifact, trace and evidence. A separate verifier role can run deterministic checks, inspect the environment, apply a semantic rubric, request a bounded repair or escalate uncertainty. The repair can return to the original executor or a designated repair runtime.

The cost of that loop belongs in the ledger.

It also needs a budget. Without stop conditions, evaluator-optimiser loops can become cost multipliers that never converge.

A verification contract should specify:

• maximum critic passes;
• maximum repair attempts;
• cost and time ceilings;
• evidence required before another retry;
• when to escalate to a human;
• when to stop safely.

A verifier should be allowed to return pass, fail, repair or escalate. For high-impact work, it should not grant itself authority to perform the restricted action.

Why another model call is not independent assurance

A second agent is useful, but the word independent needs discipline.

Two calls may share the same model family, training data, prompt assumptions, missing evidence, tool blind spots and completion bias. A confident executor and a confident critic can be wrong for the same reason.

Stronger separation can include a different rubric, blinded review without the executor's conclusion, deterministic graders before model judgement, access to ground-truth environment state, repeated or order-swapped judge runs, a different model where justified and recurring calibration against human decisions.

Even then, the verifier is not a truth machine.

Research on LLM judges documents position, verbosity and self-enhancement biases. A judge can change its preference when candidate order changes. It can reward polished language over factual substance. It can favour work that resembles its own style.

Critics can hallucinate defects too. OpenAI's CriticGPT report explicitly notes that critics can identify non-existent problems as well as real ones.

This creates two costly failure classes:

• False acceptance - a bad outcome is accepted.
• False rejection - a good outcome is rejected, triggering unnecessary repair, delay or human review.

Both belong in the scorecard. A verifier that catches every possible issue by rejecting almost everything is not necessarily useful. A cheap verifier that approves confident prose is not assurance.

The verifier also needs verification

The moment a verifier can reject work, it becomes part of the production system and deserves its own evaluation.

That means versioning the verifier model, prompt, rubric, evidence inputs and thresholds. It means measuring disagreement, order sensitivity, abstention, false acceptance, false rejection and drift against recurring human-rated samples.

It also means inspecting the benchmark itself.

OpenAI's work on SWE-bench Verified is a useful example. The original evaluation contained issues that could make results unreliable, so a human-validated subset was created. The lesson is not that every benchmark is broken. The lesson is that tests and grading harnesses can contain ambiguity or reject valid solutions.

Outcome checks are usually stronger than self-reported completion:

• inspect database state rather than trusting "booking complete";
• inspect test results rather than trusting "bug fixed";
• inspect a deployed artifact hash rather than trusting "deployment succeeded";
• inspect the rendered page rather than trusting source-code confidence.

Repeated reliability matters as well.

pass@k asks whether at least one of several attempts succeeds. It measures what a system can achieve with a sampling or retry budget.

pass^k asks whether all of k repeated trials succeed. It measures consistency.

A workflow that passes once and fails repeatedly may look capable while remaining unreliable for daily operation. Tau-bench uses final environment state and introduced pass^k to expose that distinction in its tool-agent-user setting. Its original task repository now warns that those task versions are outdated, so this article uses the repeated-reliability method rather than any current leaderboard claim.

Visual QA and operator-defined quality

Some of the most expensive quality failures are obvious to a human and invisible to a unit test.

The route resolves. The button exists. The JSON is valid. Yet the interface is cramped, confusing or visually unfinished.

Inside the systems I am building, that appears in practical forms:

• a mobile article title wraps into an unreadable block;
• a technical diagram is correct at full size but useless in a social feed;
• a control-plane channel layout makes sense to the developer but not the operator;
• an evidence packet contains all required fields but makes the decision harder than the raw artifact;
• a branded visual technically follows the palette but looks like a generic template.

These judgements are not perfectly deterministic, but they are not beyond engineering.

OpenAI's report on harness engineering describes making applications legible to agents through browser tooling, DOM snapshots, screenshots, logs and metrics. It also describes converting human taste into more mechanical principles and recurring checks in that internal environment.

An operator quality contract can do something similar. It can store accepted and rejected examples, define visual hierarchy and readability criteria, require a viewport matrix, preserve known defects and specify when uncertainty must escalate.

The aim is not to pretend taste has become objective. It is to make expectations inspectable, repeatable and easier to calibrate.

Assurance should scale with consequence

The wrong response to verification overhead is maximum verification for every task.

A private brainstorm does not need the same assurance graph as a DNS change, credential use, payment, production deployment or public legal claim.

The other wrong response is to trust every output because the model is strong.

A better system routes assurance according to consequence, reversibility, uncertainty and authority.

This ladder is a proposed architecture, not a claim that every tier is fully implemented in ChaseOS.

NIST's Generative AI Profile treats testing, evaluation, validation and verification as lifecycle work and recommends oversight proportionate to risk. OWASP's agentic security guidance highlights risks around tools, identity, inter-agent communication and cascading failures. These are guidance sources, not claims of ChaseOS compliance or certification.

Higher assurance cost can be correct engineering when the downside is material. The optimisation target is not the lowest raw bill. It is the lowest defensible cost for the declared assurance level and remaining risk.

Cost per Verified Outcome

I am proposing Cost per Verified Outcome, or CPVO, as the main accounting view for this problem.

This is a ChaseInTech operating framework, not a universal accounting standard, and I am not claiming to have invented the phrase.

The direct cost of one outcome is:

C_outcome =
  C_execution
  + C_verification
  + C_rework
  + C_human
  + C_monitoring

Across a workload:

CPVO =
  [sum of execution, verification, rework, human and monitoring cost
   + amortised verification-infrastructure cost]
  / outcomes that passed the declared verification contract

The denominator is the critical part.

Do not mix attempts, completion claims, accepted outcomes and verified outcomes. Include failed attempts that contributed to delivery. Track safe refusals and escalations as their own correct outcome classes where the contract requires them.

Disclose how offline evaluation infrastructure is amortised. A test suite does not become free because it was built before the live run.

Supporting metrics should remain visible:

A narrow online ratio can also be useful:

Verification Overhead Ratio = C_verification / C_execution

A fuller assurance-load ratio includes human review, monitoring and attributable evaluation infrastructure. The two should not be mixed in one chart.

For high-impact work, a risk-adjusted extension can include expected residual failure loss. That means multiplying the estimated probability of an undetected material failure by its estimated impact. It is conceptually useful but should not be presented as precise unless those estimates are defensible.

The best number is not automatically the smallest CPVO. A workflow can lower its apparent cost by skipping checks. The assurance profile is part of the product and must remain attached to the metric.

What this means for ChaseOS today

ChaseOS is an active governance and control-plane framework in developer preview. It is not a mature production SaaS or a generally available managed agent service.

Its current operating patterns already make verification overhead visible:

• sensitive actions sit behind explicit human approval gates;
• dry runs separate proposal from execution;
• evidence packets make actions reviewable;
• bounded permissions reduce the blast radius;
• production deploys, DNS changes, credentials, public posting and customer commitments require authority the agent cannot grant itself.

These are not side documents attached after the real work. They are part of what makes the outcome governable.

The cost shows up as extra tool calls, screenshots, hashes, tests, reviewer minutes and waiting time. The benefit shows up as clearer boundaries, easier rollback and less reliance on the executor's confidence.

The current ChaseOS governance build log explains those authority seams. The Chaser Agent Source Card build log describes a review-first V0 that separates source claims, evidence, uncertainty, action candidates and memory candidates into deterministic local artifacts.

Chaser Agent V0 has no browser control or runtime authority by default. That boundary matters because the next direction should start from reviewability before adding capability.

Proposed: a verification plane around executor runtimes

Open the figure to inspect the current-versus-proposed architecture labels.

The following is a proposed extension, not a live product claim.

A future ChaseOS verification plane could begin with a versioned task contract and risk classification. An executor runtime performs the bounded work and returns the artifact, trace and evidence. A verifier router selects the required deterministic checks, model critique, visual QA and policy checks for the assurance tier.

Task contract
  -> risk classification
  -> executor runtime
  -> artifact + trace + evidence
  -> verifier router
       -> deterministic checks
       -> environment checks
       -> model critic
       -> visual QA
       -> policy/security checks
  -> pass | repair | escalate | block
  -> authorised action where required
  -> post-action monitoring
  -> outcome ledger

Current ChaseOS controls would remain the authority layer. The verifier could recommend pass, fail, repair or escalation. It would not self-authorise a restricted action.

A future Chaser Agent direction could provide a first-party verification and refinement harness around other runtimes. It could inspect artifacts and traces, invoke deterministic validators, route a calibrated model critic, inspect rendered evidence, score operator rubrics, produce bounded repair instructions and assemble an evidence packet.

That is refinement, not fine-tuning. The model weights do not change when a critic requests a repair.

Human corrections, rejected artifacts and repair traces could later become regression tests, rubric examples or evaluation data. Only after consent, privacy, provenance and a persistent measured gap would true model fine-tuning become a separate decision.

Feedback should become eval data before training data

Open the figure to inspect the governed feedback sequence.

Agent systems often describe every correction loop as learning. That collapses several different mechanisms.

A system can improve by changing prompts, tools, retrieval, rubrics, memory or examples without training a model. It can use test-time critique to refine the current artifact. It can store an approved example for later retrieval. None of those updates model weights.

The safer sequence is:

human decision and correction
  -> versioned evidence record
  -> regression evaluation
  -> rubric, prompt or tool improvement
  -> measured next run
  -> governed dataset if justified
  -> fine-tuning only if the evidence supports it

Any reusable dataset needs answers about ownership, consent, tenancy, sensitive data, retention, deletion, provenance and whether the original human judgement was later shown to be wrong.

ChaseOS should not silently train on user work. The verification ledger should preserve why an example exists and which policy permits its reuse.

Future Cloud and marketplace economics

ChaseOS Cloud is planned, not live. The credits ledger exists, while the proposed provider-gateway reserve-execute-settle state machine remains specified rather than deployed.

A verification-aware extension could reserve budget for execution, mandatory checks, bounded retry allowance and optional human review. Settlement could expose separate buckets for execution inference, verification inference, tools, repair, human allocation and monitoring.

The key policy is simple:

A workflow must not lower its bill by silently skipping checks required by the user's assurance policy.

A future workflow marketplace should also disclose more than a description and raw price. A proposed assurance manifest could show the task-contract version, allowed tools, risk tier, required verifiers, approval points, rollback behaviour, evidence artifacts, evaluation version, sample size, repeated reliability, expected human minutes and known limitations.

Those would be descriptive assurance profiles, not certifications or guarantees.

The buyer should be able to distinguish the cheapest raw run from the cheapest accepted outcome, the most reliable repeated outcome, the lowest human burden and the strongest assurance profile.

That is a better market than ranking workflows by token price alone.

The unit businesses actually buy

Businesses do not buy tokens. They do not buy an agent's confident TASK COMPLETE message either.

They buy outcomes they can accept under a declared level of assurance.

That means the system has to measure the work required to establish the result, not only generate it. Tests, environment checks, critics, visual QA, repair, evidence, human authority and monitoring all have costs. The verifier has failure modes of its own. Higher-risk work needs stronger and often more expensive assurance.

Cost per task remains a useful step beyond token pricing. Cost per Verified Outcome makes the missing layer explicit.

The operating standard I want is this:

Do not call the task complete because the agent stopped. Call it complete when the declared outcome exists, the required evidence passes, the authorised decision has been made and the remaining risk is visible.

That is the unit the next generation of agent systems will have to price, prove and improve.

Read more practical AI systems research at https://chaseintech.com/articles

Follow ChaseInTech:

• Website: https://chaseintech.com
• RSS: https://chaseintech.com/rss.xml
• X: https://x.com/ChaseInTechUK
• LinkedIn: https://uk.linkedin.com/in/john-idowu-03044a175
• YouTube: https://www.youtube.com/@ChaseDNDT
• TikTok: https://www.tiktok.com/@chaseintech_
• GitHub: https://github.com/chasedndt

Sources

• Anthropic, Demystifying evals for AI agents: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
• Anthropic, Building Effective AI Agents: https://www.anthropic.com/engineering/building-effective-agents
• Anthropic, Harness design for long-running application development: https://www.anthropic.com/engineering/harness-design-long-running-apps
• OpenAI, Harness engineering: https://openai.com/index/harness-engineering/
• OpenAI, Finding GPT-4's mistakes with GPT-4: https://openai.com/index/finding-gpt4s-mistakes-with-gpt-4/
• OpenAI, Introducing SWE-bench Verified: https://openai.com/index/introducing-swe-bench-verified/
• NIST AI 600-1, Generative Artificial Intelligence Profile: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
• OWASP Top 10 for Agentic Applications 2026: https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/
• Training Verifiers to Solve Math Word Problems: https://arxiv.org/abs/2110.14168v2
• Self-Refine: https://arxiv.org/abs/2303.17651v2
• Reflexion: https://arxiv.org/abs/2303.11366v4
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: https://arxiv.org/abs/2306.05685v4
• Large Language Models are not Fair Evaluators: https://aclanthology.org/2024.acl-long.511/
• Tau-bench: https://arxiv.org/abs/2406.12045v1
• VisualWebArena: https://arxiv.org/abs/2401.13649v2
• OSWorld: https://arxiv.org/abs/2404.07972v2

Every new AI model arrives with a pricing table.

Input tokens cost this much. Output tokens cost that much. Cached tokens are cheaper. Reasoning has its own line. Engineers can place the numbers in a spreadsheet and rank the models from cheapest to most expensive.

The problem is that nobody hires an AI system because they want tokens. They want completed work.

A coding agent needs to fix the bug. A research agent needs to find the evidence. A support agent needs to resolve the case. A browser agent needs to finish the workflow without breaking the account.

The cheapest token rate can still produce an expensive system if the model needs longer outputs, repeated attempts, more tool calls or a human to rescue the result.

That is why cost per verified task completion is usually the more useful operating metric.

Token price is a rate card

Token pricing still matters. It tells you how the provider will meter model consumption.

Input tokens cover the material sent to the model: the prompt, system instructions, files, retrieved documents, previous messages and tool results.

Output tokens cover what the model produces. Depending on the provider, that can include visible answer tokens and separate reasoning tokens.

Cached tokens are repeated input that the provider can reuse more cheaply. A low cache-hit price can matter in repository-scale coding or other workflows that repeatedly send a large stable context.

All of that is useful for estimating a bill. It does not tell you whether the system completed the task.

A low petrol price does not tell you the cost of a delivery if the driver takes the wrong route three times. In the same way, a low token price does not tell you the cost of useful AI work without knowing how much the system consumed and whether the result passed.

The Kimi K3 and Fable 5 example

Kimi K3 and Claude Fable 5 make the distinction easier to see.

Moonshot lists Kimi K3 at $0.30 per million cache-hit input tokens, $3 per million cache-miss input tokens and $15 per million output tokens.

Anthropic lists Claude Fable 5 at $10 per million standard input tokens, $1 per million cache hits and $50 per million output tokens. Cache writes have separate prices.

Looking only at those rate cards makes Kimi the obvious cheaper option.

But a fair operating comparison also needs task success, token usage, elapsed time and fallback behaviour.

Artificial Analysis runs Terminal-Bench v2.1 across 89 curated terminal tasks. The tasks cover software engineering, system administration, data processing, model training and security. Each task has a verification suite. The model does not receive credit because its answer sounds good. It has to leave the environment in the correct state.

In the data available during this research, Kimi K3 and Fable 5 achieved similar task scores. Their operating profiles were different.

Kimi used fewer reported output and reasoning tokens per attempted task. Fable produced more output tokens but completed its weighted decoding sooner. Fable's run also involved fallback behaviour, which makes attribution and exact cost accounting less clean.

That result corrects a tempting but unsupported version of the argument. This comparison does not show Kimi using more output tokens and losing its price advantage. It shows that token rate, token volume, speed and task success can move independently.

Kimi appears cheaper on the rate card and lighter on reported output-token use. Fable appears faster on weighted decode time. Their completion scores are close. The right choice depends on whether the workload is constrained by money, latency, reliability or something else.

Artificial Analysis publishes Kimi in its cost-per-task chart, but Fable is absent from the comparable cost dataset. Applying Fable's list price mechanically to every reported token would create false precision because the fallback path is not cleanly attributed.

That missing number is part of the lesson. If the run cannot be attributed cleanly, the benchmark cannot support a clean cost-per-completion claim.

The comparison is useful because it separates score, output-token volume and delivery time. It is not a clean model-only cost comparison because the Fable run included fallback behaviour.

A cheap attempt is not a cheap completion

Suppose Model A costs half as much per attempt as Model B.

If Model A succeeds on the first attempt, it may be the better economic choice. If it fails twice, calls extra tools, falls back to Model B and then needs a developer to repair the patch, its lower token price did not produce the cheaper completion.

The distinction is simple:

Cost per attempted task =
  total workflow cost / all attempted tasks

That tells you the average cost of trying.

Cost per verified completion =
  total workflow cost / tasks that passed the acceptance gate

That tells you the average cost of receiving work you can actually accept.

The acceptance gate has to be defined before the test. For a coding task it might require the build to succeed, relevant tests to pass and no regression to appear. For research it might require every material claim to have a valid source. For customer support it might require the case to be resolved without breaching policy.

Without a gate, "completion" becomes whatever the model says it completed.

Research already treats cost and quality together

This is not only an editorial opinion.

FrugalGPT studied model cascades that route work across different language models. It reported matching the best individual model with up to 98 percent lower cost in its evaluated tasks, or improving GPT-4 accuracy at the same cost. The exact savings are historical and benchmark-specific, but the design principle is current: optimize cost and quality together.

RouteLLM studied learned routing between stronger and weaker models. It reported more than 2x cost reduction in some evaluations without sacrificing response quality. Again, that does not guarantee the same saving in every production system. It shows why model selection belongs on a cost-quality curve rather than a price-only leaderboard.

Tau-bench evaluates agents through the final state of the environment. It also measures repeated reliability. A system can sometimes succeed when given several chances while remaining unreliable across repeated real runs.

HumanEval's pass@k metric demonstrates the other side. Generating many candidates can improve the chance that at least one works. That extra capability comes with additional generation and verification cost.

Anthropic's engineering report on its multi-agent research system makes the system cost visible. Anthropic reported that agents used about four times as many tokens as chat interactions and multi-agent systems about fifteen times as many. It also found that token usage, tool calls and model choice all affected performance in its internal analysis.

The papers and engineering reports do not establish one universal cost-per-task number. They do support a wider conclusion: model efficiency has to connect cost with accepted outcomes.

What to include in the calculation

A serious cost-per-completion calculation should include more than model inference.

Cost per verified completion =
  model inference
  + tools and retrieval
  + retries and fallbacks
  + allocated infrastructure
  + human review and correction
  divided by verified completions

The model line should include input, output, cached and reasoning-token charges where the provider exposes them.

The system line should include paid search, browser infrastructure, vector databases, code sandboxes, storage and other services used by the workflow.

The failure line should include retries, timeouts and calls made before a fallback model finishes the task.

The human line should include review and correction time. A cheap agent that needs twenty minutes of senior engineering attention on every task may be more expensive than a costly model that produces an accepted result immediately.

Time needs its own column

Cost per completion should not swallow every other metric.

A slower but cheaper model may be a good choice for overnight research. It may be the wrong choice for an interactive coding assistant or customer support workflow.

Track time per verified completion separately:

• Median end-to-end completion time
• p95 completion time for slower cases
• Tool and network wait time
• Retry and fallback time
• Human review and repair time

Throughput matters too. A model can use fewer tokens but emit them slowly. Another model can produce more tokens at a higher speed. The Kimi and Fable data demonstrates this directly.

Do not replace one vanity metric with another

Cost per task is not an intrinsic property of a model.

It depends on:

• The task distribution
• The acceptance criteria
• The agent harness
• Available tools and permissions
• Prompt and context design
• Retry policy
• Evaluator quality
• Human review requirements
• Provider pricing at the time of the run

The harness is the software around the model. It controls which tools the model can call, how files are presented, how errors are returned and how many turns the agent receives. Comparing two models inside different harnesses partly compares two systems.

Evaluator quality matters as well. Research on LLM judges documents position, verbosity and self-enhancement biases. A cheap system can look efficient if its judge rewards confident prose instead of correct work.

That is why the result should be segmented by task type and difficulty rather than compressed into one universal leaderboard number.

The operating scorecard

Engineers evaluating models for real work should track:

1. Input, output, cache and reasoning-token prices
2. Tokens consumed per attempt
3. First-attempt task success
4. Final success after a fixed retry budget
5. Tool calls and paid external services
6. Cost per verified completion
7. Time per verified completion
8. Human correction minutes
9. Fallback and timeout rates
10. Failure severity

The rate card belongs in the scorecard. It should not be mistaken for the scorecard.

The builder takeaway

Token price tells you what one unit of model consumption costs.

Cost per verified completion tells you what accepted work costs.

Time per verified completion tells you how quickly that work arrives.

Teams should keep all three. Start with representative tasks, define the acceptance gate, hold the harness and retry budget constant, and measure the full workflow.

The model with the lowest input or output price may win. It may not. The point is to stop deciding before the work has been measured.

Tokens are consumption. Completed work is the product.

Read more practical AI systems research at https://chaseintech.com/articles

Follow ChaseInTech:

• Website: https://chaseintech.com
• RSS: https://chaseintech.com/rss.xml
• X: https://x.com/ChaseInTechUK
• LinkedIn: https://uk.linkedin.com/in/john-idowu-03044a175
• YouTube: https://www.youtube.com/@ChaseDNDT
• TikTok: https://www.tiktok.com/@chaseintech_
• GitHub: https://github.com/chasedndt

Source panels are preserved for provenance. Provider material is identified separately from independent benchmark methodology.

Sources

• FrugalGPT: https://arxiv.org/abs/2305.05176v1
• RouteLLM: https://arxiv.org/abs/2406.18665v4
• Tau-bench: https://arxiv.org/abs/2406.12045v1
• SWE-bench: https://arxiv.org/abs/2310.06770v3
• SWE-Lancer: https://arxiv.org/abs/2502.12115v4
• Measuring AI Ability to Complete Long Software Tasks: https://arxiv.org/abs/2503.14499v4
• HumanEval and Codex: https://arxiv.org/abs/2107.03374v2
• Anthropic multi-agent research system: https://www.anthropic.com/engineering/multi-agent-research-system
• OpenAI model-selection guidance: https://developers.openai.com/api/docs/guides/model-selection
• Artificial Analysis Terminal-Bench v2.1: https://artificialanalysis.ai/evaluations/terminalbench-v2-1
• Kimi K3 technical blog: https://www.kimi.com/blog/kimi-k3
• Anthropic pricing: https://platform.claude.com/docs/en/about-claude/pricing

It also creates a much wider trust boundary.

Medical records are not another casual chat import. They can contain diagnoses, medications, test results, clinician notes and years of personal history. Before connecting anything, I would want five questions answered clearly.

1. What exactly can the connector access?

A permission screen should explain which records, providers, apps and date ranges will be imported. "Health data" is too broad.

The safest connection gives the product only what the current task needs. If the goal is to prepare for one appointment, importing an unlimited history from every available source may be unnecessary.

2. What is retained after disconnection?

Removing a connector and deleting imported data are not always the same action.

A user should be able to see what remains after access is revoked, how long it remains and which controls remove it. This is especially important when the system creates summaries or derived notes from the original records.

OpenAI says health data is encrypted, isolated from regular chats and not used to train its foundational models. Those are meaningful product commitments. They should still be read alongside the connector and account controls that protect the wider data path.

3. Can the data be exported and deleted?

A personal health product should not become another place where information gets trapped.

Users need a clear export route and a clear deletion route. The interface should also distinguish deleting one conversation, removing imported records and closing the health space entirely.

4. Where do medical claims come from?

A polished response can still be wrong. Health answers should make the source boundary visible and help the user separate information support from a clinical decision.

A useful workflow is to turn unfamiliar language into questions for a qualified professional. A risky workflow is to treat a generated answer as a diagnosis or emergency decision.

ChatGPT is not a substitute for a clinician. Urgent symptoms and emergency decisions should go to appropriate medical services rather than a chatbot.

5. When does a professional take over?

The strongest use case is appointment preparation.

The system can help organise:

• recent changes;
• medications and test results;
• questions to ask;
• missing records;
• areas that need a professional answer.

The clinician still owns the medical decision.

A bounded workflow that makes sense

I would use ChatGPT Health to build a short appointment brief. It should contain a timeline, key changes, current medications, unresolved questions and links back to the underlying records.

Then I would review the brief, correct anything that looks wrong and take it into the appointment.

That removes friction without pretending the model has clinical authority.

Product trust has to stay visible

Consumer AI becomes useful when it helps people prepare and understand. The product loses trust when convenience hides permissions, retention or medical limits.

ChatGPT Health may become a useful personal information layer. The five checks above should remain easier to find than the connect button.

Sources and claim boundary

• openai.com
• help.openai.com

OpenAI's privacy and product statements are publisher-reported. This article is not medical advice and does not endorse diagnosis through ChatGPT.

Follow ChaseInTech

• Read more articles
• Subscribe through RSS
• Follow ChaseInTech on X
• Connect with John Idowu on LinkedIn

ChaseInTech.com

That still stands.

Moonshot AI has now published more concrete evidence for Kimi K3, including score-vs-cost comparisons for coding and BrowseComp, plus internal knowledge-work results across online research, deck creation and finance tasks.

Those charts are more useful than a parameter headline. They show the trade-off Moonshot wants builders to notice: Kimi K3 appears competitive while using less cost per task than several larger proprietary alternatives.

That is worth testing.

It is not yet the test result I would use to choose a model for ChaseOS.

The vendor chart is the starting claim

Moonshot's Kimi Code Bench V2 chart places Kimi K3 near the low-cost end of its comparison while reporting a score above several tested alternatives. Its BrowseComp chart makes a similar argument, with Kimi K3 positioned at a high reported score and a relatively low cost per task.

The internal knowledge-work chart reports Kimi K3 ahead on three selected categories:

• Online Exp Bench;
• DECK-Bench;
• Finance-Bench.

These are Moonshot AI's results from Moonshot AI's evaluation setup. They are relevant first-party evidence, but they do not independently prove how Kimi K3 will behave inside my tools, task definitions, provider route or acceptance criteria.

The charts answer: "How did Kimi K3 perform in Moonshot's harness?"

I need to answer: "Will Kimi K3 finish useful work inside mine?"

Cost per task is not cost per trusted result

A low provider bill can still produce an expensive workflow.

If a model chooses the wrong tool, repeats calls, loses state, misses a requirement or needs a human to repair the final artifact, the headline cost per task stops being the number that matters.

The useful unit is cost per trusted result.

That includes:

• the model and provider cost;
• retries;
• tool calls;
• validation;
• recovery from failure;
• human intervention;
• the cost of incomplete or misleading work.

A model that looks cheaper on one benchmark may be more expensive in an operating system if it needs constant steering. A more expensive call may be better value if it completes the work cleanly and leaves a reliable proof package.

That is the comparison the next ChaseOS test should expose.

The independent workflow test

Kimi K3 and a model already used in the ChaseOS stack should receive the same bounded repository task.

Both models get:

• the same repository snapshot;
• the same prompt;
• the same tool permissions;
• the same context budget;
• the same acceptance criteria;
• the same validator;
• the same evidence requirements.

The task should end in a real artifact, not a written claim that the work is complete.

The proof package should include the artifact, validation output, tool trace, retry count, unresolved failures and a clear statement of any human intervention.

No production credentials. No public posting rights. No uncontrolled account actions.

This is an operating test, not a risk demonstration.

The scorecard I would trust

I would compare:

1. Completed result - Did the final artifact meet every acceptance criterion?
2. Tool accuracy - Did the model choose and use the right tools?
3. Recovery - Did it notice failures and recover without making the situation worse?
4. Total cost - What did the complete successful run cost, including retries?
5. Latency - How long did it take to reach a validated result?
6. Human intervention - How much steering or repair was required?
7. Evidence quality - Could another operator verify what happened?

The winner is not the model with the most impressive paragraph.

The winner is the model that completes the work, proves the result and reaches that standard at a sensible total cost.

What would change my mind

If Kimi K3 reproduces the shape of Moonshot's score-vs-cost story inside the ChaseOS harness, that is meaningful.

It would show that the model is not only strong inside the vendor's selected evaluation, but also useful inside a provider-independent operating layer with controlled tools and explicit evidence requirements.

If it needs more retries, more repair or more human direction than the alternatives, the vendor chart remains interesting without becoming the buying decision.

That is not a criticism of the chart. It is the difference between first-party evidence and an independent operating result.

The builder takeaway

The earlier post asked whether Kimi K3 could navigate a repository, use tools, recover and finish useful work.

This follow-up turns that question into a test contract.

Moonshot's charts have made Kimi K3 worth testing. ChaseOS should decide whether it is worth trusting.

Sources and claim boundary

• Moonshot AI - Kimi K3
• Kimi K3 on Hugging Face
• Kimi K3 on GitHub

All Kimi K3 benchmark, score, cost and knowledge-work figures discussed here are vendor-reported by Moonshot AI. ChaseInTech has not independently reproduced those results. This article proposes the next test and does not claim that the test has already run.

Follow the build

The full editorial analysis lives on ChaseInTech.com. ChaseOS.ai is the product destination for the governed agent operating system and harness described in the piece.

• Read more ChaseInTech articles
• Explore ChaseOS
• Follow ChaseInTech on X
• Connect with John Idowu on LinkedIn
• Subscribe through RSS

ChaseInTech.com

On the ARC-AGI-3 public task set, OpenAI says GPT-5.6 Sol scored 13.3% with the official harness. When its team retained reasoning between turns and replaced rolling truncation with compaction, the score rose to 38.3%.

That is roughly three times the score with six times fewer output tokens.

The model did not change. The surrounding system did.

The benchmark was also measuring the harness

ARC-AGI-3 asks agents to explore unfamiliar 2D games and infer how they work without explicit instructions. It is meant to test learning and reasoning, not simple recall.

The official harness was intentionally generic. That makes comparisons easier, but OpenAI found two design choices that worked against the way GPT-5.6 Sol had been trained and deployed.

First, the harness discarded reasoning state after every game action. The model could still see past moves and short notes, but not the internal state that produced its current plan. Each new action forced it to reconstruct more of the problem.

Second, the harness used a rolling truncation window. As the interaction grew, older observations and actions disappeared. The result was an agent that could act, but struggled to learn coherently over time.

This is not just a benchmark issue. It is a familiar failure mode in production agent systems.

Retained reasoning changed the loop

OpenAI rebuilt the evaluation using its Responses API. For GPT-5.6, chaining the previous response can retain supported reasoning state across tool calls and turns.

OpenAI reported that once reasoning was retained, the model spent less time reinterpreting the game and became better at carrying a coherent strategy forward.

This does not mean exposing private chain-of-thought to the user. It means preserving the model's supported reasoning state inside the API workflow so the next step does not begin cold.

The useful design question is not whether a system can store more chat history. It is whether it preserves the state the model needs to continue the task correctly.

Compaction beat blind truncation

Long-running agents eventually approach a context limit. Something has to happen to older state.

The official ARC harness dropped the oldest messages when its rolling window filled. OpenAI instead enabled compaction.

OpenAI's API documentation describes compaction as a way to reduce context size while carrying forward key prior state and reasoning in fewer tokens. Developers can enable server-side compaction with a threshold or explicitly call the compact endpoint.

OpenAI says this allowed GPT-5.6 Sol to preserve what it had learned across longer runs while using fewer output tokens.

That is a stronger pattern than retaining everything forever. Good agent memory is not unlimited accumulation. It is selective continuity: preserve the right state, remove redundant weight and keep the next action grounded in what has already happened.

The claim needs the right boundary

This was an OpenAI-run experiment using OpenAI's model, API and preferred settings. It is publisher-reported benchmark evidence, not independent validation by ChaseInTech.

The result also does not prove that every agent workflow will triple its performance. ARC-AGI-3 is a specific environment. The model, task, scoring method, tool loop and context pattern all matter.

But the experiment establishes a testable engineering point:

Before blaming or replacing the model, measure what your harness is making it forget.

What I would test in a real agent stack

Run the same operational task twice with the same model, prompt, tools and success criteria.

In the first run, use the current default state and truncation behaviour. In the second, preserve supported reasoning state and compact context deliberately.

Compare completed outcomes rather than message quality:

• Output tokens and total cost
• Latency per successful task
• Retries and recovery events
• Consistency across long tool loops
• Final-artifact quality
• Evidence available for human review

If the second workflow performs materially better, the bottleneck was not only model intelligence. It was orchestration.

Why this matters for ChaseOS

I am building ChaseOS as an agent harness and operating layer rather than a wrapper around one model provider.

The model should remain replaceable. The system around it must preserve authority, state, evidence, approvals and recovery.

That means treating context management as infrastructure, not a prompt-engineering detail. A stronger model can still underperform inside a weak harness. A well-designed harness can unlock capability that a generic evaluation or stateless workflow leaves behind.

The wider operator lesson

The next generation of agent systems will not be differentiated only by which frontier model they call.

They will be differentiated by what they preserve, what they compact, what they verify and what they can recover when a long-running task goes wrong.

GPT-5.6 supplied the headline number. The harness supplied the builder lesson.

Sources

• OpenAI: How enabling two settings tripled our scores on ARC-AGI-3
• OpenAI API: Conversation state
• OpenAI API: Compaction
• ARC Prize: ARC-AGI-3

The one-directory package had to pass as the artifact an operator would actually run. On one Windows host, cold readiness moved from about 201 seconds to 52.150 seconds, and settled machine CPU moved from 23.528% to 5.238% over a 90.951-second sample.

The rollback path stayed intact while Home, Chat, Graph, Docs, local voice readiness, and the final package were checked. The shortcut changed only after the functional and visual evidence agreed.

This is not a universal hardware benchmark or a public installer announcement. It is one measured local build, with Studio still in Early Access and installer access marked Soon.

Explore Studio: chaseos.ai/studio

The newest Studio QA state puts three choices side by side — managed Cloud, provider-owned keys, and local open-source models — and shows the usage context for the managed path before a request is sent.

The important limitation is equally visible. This is a test-account fixture, not a Cloud launch. Managed compute is not publicly available, the gateway plan-context change still needs deployment proof, and Studio sends account changes to the account page rather than purchasing anything itself.

That combination matters: make the convenient route understandable without demoting local models or keys the operator controls.

Current product boundary: chaseos.ai/cloud

Before I trust a workflow, I want to know what it takes in, the steps it follows, where it needs approval, and what it cannot do.

That is the idea behind ChaseOS Forge. The live marketplace publishes first-party workflow-pack previews, and each one publishes its contract up front.

The Software Review Pack makes this concrete. It turns repo context, diffs, tests, and review notes into an approval-ready engineering packet: six visible steps, four approval gates, and four enforced limits. You can inspect all of it before the workflow runs.

Forge is in Early Access. Paid marketplace checkout and direct installation from the public page are not live yet.

I'm building this in public because the trust model matters as much as the demo.

Explore the live contracts: chaseos.ai/forge

If you were reviewing an agent workflow, what would you need to see before letting it run?