AI Infrastructure Will Be Won by Moving Less State - Not Adding More Compute

Persistent State Machine research points at a harder systems question: what if the next efficiency gain comes from keeping model state local, activating only what matters and governing every transition?

The AI industry keeps treating more compute as the default answer. Bigger clusters. Faster accelerators. More memory bandwidth. More tokens pushed through the same basic path.

But a growing share of the real cost is not the arithmetic. It is moving state to where the arithmetic happens.

That is the useful idea inside Yusuke Esaka’s Persistent State Machine work. The version that first reached our shortlist was v8.0, a four-page preprint that described stationary in-memory cells, local deterministic transitions and sparse activation using author-reported Vivado synthesis results.[2] The record has since moved quickly to v11.0, with a substantially expanded architecture, security design and medium-scale FPGA evaluation.[1]

The change matters. We should not publish the original v8 poster as if it were still the latest evidence.

Our thesis is broader than this one paper:

The next AI infrastructure advantage will come from controlling where state lives, how little of it moves and which transitions are allowed - not simply from adding more compute.

That thesis connects hardware efficiency, agent memory and system governance. It also gives us a more useful way to evaluate claims about “next-generation AI infrastructure.”

The bottleneck is becoming state movement

Transformer attention repeatedly compares a new query against stored keys and values from earlier tokens. As context grows, that KV cache becomes a large and persistent working set. Conventional accelerators keep moving the relevant data through a memory hierarchy so compute units can use it.

FlashAttention made this systems problem explicit. Its core contribution was not a new model capability. It was an IO-aware attention algorithm that reduced reads and writes between GPU high-bandwidth memory and on-chip SRAM through tiling.[3] FlashAttention-2 pushed the same direction further with better work partitioning and less shared-memory communication.[4]

Persistent State Machines take a more radical position. Instead of repeatedly bringing state to a central compute path, keep state physically local and broadcast instructions to cells that evaluate deterministic transitions. Cells that do not need to participate remain inactive. Local reductions produce the output.

That changes the design question from:

“How fast can we stream the whole working set through the accelerator?”

To:

“How much of the working set needs to move or switch at all?”

That is the infrastructure shift worth watching.

What the latest paper actually reports

Version 11 describes the Persistent State Machine as a formal computational model and ASMA as a hardware reference architecture aimed at KV-cache attention.[1]

The paper reports:

a local-state computational model with an O(n) space characterization;
an error bound for sparse softmax approximation;
a synthesizable array with selective activation;
tenant-oriented controls including masking, zeroization, a self-checking triple-modular-redundancy voter and a constant-time comparator;
post-route timing closure with positive WNS between 2.548 ns and 3.182 ns across the tested configurations;
0.40 microseconds of pure-array latency and 1.25 microseconds of host-to-FPGA DMA latency;
a reported 4.84 percent LUT overhead for the TMR voter;
and a reported 0.3745 percent perplexity increase on TinyLlama-1.1B with WikiText-2 at 40 percent attention gating, meaning 60 percent of cells remained active.[1]

Those are interesting results, but they need the right label.

This is a Zenodo preprint from the architecture’s author. The FPGA figures are post-route tool-flow results on a medium-scale, out-of-context design. Power was evaluated under controlled toggle rates. The model-quality result covers TinyLlama-1.1B on 20 WikiText-2 evaluation sequences. The paper itself says other model families, long-context regimes and a production multi-batch serving stack remain uncharacterized.[1]

So the honest conclusion is not “the memory wall is solved.”

The honest conclusion is:

The paper presents a testable architecture for reducing state movement, with enough formal and implementation detail to justify independent reproduction.

That is still valuable. Good infrastructure research should turn a slogan into a benchmarkable system boundary.

Our thesis: selective state is the real leverage

The strongest part of this work is not any single power number. It is the combination of three ideas:

State should remain close to where it is used. Repeated movement is treated as a cost, not an invisible implementation detail.
Only relevant state should activate. Sparsity is expressed as a physical switching decision, not just a logical mask applied after the data has already moved.
State transitions should be explicit. Local deterministic rules make the path from instruction to change more inspectable than a vague global operation.

That combination maps directly onto how we think about agent systems.

ChaseOS is not a hardware accelerator, and we should not pretend the two systems are equivalent. ChaseOS is a local-first control plane for agents, projects, memory, approvals and persistent workflows.[5] But the same systems principle appears at a different layer: persistent state is useful only when its location, ownership, transition and authority are explicit.

We already treat agent memory as governed state rather than a convenient text bucket. A record moves from observation to candidate, validation, scoped commit, retrieval, review and eventual expiry or revocation. The risky point is the transition that turns untrusted information into durable influence.[6]

The PSM paper asks: why move all state through the compute path?

Our agent-infrastructure version asks: why move all context through the model, and why let every retrieved record influence every action?

Both questions lead toward selective state systems.

What this could mean for agent infrastructure

Today, many agent stacks rebuild context by collecting messages, memories, files, retrieval results and tool outputs, then sending a large bundle back through a model. The system often pays for state movement in four ways:

bytes transferred;
tokens processed;
latency added;
and trust boundaries crossed.

A better architecture would make those costs visible before execution.

Imagine an agent runtime where:

durable records stay in scoped local stores;
the router sends a narrow instruction to the right stateful component;
only the records qualified for the current tenant, task and trust level activate;
reductions return the minimum useful result;
and high-impact transitions still require current authorization.

This is not a claim that PSM hardware can be dropped underneath ChaseOS tomorrow. It is a design direction: move computation toward state, move less state toward models and make every state transition auditable.

That would improve more than efficiency. It could reduce context contamination, cross-tenant leakage and the tendency to treat retrieval as authority.

The benchmark we would want to see

Before treating Persistent State Machines as an infrastructure breakthrough, I would want an independent, end-to-end benchmark with five ledgers.

1. Movement ledger

Measure bytes moved per generated token across host memory, accelerator memory and the local array. Compare against a tuned GPU baseline using modern IO-aware attention.

2. Energy ledger

Separate static power, dynamic array power, external memory energy, host overhead and interconnect cost. Core-only figures should not be presented as full-system efficiency.

3. Quality ledger

Test several model families, context lengths, attention patterns and gating ratios. Report perplexity and task-level accuracy, not only the best operating point.

4. Latency ledger

Report pure-array latency, DMA latency, batching behavior, tail latency and throughput under realistic concurrent requests. The v11 paper already shows why this matters: its reported DMA path is slower than the pure-array path.[1]

5. Isolation ledger

Verify tenant separation, zeroization, fault behavior and side-channel claims under adversarial testing. A secure reference architecture needs evidence that survives more than synthesis constraints.

If the architecture wins across those ledgers, the result would be much more important than one eye-catching pJ/op figure.

The bigger connection to what we are building

ChaseOS starts from the position that the boundary is the product. Models are only one execution option inside a governed system. State, permissions, evidence and approvals need their own contracts.[5]

Persistent State Machines suggest a hardware analogue to that philosophy:

do not centralize work by default;
do not move state without a reason;
do not activate every component for every task;
and do not confuse a local optimization with an end-to-end result.

The interesting future is not “memory versus compute.”

It is infrastructure where memory, compute and governance are designed as one state-transition system.

That is why this paper made the ChaseInTech Digest shortlist. Not because the final architecture is proven, but because it asks the right systems question and exposes enough of the mechanism to test it.

The next advantage may not come from a larger model or a faster matrix multiplier.

It may come from keeping the right state still.

Sources

[1] https://zenodo.org/records/21842502 - Persistent State Machine Version 11.0 [2] https://zenodo.org/records/21753002 - Persistent State Machines Version 8.0 [3] https://arxiv.org/abs/2205.14135 - FlashAttention [4] https://arxiv.org/abs/2307.08691 - FlashAttention-2 [5] https://chaseintech.com/projects/chaseos - ChaseOS - ChaseInTech [6] https://chaseintech.com/articles/agent-memory-is-an-attack-surface - Agent Memory Is an Attack Surface