Depth, Not Width

Looped transformers, and why Astra is hard to watch

Two days before GPT-6 Astra shipped, The Information reported that it uses “recurrent depth,” or looped transformers — and tied that to the model's reduced chain-of-thought monitorability. Sebastian Raschka spent this week's issue on both halves of that claim, and the second half does not survive contact with the first.

Start with the mechanism, because it is simpler than the name suggests. A looped transformer passes intermediate representations through the same transformer blocks more than once, keeping the weights identical across passes. Nanbeige4.2-3B, an open-weight model from July, applies a stack of 22 blocks twice: unrolled, that is 44 block applications, but block application 23 reuses block 1's weights, 24 reuses block 2's, and so on. Effective depth doubles; the number of distinct weight sets does not.

What it buys you is narrower than it first appears. The memory needed to store weights drops. The compute does not: the forward pass still runs 44 block applications and gradients flow backward through all of them, so training cost is close to a genuine 44-block model. Neither does the KV cache — because the hidden states entering a block differ on the second pass, its keys and values differ too, and both passes need their own entries. Nanbeige's team did try sharing the cache; it halved memory and made the model worse, so the released version keeps them separate. They also found training from scratch beat upcycling a pretrained model, and stopped at two passes because a third bought little and destabilised optimisation.

The idea is old — Universal Transformers proposed it in 2018, repeating a single block rather than a stack, with adaptive halting so a token can exit after one pass or continue for four. ByteDance's Ouro-Thinking 2.6B pushes it further, applying a 48-block stack four times for 192 block applications, with a learned exit gate choosing which pass supplies the output. Mixture-of-Recursions replaces the halting function with a small learned router, mixture-of-experts style, deciding per token how many passes to spend — either by having each recursion step select which tokens continue, or by assigning each token a path up front. Because the router reads a token's hidden representation, the same word gets different depth depending on its context.

Does it work? The honest answer is “at scale, modestly, yes.” Mixture-of-Recursions loses to a vanilla transformer at the smallest model size and wins at larger ones, especially at smaller training budgets — a good reminder that the 135M-parameter version of an experiment can point the wrong way. The cleaner result is SMELT, released this month, which does the comparison everyone had been ducking: looped versus conventional at matched compute per token, matched non-embedding parameters and matched KV cache, scaling to 54B non-embedding parameters. Its fitted curves put the looped design at roughly 6.8–18% less training compute for the same validation loss. And Beyond Parameters separates what looping does and does not give you: with parameters fixed, it leaves memorisation capacity essentially unchanged but improves multi-step maths. Looping is a computing mechanism, not a storing one.

Which brings us to the monitorability claim, and Raschka's answer is a flat no. OpenAI has hidden most reasoning traces from users since o1, so nothing changed for end users; the interpretability concern is a developer-side one. Astra's system card does report reduced monitorability relative to Sol, mostly in the form of shorter, less informative traces — but shorter traces are what capability looks like. Within one model family, GPT-5.6 Luna burns about 80% more tokens than Sol for similar performance, and nobody claims Sol is therefore less interpretable. A stronger model backtracks less. Jakub Pachocki, OpenAI's chief scientist, weighed in directly to say that the computation-graph depth of current frontier models including Astra is “within a factor of two of GPT-4,” that he wants to “prevent a race into unmonitorability kicked off by confused reporting,” and — importantly — that monitorability is fragile and trending badly, “for reasons not contingent on architecture changes.”

One paper complicates the tidy version. The full-bandwidth transformer, from August, feeds the previous token's final hidden state back in through a learned gate, and does produce shorter reasoning traces on MATH500 at equal or better accuracy — but only in the base model. The effect vanishes after instruction tuning, and the study did not test whether conventionally scaling the model produces the same shortening, nor whether the shorter traces are less faithful.

DeepSeek v4.1-Flash breaks the decoder-only mould

DeepSeek released v4.1-Flash this week: 763B total parameters, described as P8B-D16B, built as a causal encoder–decoder with vision. Latent Space's verdict was that the version number undersells it — “this should have been DeepSeek v5” — and the reason is the shape rather than the size.

Practically every major frontier model of the past several years has been decoder-only. That convergence was not obviously correct, it was just what scaled predictably, and it has meant that architectural competition largely stopped being a thing anyone competed on. A competitive model with a genuinely different fundamental structure — an encoder–decoder split, made causal, with vision integrated rather than bolted on — is a fork in a road that had looked straight for a while. Community discussion around the release picked at the parameter arithmetic, working out what a 30B dense backbone would imply for the size of an accompanying engram component, and speculating about whether Qwen would follow with a “tiny KV” design that lowers context-memory requirements directly rather than arguing about KV-cache quantisation.

It is worth being straight about what is not yet known. There is no architecture paper with the depth that the DeepSeek V3 and R1 reports had, so the parameter breakdown and the training recipe are still partly inference from the release notes and from people poking at the weights. What can be said is that it is open, which is the whole point of the comparison in Nathan Lambert's open-models reading list, also out this week. That piece is useful as a reference spine, and unusually blunt about the mechanism nobody likes discussing: distillation, including reasoning-trace extraction from frontier models, as a real route by which Chinese labs close the gap — stated without treating it as either scandal or non-event.

Ten thousand agents and a Millennium Prize problem

On 8 September OpenAI announced a solution to the Navier–Stokes Millennium Prize Problem, produced by a group of agents running a next-generation model described as significantly more capable than GPT-6 Astra. It overshadowed a week that would otherwise have led with Cognition's $48B Series E, Mistral's $24B Series D, Meta's Muse agent and GPT Image 2.5. It is also, on the available evidence, close to unevaluable — and the gap between how big the claim is and how little was shown is the story.

Here is what was actually stated. Ethan Knight said the result came from “a collaboration of ~10,000 agents working together,” that OpenAI had spent the past year training models to collaborate via multi-agent RL, and that hard problems may yield to “huge amounts of unstructured parallel test-time compute” with models deciding how to organise themselves. That is a systems claim, and a substantive one: it describes a research architecture, not a mathematical argument.

Here is what was not stated. No theorem statement. No preprint, proof sketch, or formal verification artifact. No independent referee commentary. No specification of whether the result concerns the standard 3D incompressible global regularity problem on ℝ³ or a torus, or some variant. No account of the division of labour between humans and models — “collaboration of ~10,000 agents” does not tell you whether humans decomposed the search, curated lemmas, verified steps, or launched the infrastructure and went home. No mention of theorem-prover integration, proof-assistant stack, symbolic algebra, retrieval corpora, model size, compute budget, ablations against a single-agent baseline, or proof-check success rates.

The “88 hours” deserves its own note, because it ended up in nearly every headline including the one at the top of the source roundup. It appears in a satirical post, and AINews flags explicitly that it should not be treated as confirmed from the available evidence. The cost figure travelled the same way. Neither number was disclosed; both are now load-bearing in how the result is being discussed, which is a fairly pure demonstration of a figure acquiring authority through repetition alone.

None of which means nothing happened. AINews' own read — and it is a careful one — is that the achievement is likely real and the process is in dispute. But “solution” is doing enormous work. In mathematics it could mean a complete proof, a proof strategy, a candidate counterexample, a formalised derivation, or a research lead, and a claim that a finite-time singularity can occur would imply a negative answer to global regularity — which is precisely the kind of claim that requires extraordinary precision before anyone should move. If the work genuinely touched the problem, the interesting novelty is probably not “LLM writes a proof” but distributed theorem search with learned collaboration policies: task decomposition, inter-agent communication, memory persistence, search-tree management, proxy scoring, and aggregation of candidate proof paths at a scale nobody has run before.