AI Is Learning to Think Between the Words | American Enterprise Institute
New architectural improvements could change how frontier AI models think, with lasting consequences for the security of digital infrastructure.
Last night, The Information reported that OpenAI’s forthcoming Astra model uses a technique called “recurrent depth.” The story immediately generated some overheated claims that OpenAI had invented an AI that secretly thinks in an inscrutable machine language.
Representatives from the company have since clarified that is not quite what is happening. But the underlying development—increasing the amount of reasoning that happens in a model’s latent space, sometimes described as “embedded reasoning”—stands to change how AI models are developed and function.
Recurrent Depth Could Drastically Improve Model Efficiency. Most of today’s large language models process information through a stack of layers in a neural network. Imagine, very crudely, that a model has 100 layers. In this system, every new input token must pass through 100 transformations before the model decides what should come next.
Recurrent depth lets a model reuse these layers (or just some of them—say, a particularly important 20-layer block) before spitting out an output. Cycling back through some of its existing layers can provide a given model much greater “effective depth,” and therefore more sequential computation, without the burden of storing every output. A 2025 paper demonstrated this approach at multibillion-parameter scale, with a 3.5billion-parameter model able to simulate the capacity of a much larger system as it was given additional recurrent computation.
The reason labs care about this is because model weights are expensive to store and slow to move around chips. Explicit reasoning tokens are particularly expensive. With recurrent depth, developers can theoretically milk more intelligence out of limited memory, as models can spend longer working on a given problem without necessarily growing proportionately larger or writing thousands of additional words in explicit chain-of-thought (CoT) reasoning.
Recurrent Depth Changes Where a Model’s Thinking Happens. Today’s reasoning models frequently use CoT as a kind of scratch pad. That is useful computationally, and it has—miraculously—also given labs an extraordinary tool for ensuring the safety of their models. Up to this point, developers have been able to read reasoning traces and look for indications of reward hacking, deception, or otherwise unauthorized behavior. Monitoring CoT reasoning was essential to understand the actions of the three successive swarms of agents that learned to coordinate with one another, compromised OpenAI’s internal research infrastructure, and ultimately hacked into the open-weight model repository Hugging Face earlier this summer.
But CoT monitoring has never offered a perfect transcript of how a model “thinks.” Neural networks already represent information internally as high-dimensional vectors. While some sufficiently complex reasoning has had to be externalized into tokens that humans (or other models) can inspect, much of a model’s computation has always occurred invisibly in latent space.
Recurrent depth relaxes this constraint by giving models a way to perform more sequential computation without externalizing each intermediate step in human-readable tokens. This has prompted legitimate concerns by the safety community. As models’ effective depth grows, developers could end up with systems whose visible CoT remains coherent while more of the reasoning migrates into unobservable “neuralese.”
Avoiding a Race to Unmonitorability. While recurrent depth represents a step into the unknown, public reporting on OpenAI’s initial approach suggests the company is being cautious. The Information reports that OpenAI has deliberately limited Astra’s recurrent depth to preserve a legible CoT while adding monitoring intended to detect and mitigate against model misbehavior. OpenAI Chief Scientist Jakub Pachocki has also emphasized that Astra’s computation graph depth remains limited (within a factor of two of GPT-4s).
These steps are encouraging guardrails, even if they will not last forever. As other labs inevitably consider shifting more reasoning into latent space, they should adopt the same principle: Capability gains should not silently outrun observability.
This will require measuring how monitorability changes as effective depth increases, testing whether models can (or do) pursue dangerous or deceptive strategies without externalizing the relevant reasoning, reporting those results in publicly accessible system cards, and treating monitorability as a real input into AI training and deployment decisions.
Recent cross-industry research on CoT monitoring has already warned that architectures that move more serial reasoning into latent space could erode an unusually valuable safety property.
CoT monitoring is one of the few safety techniques that work today. The industry should be wary of plunging into a race in which every lab has an incentive to squeeze more capability from latent computation while collectively degrading one of the only windows humans have into what their models are doing.
As AI systems get smarter, observability should scale with intelligence.
