This is the third and final part of the recap of NICE’s event “Looped Language Model: Rethinking Model Depth and Parameters,” covering the second half of the panel discussion. The panel was hosted by Wenyue Hua, with Chunyuan Deng and Benhao Huang as panelists.

The second half ranged more widely: Is looping along depth the same thing as looping along time (that is, CoT)? Do expressiveness proofs still mean anything? How should loops be combined with MoE? And finally, a set of questions about “how to use loops at inference”: extrapolation, adaptive computation (ACT) and parallel decoding.

Key takeaways

  • Looping along depth and looping along time look much alike from the standpoint of computational depth. Chunyuan thinks that in both cases “you have to take that many sequential steps in total; the only difference is how you take them.” One difference he sees lies in the data: explicit CoT can be synthesized, whereas it is unclear how to construct supervision for latent CoT.
  • The difference between loops and soft tokens lies in how the KV cache is used. Over the lifetime of a single token, looping twice and generating one extra soft token reach similar depths. Letting the previous token’s deep-layer state flow back into the next token’s shallow layers is exactly the starting point of the Full-Bandwidth Transformer.
  • RL brings a new opportunity, and an underlying concern. Looping itself does not get in the way of RL; the opportunity lies in assigning credit to the intermediate loops. The concern is that looping may itself already be doing something similar to RL, so the gains from RL may shrink.
  • Expressiveness proofs are not pointless. Inductive bias determines how quickly a model learns. Chunyuan’s conclusion is that “It’s all about the computational depth,” and that learnability matters more at present.
  • Loop × MoE: different loops should choose different experts. Several works find that looping twice is the sweet spot, though Benhao would like people to “loop at least three times.”
  • Extrapolation is hard for natural language, and ACT looks reasonable but is hard to make work. Both aim to change the number of loops at inference: extrapolation loops more times than in training, while ACT lets each token decide for itself how many times to loop. Both run into a mismatch between training and inference, and ACT also faces infrastructure difficulties. Benhao believes a more realistic direction for speedups is to keep the number of loops unchanged and instead, borrowing from Diffusion Forcing, stagger the different loops of different tokens and decode them in parallel.

Glossary

The table below explains several terms that recur in this article. Readers familiar with them can skip it.

Term Plain-language explanation
Soft token / Coconut Coconut feeds the model’s last-layer hidden state directly back in as the input for the next step, without decoding it into text; this “continuous thought” is a soft token
Teacher forcing In training, the ground-truth text is fed in directly as the preceding context (rather than tokens the model generated itself), so all positions can be computed in parallel. To preserve this parallelism, in the standard architecture each position can see, at a given layer, only the information of earlier positions at that same layer, not their deeper states. So a result that an earlier token reaches in a deep layer can be used by later tokens only once they reach an equally deep layer, which leaves few layers for further processing
State tracking Continuously keeping track of a state that changes with the input, for example determining where each item ends up after a sequence of swaps is carried out in order
TC⁰ / NC¹ Two classes in circuit complexity: TC⁰ corresponds to constant-depth, highly parallel computation; NC¹ allows depth to grow logarithmically with input length and is generally believed to be strictly more powerful (though this has not been proven). Some state-tracking problems are NC¹-complete
Extrapolation Looping more times at inference than in training, with the model still working or even doing better
ACT Adaptive Computation Time: letting each token decide for itself how many times to loop, according to its difficulty
Mixture-of-Depths A router decides whether each token skips a given layer, dynamically allocating compute along depth
HRM / TRM Two models that solve puzzles by applying a very small network recursively (about 27M and 7M parameters, respectively), commonly used on Sudoku, mazes and ARC-AGI
Diffusion Forcing A training method for diffusion models in which each token in a sequence has its own independent noise level, so the model can keep denoising older frames while starting to generate new ones
Speculative decoding Drafting upcoming tokens cheaply first and then verifying them in parallel with the full model, to reduce the number of sequential steps

Panelists

Chunyuan Deng: PhD student at Rice University, advised by Prof. Hanjie Chen. His research focuses on the design and optimization of LLM pretraining architectures, as well as the interpretability of language models. His representative work is LT2: Linear-Time Looped Transformers [1]; his talk at the event is covered in part one of this series.

Benhao Huang: Master’s student at CMU, advised by Prof. Zico Kolter. He works mainly on looped LMs and fixed points, with the aim of improving the reasoning ability and efficiency of foundation models. His representative work includes Equilibrium Reasoners [2] and the blog series Towards Looped Models Done Right [3].

Wenyue Hua (host): Member of Technical Staff at Microsoft AI (MAI). PhD from Rutgers University, advised by Prof. Yongfeng Zhang, followed by a postdoc at UCSB. Research focuses on LLM reasoning and LLM-based agents, with many papers at top NLP and machine learning venues, including ACL, EMNLP, ICLR, NeurIPS and TACL. Serves as an Area Chair for ACL.

1. Is looping along depth the same as looping along time?

The host, Wenyue Hua, wanted to hear the two panelists’ thoughts: Why has there been such a surge of papers on looping recently (though it could also be a coincidence)? Are looping along depth and looping along time essentially the same thing? How does this kind of implicit reasoning relate to explicit CoT? And does it affect credit assignment in RL?

Chunyuan: it depends on how you define “essentially”

Chunyuan said it depends on how we define “essentially.” One could first ask: are RNNs and Transformers essentially the same thing? A Transformer can also be trained step by step like an RNN, instead of in parallel with teacher forcing. Architecturally, what matters most is the operations: an architecture is, in the end, a collection of operators. If “essentially” refers to computational depth from the perspective of expressiveness, then he personally thinks the two are indeed fairly similar: in total, you have to take that many sequential steps, and the only difference is how you take them.

As for whether explicit CoT is needed, he sees one difference between explicit CoT and latent CoT in the training data: the former can be written by hand, or synthesized by people or models; for the latter, he cannot yet think of a workable way to synthesize the data or construct the supervision.

For today’s looped architectures, if you only do SFT or pretraining, there is no major conflict between looping and CoT: the model simply uses greater depth to learn CoT, rather than doing away with it. In RL, however, greater depth (or an architecture like the Full-Bandwidth Transformer) may indeed have an effect. RL requires sampling; in agentic settings, the model needs many intermediate variables for state tracking, and it has to rely on either CoT or depth for that. With greater depth, the CoT may become shorter, and RL itself also demands token efficiency. Likewise, with latent reasoning methods such as Coconut [4], the CoT may also get shorter. He thinks these two things really do lead to differences in RL.

Note: Coconut (Chain of Continuous Thought) feeds the model’s last-layer hidden state directly back in as the input embedding for the next step, without decoding it into text, so that reasoning takes place in a continuous latent space.

2. Loop vs. Coconut, seen through the lifetime of a token

Similar depth; the difference is in how the KV cache is used

Suppose you draw out the topology, follow the concrete lifetime of a single token and write out a loop and a Coconut-style soft token side by side: for example, a model that loops twice, versus an L-layer model that does not loop but generates one extra soft token. The two look very similar in depth. The difference lies in the details: how the KV cache is used.

When looping, it helps if the previous token’s final hidden state can be injected into the shallow layers of the next token; this is precisely the motivation for the Full-Bandwidth Transformer [5]. In an ordinary loop, because training uses teacher forcing, the deep-layer information has not yet been formed when the later token’s shallow layers are computed, so naturally it is unavailable. Soft tokens are similar: whether the previous token is a soft token or an ordinary explicit token changes which of its layers’ KVs the later tokens can see.

Note: The Full-Bandwidth Transformer takes the top-layer hidden state at the previous position, fuses it through a gating unit with the embedding of the currently sampled token, and uses the result as the bottom-layer input at the next position; the rest of the architecture, the KV cache and the training objective are the same as in a standard model. Because this dependency cannot be parallelized directly, training uses multiple forward passes: each additional pass shifts the previous pass’s hidden states one position to the right and runs the whole network again, and this feedback is added only late in training (for example, only a single pass is run for roughly the first 75% of training). In experiments at the 1B scale with up to 400B tokens, the paper reports that it can match or approach a standard model trained on about 1.5× as many tokens; by the paper’s estimate, the added compute per token at decoding time is under 1%, and the model runs about 2.3% slower when measured in vLLM.

What makes RNNs special

Benhao raised a point he found very interesting. Picking up on Chunyuan’s mention of RNNs and Transformers a moment earlier, he noted that within frameworks that measure model capability, such as complexity classes and circuit complexity, RNNs have something special about them, for example state tracking. Soft tokens can “inject” this ability back into Transformer-style architectures; looping alone, without any other mechanism, may not be able to do so.

Note: Fixed-depth Transformers and many state-space models fall into the complexity class TC⁰, so they are generally believed to be unable to solve NC¹-complete state-tracking problems such as composing permutations [6]; each additional step of CoT or soft tokens, by contrast, amounts to letting the model perform one more step of sequential computation.

3. What looping means for RL

The extra cost is mainly in prefill

Benhao mentioned that designs like the Full-Bandwidth Transformer, which send deep-layer states back to shallow layers, may carry some extra cost when used for RL: getting the deep-layer hidden states requires additional iterations or forward passes. In settings with a lot of prefill (the stage in which the model reads in the entire input at once and builds the KV cache), such as agentic tasks, this could become a problem, and RL training would also cost somewhat more.

A new opportunity: assigning credit to intermediate loops

As for credit assignment, Benhao’s view is that, looking at looping alone, nothing stands in the way of RL: you do RL on a looped Transformer the same way you do on a Transformer, with no difference. But if you want to assign credit to the intermediate steps of the loop (for example, when looping four times, giving some credit to the 1st, 2nd and 3rd loops as well), then it is worth thinking about whether there are interesting ways to do so. For instance, during RL, could some of the intermediate loops take on tasks such as tool calling?

An underlying concern: looping may already be doing something like RL

Beyond that, he also has a worry he is not entirely sure about: when RL is done on top of a looped model, will the gains be as large as before? A looped model may itself already be doing something similar to RL.

The host followed up: what does it mean that “looping itself is doing RL”? Benhao explained:

RL is a kind of distribution sharpening: it makes the distribution sharper, a bit like mode seeking. Looping, on the other hand, can be seen as a kind of power iteration: it keeps the more stable modes, like keeping the directions corresponding to the largest and second-largest eigenvalues.

Note: Power iteration is a classic method for finding a matrix’s dominant eigenvector: repeatedly multiply a vector by the matrix and normalize it, and the components along other directions gradually decay, leaving the direction corresponding to the largest eigenvalue. Benhao’s analogy is that both make the model contract toward a few dominant modes, so the two may overlap.

4. Do expressiveness proofs still mean anything?

On the looping side, there is work such as the Universal Transformer (UT) [8], which seemingly can approximate any function; other papers say that with unbounded CoT or infinite numerical precision, Transformers come close to being Turing complete. Many architectures can be proven to be fairly general once certain conditions are relaxed. So do expressiveness proofs have little to do with how architectures are designed?

Note: The Universal Transformer paper shows that, given enough memory and a number of recurrent steps that grows with input length, it is computationally universal. Pérez et al. proved that Transformers are Turing complete under the assumption that weights and activations can take arbitrary precision, but not with fixed precision [9]. Merrill and Sabharwal proved that with a linear number of CoT steps, Transformers can recognize all regular languages, and with a polynomial number of steps they recognize exactly the class P [10], that is, the decision problems an ordinary algorithm can solve in polynomial time.

Benhao: inductive bias determines how fast a model learns

Benhao said some architectures have their own inductive biases and learn certain tasks faster. For example, a looped model can learn Sudoku very quickly, while a Transformer of the same depth has a somewhat harder time, because it lacks a suitable inductive bias. Beyond that, though, if depth is not limited, and instead of training you could know, like a god, exactly which weights to use and set them directly, then the solution certainly exists within the Transformer’s function class.

Chunyuan: It’s all about the computational depth

Chunyuan went on to say that there is actually a lot of discussion about expressiveness in the linear attention community. For example, the Transformer belongs to TC⁰, while GDN (Gated DeltaNet, a kind of linear attention) is an NC¹ structure. He has read a dozen or so papers discussing such questions, including the Universal Transformer, “but UT needs ACT to have that property.” After reading so many, his final conclusion is:

It’s all about the computational depth.

You can do it in depth, or you can use a lot of CoT; essentially, there really is not much of a difference.

Note: Fixed-depth, log-precision Transformers can be simulated by TC⁰ circuit families [11]. On linear RNNs, research has proven that when the eigenvalues of the transition matrices are allowed to be negative (lying in [−1, 1]), linear RNNs whose transition matrices are products of “identity minus outer product” matrices, including models such as DeltaNet, can recognize any regular language [12] (the eigenvalues of standard GDN are all positive, so strictly speaking this relaxation is needed); and some regular languages (such as the word problem for the symmetric group S5) are NC¹-complete. As for UT and ACT, strictly speaking, the universality proof in the UT paper requires the number of recurrent steps to grow with input length, and ACT is one way to make the number of steps vary with the input.

In Chunyuan’s view, architecture design basically comes down to two lines of thinking: either improve expressiveness, so that the architecture can express more and has a stronger function class; or improve learnability, so that the model can learn that function class more easily. He thinks that, for now, learnability may matter more; expressiveness is discussed more by people on the linear attention side.

5. Loops and MoE: should different loops choose the same experts?

The host noted that MoE is now a fairly mainstream architecture, but as far as she knows, there is still little work on looping in MoE models. Benhao’s blog [3] converted both Ouro-style [13] and Huginn-style [14] architectures to MoE for comparison, and ByteDance Seed and other institutions recently released a paper, SMELT [15]. Her questions: MoE adds parameters, while looping adds FLOPs; MoE also brings issues such as dynamic top-k routing (different loops may select different experts) and communication costs. When loops are added to MoE models, the number of loops seems to be small; SMELT appears to loop only twice (R = 2). Does looping many times not work well with MoE? Or do looped LMs simply not need many loops? Also, could communication costs and the KV cache become a bigger burden with MoE?

Note: SMELT is a collaboration among Tsinghua University, ByteDance Seed, M-A-P and TokenWave.AI. It loops the middle half of the layers twice, and by making the model narrower and adding experts, it matches the baseline in per-token FLOPs, non-embedding parameter count and KV cache. The paper reports that at scales of up to 54B non-embedding parameters (1.6B active), it saves 6.8%–18.0% of training FLOPs along the compute-optimal frontier.

Benhao: the cost must be paid, and the router may favor a few experts

Benhao stressed that he has not run related experiments, so what follows is only a hypothesis: the logic is the same as with KV sharing. The hidden states of different loops are already very similar, and they sit at the same position, which reinforces the similarity, so the router may end up with a few experts that are always especially active. This is also why many works tell us that the redundancy between different loops of a looped model is still large, which is what makes KV sharing, quantization, KV eviction (discarding some of the KVs) and even router reuse possible.

He thinks a router that always picks exactly the same experts would be a problem; if there are a few very active experts while several others vary across loops, that is still acceptable. Perhaps some special designs could be built around this property.

Note: Benhao’s blog [3] has a related result: in the MoE setting, forcing later loops to reuse the experts selected by the first loop lowered accuracy on all six test tasks.

Chunyuan: different loops should choose different experts

Chunyuan largely agreed. He said looping has only two properties: depth and weight sharing. Without extrapolation, and when comparing against a model of the same depth, only weight sharing remains, and every question can be approached from there. For example, in a 12-layer MoE model, let layers 3 and 7 share the router weights W. The router’s logits are the input representation x multiplied by W; when W is the same, which experts each layer selects depends only on its own x. So if the two layers select the same experts, that shows their representations look similar to the router; it is not caused by the shared weights.

How many loops are right?

The host then asked: are there any current looped LMs that loop fairly deeply? The depth of looping should have nothing to do with whether MoE is used.

Benhao said that looping twice is quite a good sweet spot, as seen in Loopie [16] as well as SMELT, mentioned earlier.

Note: Loopie (Loop the Loopies!) follows the Qwen3-MoE architecture and pretrains two models from scratch, Loopie-20B-A2B and Loopie-6B-A0.6B; it uses “intra-layer looping,” in which each layer is executed twice before moving on to the next. Comparing 2, 3 and 4 loops, the paper finds that the marginal benefit of looping is largest at R = 2, while cautioning that this does not mean more loops are ineffective in themselves. SMELT likewise finds 2 loops to be best in ablations at the 200M scale; the paper’s explanation is that, to match FLOPs, the more loops there are, the narrower the model has to be.

6. Extrapolation: feasible on toy tasks, hard for natural language

The host asked two things: first, why is extrapolation unlikely? Second, is extrapolation, that is, looping more times at inference than in training, even necessary? Especially now that the number of loops is generally small, basically two or three, how should we think about extrapolation?

Benhao: it depends on the task

Benhao thinks the answer depends on what the task is; it is highly dependent on the data and the task. Fairly toy problems such as Sudoku, mazes or, somewhat more generally, 3-SAT are actually very well suited to looping, and training a looped model on them can yield the ability to extrapolate.

Note: Benhao’s Equilibrium Reasoners [2] is one such example: the paper reports that on Sudoku, unrolling the model to a depth equivalent to 40,000 layers raises accuracy from 2.6% for a feed-forward model to over 99%.

But if the task becomes natural language, and a very general task at that, with essentially the same distribution as current pretraining data, it becomes very hard, because this is fundamentally a problem of mismatch between training and inference. Toy tasks can reduce this mismatch because in tasks like Sudoku, even intermediate reasoning states are valid inputs: the same set of rules is applied repeatedly to a state, and the rules themselves do not change. Real language settings are not that simple, and the mismatch between training and inference will be severe.

So he thinks that training with only 8 steps and running 16 steps at inference is very hard. Staying stable may already be a good outcome; more often, performance degrades. And as discussed earlier, if the model has not learned a fixed point, using more steps may not work particularly well, all the more so beyond the training distribution.

Chunyuan: it may also be a data problem

Chunyuan said that extrapolation is not only an algorithmic problem; it may also be a data problem: algorithms alone cannot make extrapolation stable, and what helps is finding a recipe that works in certain settings.

7. ACT: reasonable on paper, hard in practice

The host asked: is ACT feasible from an engineering standpoint?

Chunyuan: the infrastructure is too hard, and the gains are unproven

Chunyuan said he has no idea at all how to build the infrastructure. He had previously tried something similar to Mixture-of-Depths [18] himself, that is, MoE along depth. If you route along depth, the first token might want to go through layers 1, 3 and 7, and the second token through layers 1, 4 and 6, and then the problems start: how do you batch? How do you train? He has no idea how such an ML system should be built.

Editor’s note: ACT was proposed by Graves in 2016, originally for RNNs: at each step the model additionally outputs a halting probability, and computation stops once the cumulative probability exceeds a threshold. Mixture-of-Depths uses a router to score each token and lets only the top-scoring fraction of tokens (for example, 12.5%) pass through a layer’s attention and MLP, while the remaining tokens skip it via the residual connection; because the number of tokens each layer processes is fixed in advance, the computation graph remains static.

Suppose a fairly good scenario: ACT works, and people first discover that letting each token take a different number of steps brings very large gains; that would in turn push the ML systems community to come up with better solutions. But he has not seen any recent evidence that token-level ACT brings much benefit; it may instead bring more instability, and training at different depths is also hard.

Benhao: it looks reasonable, but is not worth pursuing further

Benhao agreed with Chunyuan. He sees ACT as a path that seems to make a lot of sense but in practice may not be worth pursuing further; some people have spent a long time on it and found that it still does not really work. One reason is the ML-system problem Chunyuan mentioned; a more fundamental reason, in his view, is that the number of loops is small to begin with: with so few loops, how much room is there for adaptivity?

Moreover, from the perspective of fixed points, the hidden states produced by very shallow loops are still quite unstable, so ACT would aggravate the mismatch between training and inference. The results he has seen are basically all performance drops in exchange for a tiny bit of inference speedup, so he does not think ACT suits the language setting well. However, in the HRM [19] and TRM [20] line of work, which was very popular a while back, ACT is very well motivated, and he sees no problem with those approaches.

Note: HRM (Hierarchical Reasoning Model) has only 27M parameters and achieved strong results on Sudoku-Extreme, Maze-Hard and ARC-AGI with about 1,000 training examples; it implements ACT with a “halt / continue” head trained by Q-learning. TRM recursively applies just a single 2-layer network of about 7M parameters and further simplifies ACT to a single halting head that judges whether the current answer is correct, removing the extra forward pass previously required. On such puzzles, difficulty varies greatly from one puzzle to another, and intermediate states are themselves valid inputs, which may be one reason ACT is well motivated here but hard to make work in language modeling.

8. More realistic speedups: parallel and speculative decoding

Benhao said that, as for the best engineering solution, speculative decoding and parallel generation for looped LMs are a very interesting direction.

Diffusion Forcing [21] from video generation, and the techniques commonly used for real-time game generation in world models, can all be used in looped models. The reason is precisely weight sharing: in diffusion, the same model is fed different diffusion time steps; in looping, the same model is applied repeatedly across different loops. The two are structurally very similar, so a similar kind of parallelism is possible. Rather than doing adaptive looping, it is better to put the effort here, where the speedup will be large.

Note: Diffusion Forcing trains a diffusion model to denoise a set of tokens that each carry an independent noise level, so that, like a causal model, it can start generating later tokens before earlier ones are fully denoised.

He also recommended a paper by the authors of Huginn [14]: Efficient Parallel Samplers for Recurrent-Depth Models [22].

Note: The paper has been accepted at ICML 2026. It proposes a diffusion-forcing-style sampler: every forward pass decodes at least one new token from the intermediate hidden states, while several earlier tokens continue to be refined in parallel within the loop, forming a “wavefront” that moves forward (new tokens have just started looping while older ones have already looped many rounds; within one forward pass, the tokens are at different rounds, like a diagonal line advancing step by step). Total compute stays the same, and the speedup comes from making fuller use of the GPU; on the 3.5B Huginn model, the paper reports speedups of up to about 5× on tasks such as GSM8K, MATH500, HumanEval and MBPP, with very little change in accuracy.

References

[1] LT2: Linear-Time Looped Transformers. arXiv:2605.20670, 2026.

[2] Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning. arXiv:2605.21488, 2026.

[3] Towards Looped Models Done Right. Blog series, 2026.

[4] Training Large Language Models to Reason in a Continuous Latent Space (Coconut). arXiv:2412.06769, 2024.

[5] Full-bandwidth transformer. arXiv:2608.08888, 2026.

[6] The Illusion of State in State-Space Models. ICML 2024, arXiv:2404.08819.

[7] T²MLR: Transformer with Temporal Middle-Layer Recurrence. arXiv:2607.15178, 2026.

[8] Universal Transformers. ICLR 2019, arXiv:1807.03819.

[9] Attention is Turing Complete. JMLR 22(75), 2021.

[10] The Expressive Power of Transformers with Chain of Thought. ICLR 2024, arXiv:2310.07923.

[11] The Parallelism Tradeoff: Limitations of Log-Precision Transformers. TACL 2023, arXiv:2207.00729.

[12] Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues. ICLR 2025, arXiv:2411.12537.

[13] Scaling Latent Reasoning via Looped Language Models (Ouro). arXiv:2510.25741, 2025.

[14] Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (Huginn). NeurIPS 2025, arXiv:2502.05171.

[15] SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers. arXiv:2609.01343, 2026.

[16] Loop the Loopies! (Loopie). arXiv:2607.16051, 2026.

[17] Adaptive Computation Time for Recurrent Neural Networks (ACT). arXiv:1603.08983, 2016.

[18] Mixture-of-Depths: Dynamically allocating compute in transformer-based language models. arXiv:2404.02258, 2024.

[19] Hierarchical Reasoning Model (HRM). arXiv:2506.21734, 2025.

[20] Less is More: Recursive Reasoning with Tiny Networks (TRM). arXiv:2510.04871, 2025.

[21] Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion. NeurIPS 2024, arXiv:2407.01392.

[22] Efficient Parallel Samplers for Recurrent-Depth Models and Their Connection to Diffusion Language Models. ICML 2026, arXiv:2510.14961.

This series: Part 1｜LT2 and Historical KV Replay · Part 2｜Panel, First Half · Part 3｜Panel, Second Half (this post)