This is the second part of the recap of NICE’s event “Looped Language Model: Rethinking Model Depth and Parameters,” covering the first half of the panel discussion. The panel was hosted by Wenyue Hua, with Chunyuan Deng and Benhao Huang as panelists.

The first half started from a general question, “Where are the bottlenecks in LLM architecture?”, and gradually narrowed down to looping itself: What does it cost? Do its gains really come from the architecture, or just from spending more compute? And a more technical question: Should looped models learn fixed points?

Key takeaways

  • FLOPs are relatively cheap; memory and bandwidth are more expensive and grow more slowly. Benhao prefers architectures that “add only FLOPs, not memory” (improving performance by adding compute without adding memory footprint such as parameters and KV cache), and the ideal latent CoT is exactly that. Chunyuan thinks the Transformer has no particularly big bottleneck and is more optimistic about architectures for settings where training data is repeated. Benhao added that the data wall (the total amount of high-quality training data is limited and will eventually fail to keep up with the growth of models and compute) may matter more than architecture.
  • Looping is not a free lunch. Without KV sharing or compression, a looped LM offers little beyond some inductive bias; almost everything else is a cost: more KV cache, and performance that cannot catch up with a standard Transformer of the same logical depth without weight sharing. Many papers compare only against models with the same number of parameters, not against this “upper bound.”
  • MoE may make up for having fewer parameters. Parameters determine how much knowledge a model can store. Benhao observed that in looped MoE models, different loops do activate different experts, and adding MoE narrows the gap to the upper bound.
  • Fair comparisons look at entire scaling curves. Looping spends extra compute. When comparing with a model that does not loop (the baseline), Chunyuan argues that one should not change the baseline’s architecture just to equalize FLOPs, for example by changing the size of the feed-forward layers (FFN) or the dimension of the attention heads (head dim). A more sensible approach is to train a set of models from small to large (a scaling ladder) and compare the entire curve of “loss versus compute.” Benhao added that the baseline should be tuned to its best, and its width-to-depth ratio should not be changed arbitrarily just to match compute; spending more compute in training without adding cost at inference is, in his view, acceptable.
  • Whether to learn fixed points depends on the goal. “Learning fixed points” means training the model so that its hidden state essentially stops changing after a certain number of loops, and looping a few more times does not make it worse. Benhao argues that if you only want the best performance at a fixed number of loops (say, 4), there is no need to learn fixed points. If you do want to, the common method is depth sampling: during training, a number of loops is drawn at random for each batch, the model loops that many times and the loss is computed at the last step. The benefit is stability: every depth may be drawn and have a loss computed on it, so the model learns not to let the loss rise when it takes a few more steps. The cost is that supervision is spread across depths, so less supervision falls on the number of loops actually used at inference, which may lower performance.
  • Path independence is key. Path independence means that no matter which initial state the model starts from and how many steps it iterates, it ends up at the same fixed point. Benhao believes that Huginn (a looped language model that loops only its middle layers, introduced in Section 4) learns fairly good fixed points mainly thanks to two training techniques, each corresponding to one half of path independence: random depth sampling makes the result independent of the number of iteration steps, and truncated backpropagation (computing gradients only through the last few loops during training, while earlier loops run only the forward pass) makes the result independent of the starting point. At the same time, suitable normalization and input injection can better guide how fixed points form.

Glossary

The table below explains several terms that recur in this article. Readers familiar with them can skip it.

Term Plain-language explanation
Loop / Looped LM Running the same set of layers (or the whole model) several times with the same parameters each time
Logical depth The number of layers a token actually passes through: the number of looped layers × the number of loops, plus the layers that are not looped
Upper bound / lower bound Two reference points for evaluating a looped model: the lower bound is a model with the same number of parameters that does not loop; the upper bound is a standard Transformer of the same logical depth in which no layer shares parameters
KV cache The keys and values that standard attention caches for every past token during generation; looping n times usually means storing n copies
FLOPs Floating-point operations, commonly used to measure a model’s compute; “adding FLOPs” means making the model compute more
Memory wall The trend of hardware compute growing much faster than memory bandwidth, which leaves many models bottlenecked on “moving data” rather than on “computing”
Data wall The total amount of high-quality data available for pretraining (mainly public human-written text) is limited, while model scale and compute keep growing quickly, so data will sooner or later fall behind and become the bottleneck to further scaling
MoE Mixture of experts: each layer has many expert subnetworks, and a router picks only a few of them to compute for each token, so the parameter count grows greatly while compute stays the same
Scaling ladder Training a series of models from small to large with the same recipe, fitting the curve of “loss versus compute,” and then comparing different architectures’ entire curves
Ouro / Huginn Two representative looped language models: Ouro loops the whole model, while Huginn loops only its middle part
Prelude / core / coda Huginn’s three-part structure: the first part encodes the input into the latent space, the middle part loops repeatedly, and the last part decodes the hidden state into the output
Input injection Re-injecting the encoding of the original input into the hidden state in every loop, so the model does not “forget” the input after many loops
Fixed point / attractor A state that barely changes when iterated once more. An attractor is a fixed point that nearby states are drawn into as they keep iterating
Depth sampling During training, a number of loops is drawn at random for each batch, and the loss is computed at that depth
DEQ Deep Equilibrium Models: solve directly for the network’s fixed point and compute gradients with implicit differentiation, equivalent to an “infinitely deep” weight-sharing network
Truncated backpropagation Backpropagating gradients only through the last few iterations, with earlier steps running only the forward pass; a common practice in training RNNs that saves GPU memory
CKA A metric of how similar two sets of neural network representations are
Jacobian and contraction The Jacobian describes how small changes in the input affect the output; when all its eigenvalues have absolute value less than 1, the iteration contracts near the fixed point and converges to it

Panelists

Chunyuan Deng: PhD student at Rice University, advised by Prof. Hanjie Chen. His research focuses on the design and optimization of LLM pretraining architectures, as well as the interpretability of language models. His representative work is LT2: Linear-Time Looped Transformers [1]; his talk at the event is covered in part one of this series.

Benhao Huang: Master’s student at CMU, advised by Prof. Zico Kolter. He works mainly on looped LMs and fixed points, with the aim of improving the reasoning ability and efficiency of foundation models. His representative work includes Equilibrium Reasoners [2] and the blog series Towards Looped Models Done Right [3].

Wenyue Hua (host): Member of Technical Staff at Microsoft AI (MAI). PhD from Rutgers University, advised by Prof. Yongfeng Zhang, followed by a postdoc at UCSB. Research focuses on LLM reasoning and LLM-based agents, with many papers at top NLP and machine learning venues, including ACL, EMNLP, ICLR, NeurIPS and TACL. Serves as an Area Chair for ACL.

1. Bottlenecks in LLM architecture: compute, memory and data

What are the main bottlenecks of today’s large models? Have they driven most current architecture design? And what are the main solutions to them?

Benhao: FLOPs will always be cheaper

Benhao thinks that GPU memory, bandwidth and FLOPs are certainly all problems. Architecture design usually takes alignment with hardware into account, and a well-known trend line tells us that as hardware develops, memory and bandwidth grow more slowly and are bound to be more expensive, while FLOPs will always be cheaper.

Note: This trend is often called the “memory wall.” According to statistics compiled by Gholami et al. [4], over the past 20 years the peak compute of server hardware grew by about 3.0× every two years, while DRAM bandwidth grew by only about 1.6× and interconnect bandwidth by about 1.4× every two years; over 20 years, that adds up to roughly 60,000×, 100× and 30×, respectively.

So Benhao generally prefers architectures that get better results by scaling up FLOPs alone, without enlarging memory. This of course has a cost, namely latency: the more FLOPs, the slower it naturally gets. “FLOPs are cheap” is about cost, while “slow” here refers to the time it takes to generate each token. Take looping as an example: the next loop for a given token has to wait until the previous one finishes, so the extra computation can only run sequentially, and the more loops there are, the longer each token takes to generate. But this is still acceptable for now, especially assuming that frontier labs have nearly unlimited FLOPs.

What he likes about one of his own recent papers is precisely that it explores whether there is a way to get better performance purely by adding FLOPs, without increasing memory overhead. He sees this as perhaps the ultimate form of latent CoT (reasoning in the latent space without writing out the intermediate steps as text):

You don’t externalize your thought tokens, you don’t have extra KV cache, and you don’t have any extra parameters. You just have more FLOPs: you think more in your head, spend more energy, and then come up with a better method, a better idea.

Chunyuan: the Transformer has no particularly big bottleneck

Chunyuan said he does not think the Transformer has any particularly big bottleneck; in architecture design, we can always do better: be more efficient, use less memory, compute more efficiently.

One direction he finds fairly promising: many labs have recently been studying architectures for the data-repetition setting. Data is becoming scarcer, and training on repeated data has become common. So what kind of architecture can provide regularization and avoid overfitting when trained on repeated data? And what kind of optimization method should go with it?

Note: A frequently cited result on data repetition comes from Muennighoff et al. [5]: with a fixed compute budget, training on the same data for up to about 4 epochs yields almost the same loss as using fresh data; but with more repetitions, the returns from additional compute eventually decay to zero.

Benhao: the data wall matters more

Benhao added that the data-efficiency and data-repetition questions Chunyuan raised are very interesting. The data wall is also a bottleneck, but he thinks it matters more than architecture: “Data always comes first.” If data is a first-class citizen, then finding architectures that can adapt as data becomes scarcer would be a very valuable direction.

2. The cost of looping: KV cache, the performance upper bound and MoE

Looping does not reduce the memory taken up by KVs. It adds FLOPs under a parameter constraint, but the demands on memory and bandwidth also seem to go up. How should we think about this?

Apart from a bit of inductive bias, it’s almost all cost

Benhao’s answer was quite blunt: following the usual practice so far, without any KV sharing or compression techniques, a looped LM may offer a bit of inductive bias (a preference for certain kinds of solutions that comes from the architecture itself), and everything else is basically a cost: it has more KV cache, and its performance cannot catch up with a vanilla Transformer of the same logical depth.

He also pointed out that most papers do not compare against the upper bound and usually report only the lower bound: they compare only with a model that has the same number of parameters and does not loop, not with a standard Transformer of the same logical depth without weight sharing. “I haven’t seen one that really beats that upper bound, and I think it may simply be impossible to beat.” Chunyuan added: “It can’t beat it.” Benhao went on to say that some might argue it could win at a large enough scale, but in his view this resembles the debate over whether diffusion language models can beat autoregressive models, and his judgment is that it cannot.

Chunyuan agreed, and admitted that his own work also did not compare against models of the same depth. He had tried it himself, and the looped model really did fall short, especially in pretraining; if there is no extrapolation and it still falls short, that is a real problem. One argument he could think of: if extremely sparse, very large MoE models appear in the future, say with 1000T parameters that cannot fit in GPU memory, weight sharing might help. “Other than that… it gets a bit tricky.”

Can MoE make up for having fewer parameters?

The host then turned to MoE: since Chunyuan said architecture has no particular bottleneck, why has MoE become mainstream? As she understands it, MoE adds parameters without adding FLOPs to the same extent, while still keeping fairly good results. Is this choice mainly driven by latency?

Benhao’s take is that a major problem for looped LMs is still parameters, and parameters determine how much knowledge a model can store. MoE is a good way to alleviate this: different activations lead the router to choose different experts and thus draw on different knowledge. In looped MoE models, different loops do activate different experts, which amounts to retrieving different knowledge at each step.

Conversely, without MoE, the hidden states of different loops are quite similar, which hurts performance. In his earlier blog [3], he found that without MoE the gap between a looped model and the upper bound is fairly large; once both the looped model and the standard Transformer use MoE, the gap becomes smaller.

Note: According to the first part of the blog, the comparison is with a 112-layer model that has the same logical depth and the same active compute as the looped model but does not share weights across layers. Switching from dense to MoE narrows the looped model’s average gap to it from 4.96 points to 1.71 points, while the looped model has 75% fewer resident parameters. The blog also found that forcing later loops to reuse the experts selected by the first loop lowered accuracy on all six test tasks.

3. Gains from architecture or from compute: what makes a fair comparison

Do the gains of looping really come from the architecture, or just from spending more compute? What is a fair way to test this?

Chunyuan: one approach is definitely wrong

Chunyuan began by saying that comparing looped models is “really tricky.” First of all, one approach is definitely wrong: since looping adds FLOPs, people change the vanilla Transformer’s FFN, head dim or other parts of its architecture so that its FLOPs match those of the looped model. Changing a lot of the original architecture just to reach equal FLOPs “definitely doesn’t work.”

He thinks the most standard way to compare is with a scaling ladder in pretraining. For example, train a set of standard Transformers from small to large in the usual way, with the third rung being, say, a 300M model. Looping adds FLOPs, so every point of the looped version shifts to the right along the horizontal axis (compute). “Isn’t that the whole point of a scaling ladder?” A scaling ladder is meant to fit a power law and compare two curves anyway, so there is no need to worry about whether each rung lines up exactly with the baseline.

As for comparing wall-clock time (actual running time), he thinks it is just too hard for academia: the various GPU setups are hard to compare with real 10,000-GPU clusters.

Benhao: tune the baseline well, and look at training and inference separately

Benhao added two points:

  • Always tune the baseline to its best, and never compare against a weak baseline. He gave the example that some recent work widens or narrows the model’s dimension in order to match FLOPs. But there is a relatively optimal range for the balance between width and depth, which cannot be adjusted arbitrarily; going outside that range makes the comparison unfair.
  • Training and inference costs can be considered separately. If paying some extra cost in training (usually compute) buys a performance gain with no extra cost at decoding time, he considers that acceptable. In his view, a major difference between academia and industry is that academia is very preoccupied with fair settings, while industry cares more about what gains a model brings when it is actually served.

The host remarked that this means many points are needed to compare two curves properly, which is expensive. Benhao said: “That’s the brutal part of pretraining.” It is genuinely hard for academia: even with a ladder, getting the hyperparameters right (including the width and depth just mentioned) is very difficult. As a result, many papers do claim some benefits, but benefits that hold up are rare.

Follow-up: what does “no extra cost at decoding time” mean?

The host followed up: when Benhao said earlier “no extra cost at decoding time,” did he mean that inference speed is unaffected? Could this be achieved with pipelined looping (staggering different loops of different tokens so that they compute in parallel, like a pipeline) and with how the KV cache is handled?

Benhao replied that he mainly had in mind approaches like the Full-Bandwidth Transformer [6]: add a side path to the model that costs some extra compute in training but adds almost no overhead at decoding time. In his view, this is a more typical example than pipelining; the pipelined approach also makes sense, but it has some subtleties, which he would come back to later (see Section 8 of part three of this series).

Editor’s note: The side path in the Full-Bandwidth Transformer takes the top-layer hidden state at the previous position, fuses it with the next token’s embedding through a gate, and uses the result as the input at the next position. Training requires a few extra forward passes, but by the paper’s estimate, the added compute per token at decoding time is under 1% (about 2.3% slower when measured in vLLM). See part three of this series for details.

4. Two representative architectures: Ouro and Huginn

Before moving on to fixed points, the host asked the two panelists to give some background on two architectures, Ouro and Huginn.

Chunyuan: the core difference is whole-model looping versus middle looping

Chunyuan summed it up: Ouro [7] is a whole-model loop: it loops the entire model and uses early exit, so it can stop early; Huginn [8] is a middle loop: it loops a middle part of the model and uses a latent state and input injection. In his view, the main difference between the two is whole-model looping versus middle looping; the other components (such as input injection) can be borrowed by either.

Note: The table below summarizes details reported in the two models’ papers.

  Ouro [7] Huginn [8]
How it loops The whole model loops Only the recurrent core in the middle loops, with a prelude before it and a coda after it
Scale 1.4B (24 layers) and 2.6B (48 layers) 3.5B; the prelude, core and coda have 2, 4 and 2 layers, respectively
Training data About 7.7T tokens About 800B tokens
Number of loops in training Fixed at 4 Randomly sampled for each micro-batch, with a mean of 32 and a long-tailed distribution
Training details A language-modeling loss at every step, plus a learned early-exit distribution Gradients are backpropagated only through the last 8 iterations; in each loop, the input embedding is concatenated with the current state and mapped through a linear layer
Inference After each loop, an exit gate gives the probability of “stopping here”; looping stops once the cumulative exit probability reaches a set threshold, so easier tokens can use fewer loops The number of loops can be chosen freely; the paper reports results up to 64

Benhao’s three additions

First, input injection helps somewhat with middle looping. For a middle-looping architecture like Huginn, input injection does help to some extent, especially with somewhat more loops; without it, a Huginn-style architecture tends to lose to Ouro at the same logical depth.

Second, Ouro “grows” a middle loop inside itself. If you match the two models’ logical depth, unroll the looped parts and compare similarity metrics such as CKA [9] for the hidden states at the same layer, you find that Ouro has learned on its own a structure very similar to middle looping: it has layers that correspond to a prelude and a coda. The prelude and coda can be understood as encoding and decoding, with step-by-step refinement of the hidden representation in between.

He also mentioned that similar phenomena may exist in other looped models, so this is not necessarily a new finding; but Huginn’s approach of looping in the middle may happen to fit this inductive bias. He further mentioned that some papers argue that RL mainly changes a small part in the middle of the model, which he guesses may be responsible for abilities such as reasoning.

Note: One example is a July 2026 preprint [10]: it found that the gains from RL are highly concentrated in a few layers, that training a single layer alone can often come close to the effect of full-parameter RL, and that the layers contributing most are concentrated in the middle of the network.

Third, Huginn’s random depth sampling is very important. It is crucial both for shaping fixed points and for adjusting compute at inference. Benhao thinks the term “test-time scaling” is somewhat ambiguous: in his view, what it really means is extrapolation, for example having seen only 16 loops in training and wanting to loop 64 times at inference. “Unfortunately, I don’t think that’s very likely,” especially for language models. What random depth sampling really supports is elastic compute: loop at most 16 times in training, and at inference you can freely choose how many times to loop, as long as it does not exceed 16.

5. Whether to learn fixed points: the pros and cons of depth sampling

With the architectures laid out, the host put the question to Benhao. He gave his conclusion first: if you only want the best performance at a fixed number of loops (say, 4 or 5), using the model as an ordinary Transformer and ignoring other costs such as memory, then you should not try to learn fixed points, and performance will be better that way.

The common approach: depth sampling

If you do want to learn fixed points, you need some additional algorithms, the best known of which is depth sampling: during training, sample a depth for each batch (or even each micro-batch), run that many steps forward, and then apply the loss at that step.

One of Benhao’s core points is that supervision itself is a resource. Suppose the target budget is 5 loops. Depth sampling puts part of the supervision on steps 1 to 4, or at deeper steps, so step 5 gets correspondingly less supervision, and the model’s performance at 5 steps drops.

The benefit it brings is stability: it requires that the loss not rise with each additional step. This does not necessarily improve performance, but it makes the model more stable. There may be some math behind this: from the perspective of the Jacobian and its eigenvalues, for the iteration to contract, the eigenvalues need to be less than 1, which can be proven. But the most intuitive way to see it is this: if every depth has a loss and the loss is not allowed to go up, then the simplest shortcut the model can learn is to stay put in a stable state, which is what we call a fixed point, or attractor.

A counterintuitive experimental result

But his own experiments found that even when the target budget is 5, the mean of the depth distribution is also set to 5, the long tail extends all the way to 64, and the model is unrolled to 64 steps at test time, it does not necessarily beat a control model trained at a fixed depth of 5 without learning fixed points.

His conclusion: this may be a problem with the algorithms for learning fixed points, not with fixed points themselves. Depth sampling is just one way to learn fixed points. DEQ (Deep Equilibrium Models) [11], for example, makes training and inference fully consistent: in the forward pass during training, it iterates with a fixed-point solver until convergence reaches a certain threshold; in the backward pass, it uses the implicit gradient, which in the ideal case corresponds exactly to the mathematical solution of backpropagation. In this case, it may be possible to reach very good performance without necessarily running into the problem above; the difficulty is that it is very hard to train: DEQ is notoriously unstable and needs tricks such as warm-up. Depth sampling is a compromise.

Note: DEQ was proposed in 2019 by Shaojie Bai, Zico Kolter and Vladlen Koltun. It can be seen as an infinitely deep weight-sharing network: instead of computing layer by layer, the forward pass uses a root-finding algorithm (Broyden’s method in the original paper) to solve directly for the fixed point satisfying z* = f(z*, x); the backward pass computes gradients with the implicit function theorem at the fixed point, without storing intermediate steps.

6. Are fixed points ideal solutions or degenerate states?

The host asked: is a learned fixed point the solution we want, or might it be a degenerate state?

Fixed points and performance are decoupled

Benhao said that the argument of one of his earlier papers [2] is that fixed points themselves are actually decoupled from model performance. A model may well have many fixed points (or attractors), and what training does is align these attractors with the correct task objective as much as possible; that is, we want the hidden state corresponding to an attractor to decode into exactly the correct tokens, the correct answer. So fixed points are not necessarily degenerate.

Note: The paper is Equilibrium Reasoners. It argues that reasoning generalizes because the model learns “task-conditioned attractors”: the stable fixed points of the latent dynamics correspond exactly to valid solutions. On tasks such as Sudoku and mazes, the paper scales test-time compute by increasing the number of iteration steps and using multiple random restarts; it reports that when unrolled to a depth equivalent to 40,000 layers, accuracy on Sudoku-Extreme rises from 2.6% for a feed-forward model to over 99%.

If degeneration really does occur, it may be because the model had not yet reached the attractor when supervision was applied, so the supervision was placed on the path before the attractor. In that case, after a few more iterations, the model may indeed converge, but to a bad point.

Why Ouro gets worse with more iterations

He thinks this may also be why Ouro’s performance drops markedly when it runs more iterations than in training. The host asked: does Ouro’s design simply never reach a fixed point? Benhao’s view is that Ouro may not do enough to “push” supervision onto the fixed point, but having a loss at every loop helps to some extent, and iterating further is unlikely to diverge. The problem is that it loops only 4 times in training and has no depth sampling, so nothing encourages the model to stay near its step-4 state after looping more than 4 times; it may therefore keep moving until it reaches a degenerate place. By comparison, Huginn should be fine as long as it stays within 64 steps, with only a slight decline beyond 64.

Note: Ouro loops a fixed 4 times in training (it initially tried 8, but switched to 4 because of loss spikes and gradient oscillations), with a language-modeling loss at every step. The paper reports that increasing the number of loops to 5–8 leads to a “moderate degradation” in the base models’ performance: at 8 loops, MMLU drops from 67.45 to 64.49 for the 1.4B model and from 74.60 to 72.24 for the 2.6B model; the reasoning versions drop more on hard problems, for example AIME25 for the 2.6B reasoning model falls from 50.30 to 24.33. As for Huginn, performance keeps improving up to about 64 steps and stops improving after that, but usually does not get worse either.

Decoupled, but also coupled

The host summarized that it sounds as if there is no necessary connection between fixed points and performance. Benhao agreed with this way of thinking, but pointed out some subtleties: some methods couple the two. Take depth sampling: on the one hand, it places supervision at very large depths, which amounts to aligning the fixed point with the task objective; on the other hand, it can also shape fixed points itself, as a technique for achieving “path independence,” which the next section covers in detail. “So it solves two problems at once.” The two are coupled and not so cleanly separated, but thinking about them separately like this is helpful.

Note: Fixed points have a very practical use as well. In the second part of their blog series [13], Benhao’s team reports that keeping only each token’s KVs from its last loop during decoding (terminal KV sharing) can shrink the KV cache of a 1.6B model to one third. A model trained at a fixed depth loses a lot of accuracy when doing this (GSM8K drops from 50.6 to 21.2), while a model trained with a learned depth prior (the distribution of loop depths in training is no longer set by hand but learned from predictive performance, with an entropy term that keeps the distribution wide enough) is essentially unaffected.

7. How architecture and algorithms affect convergence: path independence

The host then asked: how do architecture design and algorithm design affect convergence, including what kind of point the model ultimately converges to? Benhao listed three kinds of approaches:

  • Residual scaling: architecturally, an important factor is scaling the residual connections; some papers multiply the residual by a scaling coefficient so that later layers contribute less [14].
  • Input injection: well-designed input injection can aid convergence, but injection by itself does not guarantee it; its bigger role is to make the point the model converges to depend on the original input context. If you iterate to infinity without continually injecting the context, you get a degenerate attractor that has nothing to do with the prompt and is therefore meaningless. This is how the DEQ line of work understands the value of input injection.
  • Jacobian regularization on the loss side: this is no longer architecture design but an algorithm: imposing some constraints on the Jacobian [12] can directly promote convergence. He mentioned that the recently popular Fixed-Point Reasoners [14] also uses a similar technique.

Note: Fixed-Point Reasoners does not use Jacobian regularization directly. Instead, it keeps the state bounded with pre-norm plus a shared, learnable residual scaling coefficient (also applied to the input re-injected in each loop), and promotes convergence with a contraction condition encouraged through initialization and with damped updates; at inference, it stops looping once the fixed-point residual is small enough. Its goal (making the iteration contract) is the same as that of Jacobian regularization, but its specific means are closer to the first category above.

In Huginn’s case: depth sampling and truncated backpropagation

The host followed up: does Huginn converge to a relatively good fixed point because it has input injection and uses normalization, plus depth sampling?

Benhao thinks that, at least in the Huginn paper, normalization and input injection are not the most important factors. Two things matter more: one is depth sampling, and the other is truncated backpropagation, which has not come up yet. Truncated backpropagation means not backpropagating gradients all the way: for example, once the number of loops exceeds a certain number of steps, you always backpropagate through only a fixed number of final steps, so memory does not keep growing with the number of steps. This is a very common practice in training RNNs, and it also helps shape fixed points.

People may think of both from the perspective of efficiency, and both are indeed economical. Depth sampling does not bet a lot of resources on very large depths, which makes sense because not every token needs such deep computation. The same goes for truncated backpropagation: if the memory and activations for backpropagation grow linearly with the number of forward steps, training can easily blow up. But beyond efficiency, both also help shape fixed points.

Path independence

Benhao said this can be understood from the perspective of path independence. A well-known paper in the DEQ line of work [15] argues that for attractors to perform well enough, or in its words to generalize well enough, path independence must be ensured. Path independence means that no matter which point you start from and how many steps you take, you end up at the same fixed point or attractor.

Note: The paper found that whether an equilibrium model can solve harder problems by increasing the number of iterations at test time is highly correlated with how path-independent the system is.

In his view, the two techniques correspond exactly to the two halves of path independence:

  • Depth sampling makes the result independent of the number of steps. The number of steps taken in training is random, so the model has to learn to reach the same state for any number of steps.
  • Truncated backpropagation makes the result independent of the starting point. The states of the early steps contribute no gradient, so the model “doesn’t know where it came from, but knows where it is heading in the end.”

So he concluded that, for Huginn, these two are the more important factors in shaping fixed points; suitable normalization and input injection are also important design choices for preventing divergence.

Closing and what’s next

The first half of the discussion follows a clear thread: the costs of looping are real, namely more KV cache and performance that struggles to reach the upper bound set by a standard Transformer of the same logical depth without weight sharing; its benefits have to be validated with complete scaling curves and well-tuned baselines. “Whether to learn fixed points” is essentially a trade-off between the best performance at a fixed depth and stability over more iterations. Benhao also offered a very useful perspective: depth sampling and truncated backpropagation make the result independent of the number of steps and of the starting point, respectively.

The next part covers the second half of the panel: whether looping along depth is the same thing as CoT, whether expressiveness proofs still mean anything, how loops can be combined with MoE, and extrapolation, ACT and parallel decoding.

References

[1] LT2: Linear-Time Looped Transformers. arXiv:2605.20670, 2026.

[2] Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning. arXiv:2605.21488, 2026.

[3] Towards Looped Models Done Right. Blog series, 2026.

[4] AI and Memory Wall. IEEE Micro; arXiv:2403.14123, 2024.

[5] Scaling Data-Constrained Language Models. NeurIPS 2023, arXiv:2305.16264.

[6] Full-bandwidth transformer. arXiv:2608.08888, 2026.

[7] Scaling Latent Reasoning via Looped Language Models (Ouro). arXiv:2510.25741, 2025.

[8] Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (Huginn). NeurIPS 2025, arXiv:2502.05171.

[9] Similarity of Neural Network Representations Revisited (CKA). ICML 2019, arXiv:1905.00414.

[10] Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training. arXiv:2607.01232, 2026.

[11] Deep Equilibrium Models (DEQ). NeurIPS 2019, arXiv:1909.01377.

[12] Stabilizing Equilibrium Models by Jacobian Regularization. ICML 2021, arXiv:2106.14342.

[13] Towards Looped Models Done Right, Part II: Rethinking at Fixed Points. arXiv:2610.06833, 2026.

[14] Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers. arXiv:2606.18206, 2026.

[15] Path Independent Equilibrium Models Can Better Exploit Test-Time Computation. NeurIPS 2022, arXiv:2211.09961.

This series: Part 1｜LT2 and Historical KV Replay · Part 2｜Panel, First Half (this post) · Part 3｜Panel, Second Half