On October 17 at 9 p.m. ET (9 a.m. on Sunday, October 18, Beijing time), NICE hosted an online talk and panel, “Looped Language Model: Rethinking Model Depth and Parameters.” There was a lot to cover, so this recap comes in three parts: the first covers Chunyuan Deng’s talk, and the second and third cover the two halves of the panel discussion. This is part one.

The idea behind looped LMs is simple: run the same set of layers several times, adding computational depth without adding parameters. But a loop saves parameters, not KV cache. That is where Chunyuan’s talk starts: can combining loops with linear attention keep the benefits of looping while lowering the cost of inference? Beyond the analysis in the published paper, he also presented Historical KV Replay, a new method from his team that is not yet in the public version of the paper.

Key takeaways

  • Looping saves parameters, not KV cache. When a standard (full-attention) Transformer loops, its KV cache grows linearly with the number of loops. At long context lengths, a full-attention model that does not loop at all is more expensive than a linear-attention model that loops many times. This is why LT2 moves looping onto linear attention.
  • Writing to the same memory several times increases expressiveness. Linear attention compresses the history into a fixed-size “memory,” which you can think of as a lookup table. For each incoming token, the delta rule “erases, then writes” the entry associated with that token: it first erases the old content, then writes the new content. Per the DeltaProduct analysis, erasing and writing n times at the same position updates n entries in one step instead of just one, provided the n writes really modify different, non-overlapping entries, which the team verified largely holds in common settings.
  • But when linear attention is looped directly, each loop writes to its own memory. The theory above therefore does not strictly apply, and the recurrence over time and the recurrence over depth are never connected. This leaves room for co-design.
  • The new method, Historical KV Replay, replays earlier loops’ KVs in reverse order. It brings large gains in both recall and language modeling. Order is crucial: reverse order works, while forward order barely helps.
  • The cost is manageable, and the gains are not just from “computing more.” The amount of replay grows quadratically with the number of loops (loop k writes its own KVs and also rewrites the KVs of the previous k−1 loops, so 4 loops require 1+2+3+4 = 10 writes in total), but models today usually loop only two or three times. In controlled comparisons, the team also tried other ways of adding the same amount of compute: looping more times (just more rounds, with each loop still writing only its own memory), widening GDN (Gated DeltaNet, the linear attention that LT2 loops), and stacking DeltaProduct inside each layer (erasing and writing several times at the same position). All of these helped, but none matched replay.
  • Extrapolation matters more than tinkering with the architecture. Chunyuan argues that without extrapolation, one has to work very hard to justify the value of weight sharing (if a looped model cannot loop more at inference than in training, it is just a fixed-depth model whose layers share parameters, and its main advantage is fewer parameters; extrapolation is what lets it loop more on demand at inference time, trading extra compute for better results). ACT (adaptive computation, which lets each token decide how many loops to run based on its difficulty) is very hard to implement, and it is unclear whether it can be solved. Post-training, especially combining on-policy distillation with loops, deserves attention.

Glossary

The table below explains several terms that recur in this article. Readers familiar with them can skip it.

Term Plain-language explanation
Loop / Looped LM Running the same set of layers (or the whole model) several times with the same parameters each time. Looping 4 times is like stacking those layers 4 times while keeping only one copy of the parameters
Weight sharing Multiple loops use the same set of parameters; this is where looping’s parameter savings come from
KV cache The keys and values that standard attention caches for every past token during generation, so that later tokens can query them; it grows with context length, number of layers and number of loops
Linear attention Instead of storing all past KVs, compresses the history into a fixed-size state matrix (recurrent memory) that is updated token by token; its memory does not grow with context length
Delta rule An update rule for linear attention: before writing new content, it partially erases the old content in the current key’s direction, i.e., “erase once, write once”
GDN / KDA Two linear-attention variants based on the delta rule. GDN (Gated DeltaNet) adds a forget gate; KDA (Kimi Delta Attention) makes the forget gate finer-grained
Head dim The dimension of the key and query vectors in each attention head, commonly 128 or 256
SWA Sliding window attention: each token only attends to tokens within a fixed-length window of recent tokens
Hybrid An architecture that combines a few full-attention layers with many linear or sparse attention layers, e.g., one full-attention layer for every three linear-attention layers
Recall The ability to accurately retrieve a specific piece of information from a long context, typically tested with “needle in a haystack”; a traditional weakness of linear attention
Chunkwise A common parallel algorithm for linear attention: split the sequence into chunks, compute in parallel within each chunk, and recur across chunks, which makes efficient GPU kernels possible
Extrapolation Looping more times at inference than in training (e.g., training with 8 loops and running inference with 16) while the model still works or even improves
ACT Adaptive Computation Time: each token decides how many loops to run based on its difficulty
On-policy distillation The student model generates its own content and the teacher gives token-by-token guidance on it, instead of the student simply imitating text the teacher wrote

Speakers

Chunyuan Deng (speaker): PhD student at Rice University, advised by Prof. Hanjie Chen. His research focuses on the design and optimization of LLM pretraining architectures, as well as the interpretability of language models. The work presented in this talk is LT2: Linear-Time Looped Transformers [1].

[Figure: cover slide of Chunyuan’s talk]

1. Motivation: looping saves parameters, but not KV cache

The selling point of looped models is parameter efficiency: reuse the same layers several times, and the computational depth multiplies while the parameter count stays the same. But when Chunyuan’s team first started working on loops, their impression was that they were “very expensive.” The reason is that a loop shares parameters, but not the KV cache.

In a full-attention Transformer, every layer stores a key and a value for every token. If the model loops 4 times, the same token has to store a copy in each loop, so the KV cache grows in proportion to the number of loops. For looping on full attention, this overhead is “simply unacceptable,” and it is hard to imagine extrapolating it to many loops.

Linear attention is different: its recurrent memory is small, and both its training FLOPs and its inference memory scale well. As a result, at long context lengths a full-attention model that does not loop at all is far more expensive than a linear-attention model that loops many times.

Note: Here is a back-of-the-envelope estimate for a single layer and a single attention head (ignoring compression techniques such as GQA). Let the head dimension be d and the context length be N. Full attention stores a key vector and a value vector for every token, 2N·d numbers in total. The state of a linear attention such as GDN is a d×d matrix, d² numbers, independent of N (linear attention takes the outer product of each token’s key and value to get a d×d matrix and accumulates it into the same state, so the state stays the same size no matter how many tokens are read). With d = 128 and N = 32K, the former is about 512 times the latter. So even if a linear-attention model loops a hundred times or more, its total per-layer state is still smaller than that of a full-attention model that does not loop.

This is the starting point of LT2 [1]: loop on linear attention, so that both training FLOPs and inference memory stay well-behaved. A set of numbers from Chunyuan’s slides makes this concrete: for a 0.6B model at 32k context, the looped Transformer’s KV cache is 12.1 GB, while LT2’s is only 0.05 GB; decoding speeds are 6.3 and 19.4 tokens per second, respectively.

[image: Chunyuan’s slides, p. 3]

Figure: A looped Transformer stores a separate KV cache for each loop, which grows with context length; LT2 keeps only one fixed-size state per loop.

2. The intuition: write to the same memory several times

LT2 mainly studies linear attention of the GDN [2] and KDA [3] type. As Chunyuan explained, their state transition matrix (the matrix that, each time a token is read, determines how the old memory changes: how much it decays and what gets erased; see α_t(I − β_t k_t k_tᵀ) in the formula below) can be written in a “diagonal plus low-rank” (DPLR) form. This class of linear attention currently works well and is widely used.

First, what does the delta rule do? For each incoming token, the model updates the memory matrix S using the token’s key k_t, value v_t and write strength β_t (using GDN as the example, with slightly simplified notation):

S_t = \alpha_t \left(I - \beta_t k_t k_t^{\top}\right) S_{t-1} + \beta_t k_t v_t^{\top}

The term I − β_t k_t k_tᵀ in parentheses is the identity matrix minus a rank-1 matrix: it partially erases the memory’s old content in the k_t direction, and the second term then writes the new value into that direction. This is what Chunyuan meant by “erase once and write once along k_t.” α_t is the forget gate, which controls how much the old memory decays overall; KDA makes this gate finer-grained than GDN does. In the GDN paper’s framing, the gate enables rapid memory erasure, while the delta rule enables targeted updates.

The earlier DeltaProduct [4] work goes a step further: instead of erasing and writing once per token, it does so n_h times in a row at the same position. The state transition matrix then becomes a product of n_h such matrices:

A_t = \prod_{j=1}^{n_h} \left(I - \beta_{t,j}\, k_{t,j} k_{t,j}^{\top}\right)

Each factor is “identity minus rank 1,” so the product is “identity minus a matrix of rank at most n_h”; together with the forget gate, this is what the paper calls “diagonal plus rank n_h.” Intuitively, one erase-and-write can only change the memory’s content in one direction, while n_h of them can change n_h directions. A “direction” here is the direction of a key vector: if you think of the memory as a table looked up by key, each key direction roughly corresponds to one entry, and the closer the keys of the successive writes are to orthogonal, the more they modify unrelated entries. DeltaProduct’s theoretical analysis shows that the larger n_h is, the stronger the model’s state tracking (e.g., working out where each item ends up after a sequence of swaps); experimentally, it outperforms DeltaNet, which writes only once per token, on both state tracking and language modeling.

Chunyuan put it more plainly: the theory looks complicated, but it is actually easy to understand. As long as the keys used in the successive writes are mutually orthogonal, or at least not fully collinear, each write goes in a different direction, which is like being able to erase several times. So doing a few more products at the linear-attention step, i.e., looping at the position of the delta rule, gives stronger control over the recurrent memory.

But do keys “collide” in practice, i.e., are the keys of successive writes nearly collinear? The team checked: the larger the head dim, the harder it is for keys to collide, and the smaller, the easier. According to the statistics in the slides, at default settings such as a head dim of 128 or 256, the absolute cosine between the keys of two adjacent loops in GDN averages only about 0.05, so collisions are rare; when the head dim drops to 32 and the number of loops rises to 16, collisions become much more frequent. The gain in expressiveness from rank 1 to rank n_h is therefore fairly reliable.

Note: In high-dimensional space, two vectors with random directions are naturally close to orthogonal. The cosine of the angle between two random d-dimensional unit vectors has a mean of 0 and a standard deviation of about 1/√d, so it typically deviates from 0 by only about 1/√d, roughly 0.09 for d = 128. The closer the cosine is to 0, the closer the two vectors are to orthogonal. So the larger the head dim, the less likely keys are to collide.

There is a subtlety here, though. When a GDN or KDA is looped directly, each loop maintains its own recurrent memory: loop 1 writes its own KVs into its own memory, loop 2 writes into another one, and so on. The theory above is about repeatedly erasing and writing the same memory, whereas direct looping writes to different memories. Chunyuan admitted that applying the theory this way is “actually not rigorous.”

Seen from another angle, the recurrence over time (updating the memory token by token) and the recurrence over depth (updating representations loop by loop) could be treated as a single whole. Arrange all tokens and all loops in a “token × loop” grid: unrolled step by step, it is a single data stream, and training with teacher forcing (feeding in the whole ground-truth text at once and computing all positions in parallel) amounts to computing the whole grid at once. With direct looping, the two dimensions run separately and have never been designed jointly. Chunyuan thinks there may be a good opportunity for co-design here, which leads to the new method in the next section.

3. New work: Historical KV Replay

This section covers the team’s recent work, which is not yet in the public version of the paper; Chunyuan said they may update the paper later.

How it works: rewrite the old KVs when writing new memory

Take a single token as an example: going up the network, it passes through loops 1, 2, 3 and 4 in turn. Normally, each loop writes only the KVs computed in that loop into its own memory, and the KVs of different loops have nothing to do with each other. Replay works like this: when loop k writes its memory, it also brings over the KVs of the earlier loops and writes them again. For example, loop 2 brings over loop 1’s KVs and writes them once more; loop 3 rewrites the KVs of loops 2 and 1; and so on. This creates a form of co-design between the recurrence over depth and the recurrence over time.

Chunyuan offered an analogy: you read the first book, then the second; after finishing the second, you flip back through the first because you are afraid you have forgotten it; after finishing the third, you review the second and the first.

From the perspective of the previous section, replay makes loop k’s memory receive, at the same token position, erase-and-write operations with keys from multiple loops. In other words, it “writes to the same memory several times”: each step’s update goes from rank 1 to rank at most k. Replay introduces no new parameters and does not change the memory size. According to the formulas in the slides, the forget-gate decay is applied only once per token, and each write uses the write strength β of the loop it comes from.

[image: Chunyuan’s slides, p. 7]

Figure: The number j in each cell denotes the KVs that the token computed in loop j; loop k writes its own KVs first and loop 1’s KVs last.

Order is crucial: replay in reverse

Replay in reverse order (4→3→2→1), not in forward order (1→2→3→4). For loop 4, this means writing its own KVs first, then those of loops 3, 2 and 1 in turn. In terms of the reading analogy: after finishing a new book, don’t start rereading from the first book; learn the new one first, then work backwards to review the older ones.

In experiments, forward order brings almost no gain, while reverse order substantially improves recall and also brings large gains in pretraining language modeling. A comparison in the slides (1.3B model, 4 loops) makes the point clearly. Both orders write exactly the same KVs the same number of times; only the order differs. On a recall test at length 2k, looped GDN without replay scores 34.58, forward replay 34.71 and reverse replay 41.10; the averages over 8 zero-shot tasks are 58.39, 58.45 and 61.10, respectively. Chunyuan noted that the improvement is especially pronounced on recall tasks close to real-world use, such as “needle in a haystack.” He also offered a way to think about it: the more loops there are, the more representations replay accumulates along depth, somewhat like a latent chain of thought (latent CoT) unrolled along depth.

Note: The explanation Chunyuan gives in his slides is “the last write wins”: each write erases part of the old content along its own key direction, so the KV pair written last is overwritten the least. In forward order, the last write is the current loop’s own KVs, so the final state of the memory is almost the same as without replay; in reverse order, the last write is loop 1’s KVs, which are therefore preserved most completely in every loop’s memory. A test-time intervention experiment supports this. For a 1.3B model trained with Full Replay, changing only how replay is done at inference, switching to forward order lowers 2k recall by 7.7 points and dropping the loop-1 pair lowers it by 8.5 points, while dropping the previous loop’s pair lowers it by only 0.8 points. The model clearly relies mainly on loop 1’s KVs, which are written last.

Cost: quadratic growth, but few loops

A natural question is whether replay is expensive. Each replay means one more delta-rule erase-and-write. With n loops, the total number of writes is 1 + 2 + … + n = n(n+1)/2, which grows quadratically with the number of loops. Fortunately, people do not loop very many times today (“nobody loops ten thousand times”), usually just two or three, and in that range the extra cost is not large.

There is also a small engineering trick. Replay forms a small set of KVs along the depth dimension, so instead of doing several rounds of reads and writes one after another, you can stack them into a small matrix and run a chunkwise computation along depth (chunkwise normally splits along the sequence length; here it splits along depth). This lets you reuse existing GDN kernels and keeps the decoding cost down. According to the slides, during decoding each loop still reads and writes the state only once; what is added is computation, not state reads and writes.

Chunyuan presented the following cost comparison. Compared with a looped model that is also based on linear attention but does no replay, Full Replay has lower training throughput and slightly slower decoding because of the extra replay and writes, but overall inference remains acceptable. The specific numbers (0.6B model, a single H100): training throughput at 4k length drops from 41.2k to 34.6k tokens per second (about 16% lower), and decoding speed at 32k context drops from 19.4 to 18.4 tokens per second (about 5% lower). The slowdown here is the extra cost of replay relative to plain looping; compared with a looped full-attention model, linear attention is still far cheaper at long context lengths.

A cheaper variant: First-loop Replay

Instead of replaying the whole history, this variant replays only the first loop’s KVs: each loop no longer writes its own KVs but writes loop 1’s KVs instead, so the number of writes per token is the same as with plain looping. This is somewhat like the idea behind YOCO [5]: early depths provide the KVs for later depths.

Note: YOCO (You Only Cache Once) is a decoder-decoder architecture: the first half of the network produces a single global KV cache, which all layers in the second half share through cross-attention, greatly reducing the size of the KV cache.

One key detail: only the earlier loops’ KVs are reused, not their decay coefficients (α), write strengths (β) or queries; each loop still uses its own α, β and query. If α and β are reused as well, results get much worse. Chunyuan said this has a bit of a “hybrid” feel: it uses some historical information while keeping each loop’s own information. It is a good balance point they found: training and decoding speeds are basically unaffected, and it still captures part of replay’s gains. The numbers in the slides: training throughput drops by only about 1%, decoding is actually about 4% faster, and the gain in average score is about 70% of Full Replay’s (+1.94 vs. +2.78 on the 0.6B model).

In the ablations, they compared forward order, reverse order, replaying only the first loop, and several other combinations and orders. Taking Full Replay’s improvement as 100% (0.6B model), forward order captures only 2%, “current loop + previous loop” 16%, “current loop + loop 1” 84%, and First-loop Replay 70%. The key, then, is to keep loop 1’s KVs and write them last. Based on these ablations, the recommended approach is to keep the first loop’s KVs while letting each later loop use its own write strength and decay, capturing the gains at a lower cost. After recent discussions, however, the team also feels that First-loop Replay is sometimes not stable enough. In the talk, Chunyuan recommended Full Replay instead: it costs a bit more, but it does work better.

The gains are not just from “computing more”

To rule out the explanation that replay simply spends more compute, they compared replay with other ways of adding compute: looping more times (e.g., 6 times), widening GDN, and, while looping the whole model, stacking the DeltaProduct-style repeated erase-and-write from the previous section inside each layer. All of these help, and more compute generally brings more gain, but the improvements may all fall short of replay. In the comparison plot in the slides (0.6B model), these approaches raise the average score only from 55.7 to somewhere between 56 and 57, and both their training and decoding speeds are lower than First-loop Replay’s, whereas First-loop Replay and Full Replay reach 57.7 and 58.5, respectively. Chunyuan concludes that replay’s gains cannot be explained simply by compute; moreover, some of the other approaches run into system-level problems and are less efficient.

Finally, Chunyuan added that what he showed is the pure linear-attention setting. The more realistic setting is a hybrid one, and replay works on hybrid models as well, with good results: the slides show that on 0.6B Full+GDN and GDN+DSA hybrids, Full Replay raises the average score by 1.4 and 2.0 and 2k recall by 3.1 and 4.3, respectively. In his view, combining loops with linear attention is always a trade-off between speed and quality: wherever a looped Transformer would be used, you can fairly freely swap in a looped GDN and get a good balance between speed and quality.

4. Other findings from the paper

Besides replay, Chunyuan picked out a few points from the published paper [1] that are worth sharing. The paper itself tries many hybrid combinations of linear attention, sparse attention, SWA and more, with fairly thorough ablations.

Note: LT2 studies two looped variants, one with linear attention (LT2-linear) and one with sparse attention (LT2-sparse). For the former, looping repeatedly refines the memory; for the latter, it expands the effective receptive field. The receptive field is how wide a range of inputs can influence the output at a given position; in a language model, it is how far back a token can draw information from. Take sliding window attention as an example. In each round, every token sees only the most recent w tokens; in the next loop, those w tokens have already absorbed the information in their own windows during the previous round, so earlier content can reach the current token indirectly through them. Each additional loop therefore extends the coverage back by one window, to about n·w tokens after n loops, the same coverage as stacking n layers of sliding window attention. “Effective” refers precisely to this indirect coverage relayed through intermediate tokens. The paper also proposes two hybrids: one combining GDN with the sparse attention DSA, which runs in linear time throughout and matches the quality of a standard looped Transformer; and one combining GDN with a small number of full-attention layers, which surpasses the standard looped baseline.

Loop-level hybrids

When building hybrids, the first idea that comes to mind is mixing along depth, e.g., one full-attention layer for every three linear-attention layers. With loops, you can also mix across loops, which gives a cross-loop hybrid: for example, the first loop uses full attention throughout and the second switches to SWA.

This works because a loop shares only the parameters, which Chunyuan calls the slow weights; anything other than the parameters can change from loop to loop. The attention mask is a typical fast weight, and if you loop the whole model, even the loss function can differ across loops. This opens up a new design space: besides depth-level hybrids, there are loop-level hybrids.

The team compared two schemes that change the attention window size from loop to loop, and the results were somewhat surprising:

  • Shrinking the window loop by loop (like a CNN, e.g., full attention → 512 → 256 → 128): lower loss (perplexity 9.26), but downstream performance does not improve (average score 60.71).
  • Growing the window loop by loop (e.g., 128 → 256 → 512 → full attention): slightly worse loss (perplexity 9.42), but downstream performance is actually a little higher (average score 62.47).

For reference, a conventional depth-level hybrid (with full attention interleaved within every loop) has a perplexity of 9.30 and an average score of 61.51. Lower loss usually goes with better downstream performance, but these results go the other way.

Randomly sampled attention patterns

If you treat each position along the depth and loop dimensions as a “slot,” you can choose independently whether each slot uses full attention or efficient attention. A more aggressive approach is to randomly sample the attention type of each slot during training: the overall ratio of full to efficient attention stays fixed, but their positions vary randomly. At inference, majority voting is done over multiple sampled configurations.

The method does bring some improvement, but at too high a cost: at inference, each sampled configuration has to be run separately before voting, so the compute grows in proportion to the number of samples (in the slides, 5 samples at 5× the compute raise the average score only from 61.51 to 61.75). Chunyuan considers it impractical for real use; this set of experiments was only an exploratory attempt.

5. Distilling from Ouro: no need to train from scratch

The last part of the talk was about distillation: converting a full-attention looped model such as Ouro [6] into a linear-attention hybrid model without training from scratch. Ouro is an open-source family of whole-model looped language models, which part two of this series introduces in detail. Chunyuan noted that switching to sparse attention is relatively easy, while switching to linear attention is bound to cost some performance; because this work leans toward linear attention, they tried it anyway.

Note: According to the LT2 paper, the Ouro-hybrid-1.4B obtained this way (called Ouro-LT2-1.4B in the slides) was trained on only about 1B tokens, yet it outperforms industry 1B-class models and is competitive with 4B-class models. According to the slides, the teacher is Ouro-1.4B, which loops 4 times and whose 24 layers all use full attention; the student keeps full attention in only 6 of those layers and replaces the other 18 with GDN plus replay. Training has three stages: first, aligning each layer’s hidden states and selecting the 6 layers to keep (100M tokens, length 512); then, per-loop logit distillation (600M tokens, length 4k); and finally, 32k long-context training on OpenThoughts-v3 (600M tokens). The final student reaches about 95.5% of the teacher’s average performance across 6 tasks. The contribution of each component is also clear: without replay, interleaving the 6 full-attention layers uniformly at fixed intervals retains only 65.3%, the best previous layer-selection method reaches 86.9%, and their method reaches 92.9%; adding Full Replay raises this to 95.5%.

Chunyuan shared three key points.

First, distillation data is crucial: different data maps to different downstream abilities. The data you distill on tends to determine which benchmarks improve: pretraining data corresponds to commonsense benchmarks, while open-source reasoning data may line up with math-reasoning benchmarks such as GSM8K and AIME. As for MMLU, they have not yet found data that lines up with it: “maybe our experiments weren’t thorough enough, but MMLU may also just be harder.”

Second, choosing which layers keep full attention is crucial. This point comes from earlier work [7]: when distilling into a hybrid, you should not interleave linear-attention layers uniformly at a fixed ratio. Instead, first assess the importance of each layer, then decide which layers keep full attention (that work estimates each layer’s importance with a short training run on generic text). This step substantially improves distillation. According to Chunyuan, one usually finds that the early-to-middle layers should keep full attention, while the other layers can be replaced.

Third, a loop-specific trick: a dual curriculum over depth and length. A curriculum arranges training from easy to hard. Long-context training usually increases the length gradually from short to long; with loops, you can likewise go from shallow to deep, first distilling the first loop until it closely matches the teacher, then the second loop. Their preferred approach is to advance both together: loop 1 paired with short contexts, loop 2 with longer ones. In their method, this works well.

6. Future directions

For depth-recurrent models, Chunyuan sees three directions worth watching:

  • Extrapolation matters more than tinkering with the architecture. In his view, continuing to tweak architectures is not that critical; what really matters is extrapolation. Without extrapolation, one has to work very hard to justify the value of weight sharing. He explained this later in the panel discussion: a loop has only two properties, depth and weight sharing, and if there is no extrapolation and the comparison is against a model of the same depth, only weight sharing remains (see part three of this series). Conversely, with extrapolation, weight sharing provides value that a standard Transformer cannot: loop only a few times in training, then loop more on hard problems at inference, trading extra compute for better results without adding parameters or producing a longer chain of thought. Benhao also said in the panel that this kind of extrapolation is what test-time scaling really means (see part two of this series).
  • ACT is very hard to implement. He is not sure whether this problem can be solved. The main difficulty is at the infra level: different tokens loop different numbers of times, which makes batching and training systems hard to design. Even so, he still lists ACT as a direction worth watching.
  • Pay more attention to post-training. Not just SFT and RL; combining on-policy distillation [8] with loops in particular may work well.

Note: On-policy distillation means the student model receives token-by-token guidance from the teacher on sequences it generates itself, rather than only learning from text the teacher wrote in advance. This way, the student sees and corrects the mistakes it would actually make during training; a representative work is [8].

Extrapolation and ACT are discussed in more depth in the second half of the panel; see part three of this series.

Closing and what’s next

Before the Q&A, host Wenyue Hua remarked that the work seems to start from some findings about linear attention and then discover that adding loops is a very natural combination. Looking back at Chunyuan’s talk, the reasoning follows a single thread: the KV cache cost that looping brings is resolved with linear attention, and joint time-and-depth designs such as replay substantially improve linear attention’s weakness in recall.

Next up is the first half of the panel: Where are the bottlenecks in LLM architecture? Do the gains of looping really come from the architecture, or just from spending more compute? Should looped models learn fixed points?

References

[1] LT2: Linear-Time Looped Transformers. arXiv:2605.20670, 2026. Code: github.com/chili-lab/LT2

[2] Gated Delta Networks: Improving Mamba2 with Delta Rule (GDN). ICLR 2025, arXiv:2412.06464.

[3] Kimi Linear: An Expressive, Efficient Attention Architecture (KDA). arXiv:2510.26692, 2025.

[4] DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products. NeurIPS 2025, arXiv:2502.10297.

[5] You Only Cache Once: Decoder-Decoder Architectures for Language Models (YOCO). NeurIPS 2024, arXiv:2405.05254.

[6] Scaling Latent Reasoning via Looped Language Models (Ouro). arXiv:2510.25741, 2025.

[7] Distilling to Hybrid Attention Models via KL-Guided Layer Selection. ICLR 2026, arXiv:2512.20569.

[8] On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024, arXiv:2306.13649.