On the afternoon of October 17, NICE held an in-person event in Shenzhen, “Agentic Data: From Computer Use to Robot Use.” The event covered a lot of ground, so the recap is being published in several parts. This is the first.

This session was given by Yiming Du, who holds a Ph.D. from The Chinese University of Hong Kong and is a senior researcher at Huawei Hong Kong Research Center. His talk was titled “LegoX: Data Science and Reinforcement Learning in Practice for Coding Agents.” Training a coding agent requires real tasks that can be executed and verified, trajectories that the model leaves as it interacts with an environment step by step, and training and evaluation that quickly report back how good the data is. In Du’s view, this kind of Agentic Data is becoming new industry infrastructure.

LegoX is a series of open-source projects built around this chain by the team Du is part of. The talk was organized around data, focusing on LegoFlow, the team’s framework for producing coding data, and several lessons learned while building that data. It closed with a brief introduction to Lego-RL, the team’s reinforcement learning framework, to show how the data performs in RL training.

Key takeaways

  • Coding agent = model + harness. Training one relies mainly on two kinds of data: executable, verifiable task instances, used more for reinforcement learning, and trajectories the model leaves while interacting with real environments, used more for SFT. Both depend on the harness. A trajectory records every turn in which the model calls tools through the harness and gets the results back; in RL, the model also solves tasks in the task environment through the harness and is then scored by tests.
  • LegoFlow divides data production among several agent roles. Curator builds tasks, Tracer generates trajectories, Trainer and Evaluator handle training and evaluation, and Root coordinates them. Every step can be launched with a single command through a built-in skill.
  • Small, carefully selected data. From about 12 million candidate PRs, the team selected 5,000 SWE tasks and open-sourced them. With SFT on only 1K LegoFlow-SWE trajectories, Qwen3.5-35B-A3B reaches 70.2 on SWE-bench Verified, 6.8 points above the official Instruct version.
  • Letting an agent run the whole pipeline on its own. Starting from scratch, an agent crawled repositories, generated trajectories, trained, and evaluated, then fixed the data based on what the evaluation revealed and trained a second time. SWE-bench Verified rose from the base model’s 7.6 to 64.4.
  • Four lessons on building data: task complexity and the number of turns quickly predict data quality; the stronger the model, the more it cheats, and with internet access the confirmed cheating rate of GLM-5.2 reaches 21.2%; when selecting trajectories, look at how deeply the model reasons in each turn, not how long the whole trajectory is; with limited compute, add interaction turns first, then more rollouts.
  • Lego-RL runs RL inside real harnesses. The harness runs unmodified. The tokens the model actually generates are recorded during sampling and fed back as-is to compute gradients during training, and MoE models additionally get routing replay. After RL, SWE-bench Verified improves by 5.8–9.4 percentage points across three harnesses.

Quick glossary

The table below explains two terms that come up repeatedly. Readers already familiar with them can skip it.

Term Plain-language explanation
Training-inference consistency In RL training, the token sequence the trainer uses and the probabilities it computes must match what the model actually generated during sampling; otherwise the gradient estimate is biased.
MoE routing replay An MoE model sends each token to only a few “experts.” Sampling and training are two separate computations and may select different experts. Routing replay records the selections made during sampling and reuses them during training.

About the speaker

Yiming Du (speaker): Ph.D., The Chinese University of Hong Kong; senior researcher at Huawei Hong Kong Research Center. The LegoX projects presented in this talk come from the team he is part of and cover data and environment pipelines, SFT training, evaluation, and reinforcement learning for coding agents.

1. Four years of change: Agentic Data is becoming infrastructure

Du opened with word-frequency statistics covering the past four years. The sources were the official release announcements and technical blogs of frontier model developers outside China, including OpenAI, Anthropic, Google DeepMind, xAI, Mistral, and Meta (2023 through the first half of 2026). The analysis tracked the most frequent topic words in each year:

  • 2023: building foundational capabilities. Everyone was scaling, model and data sizes kept growing, and “Scaling” was the top word of the year.
  • 2024: reasoning and multimodality. “Reasoning” and “Multimodal” took the top two spots.
  • 2025: agents and the tool ecosystem. Agents began to enter everyday work. “Coding,” “Agent,” and “Tool Use” all made the top five, and the theme shifted from answering questions to completing tasks.
  • First half of 2026: real-world execution. “Agent” and “Coding” took the top two spots, and “Long Horizon,” “Reliability,” and “Real-world Tasks” entered the top ten: models now need to complete longer-horizon tasks in real environments, and do so more reliably.

Throughout this period, data has become more and more important. Du divided the evolution of training data into four layers, which show how Agentic Data has step by step become industry infrastructure:

  1. Data supply: web pages (such as Wikipedia), open-source code repositories, and human feedback data are used to train foundation models, and instruction tuning is added to strengthen question answering.
  2. Specialized data: code, math, and domain knowledge in fields such as finance and law are used in mid-training to improve the model’s foundational abilities in specialized domains.
  3. Agent trajectories: for models to take part in real work, the data has to cover terminals, browsers, and code repositories, with agents interacting with external environments and leaving trajectories. Post-training methods such as SFT then improve the model’s ability to solve problems in real scenarios.
  4. Reproducible systems: the end goal is a reproducible system. Whether tasks are truly executable, whether external environments can be reproduced, how the data performs in training, and whether evaluation is accurate are all problems that Agentic Data runs into when it enters the training process.

2. LegoX at a glance

LegoX builds on real-world data sources such as GitHub and Stack Overflow and explores data and environment pipelines, SFT training, evaluation, and reinforcement learning, with the goal of assembling, piece by piece, an open-source data infrastructure for coding agents. The pieces released so far:

Project What it does Progress
SWE-Lego Large-scale real SWE tasks and trajectories, with all data open-sourced 200K+ downloads on Hugging Face
SWE-Review Builds real development feedback to improve code collaboration intelligence Data pipeline efficiency up 50%
Terminal-Lego Interactive terminal environments that generate long-horizon agent execution experience 80K+ dataset downloads on Hugging Face
Lego-RL A code RL framework that supports mixed training across multiple harnesses and connects data, training, and evaluation RL training-inference alignment of 0.999+
LegoFlow An easy-to-use data engineering framework for coding agents that automates the whole pipeline across tasks, trajectories, training, and evaluation, supporting a closed loop of data production and self-iteration Newly open-sourced 5K curated SWE tasks and trajectories

This talk covered two of them: mainly LegoFlow, followed by a brief introduction to Lego-RL, looking at how code reinforcement learning verifies that the data is effective.

3. Coding agent = model + harness

First, how a coding agent works. Du summarized it as “model + harness.” The user gives a task, such as “implement a new function in src/utils.py and make the tests pass.” The model understands the goal, plans the steps, and generates concrete actions, which it outputs as structured instructions such as read_file, edit_file, and run_command.

These instructions go to the harness (also called a scaffold). The harness provides the tool interface and executes the instructions on the environment side, reading and writing files and running commands on the file system of the working directory. It then returns the execution results and file state and appends them to the context. The model adjusts its strategy accordingly, which completes one turn of interaction. Real tasks often take many turns, and the trajectory left by these turns is exactly the form of data that training needs.

Different harnesses expose their tool interfaces in different forms, and this difference directly affects training.

Why traditional data pipelines are expensive

A traditional data pipeline for coding agents has three steps:

  1. Repository collection and image building: extract installation and test commands from key files in open-source repositories, build images, and verify them in a real environment to obtain a usable execution environment.
  2. SWE task construction: there are two routes. One uses real PRs, which are limited in number but authentic. The other is bug injection, which creates bugs by having an LLM rewrite code or by applying AST transformations (parsing the code into an abstract syntax tree and modifying it by rule, such as changing “>” to “>=”), and can generate tasks at scale. Every task must pass test validation: the reference patch must make the originally failing tests pass (FAIL-TO-PASS) without breaking the tests that originally passed (PASS-TO-PASS). Tasks that pass validation become usable task instances.
  3. Trajectory generation, filtering, and scoring: run agents on these tasks to produce trajectories, then filter and score them.

This yields two forms of data. Task instances are used more in RL training; Xiaomi’s MiMo RL training, for example, is built on a set of high-quality task instances. Trajectories are used more for SFT.

The problem is that such a pipeline is complex, full of details, and expensive to develop. At many companies in the industry, each department runs its own data pipeline. The more departments, the more pipelines, and the cost and the details to handle add up quickly. This is exactly the problem LegoFlow aims to solve: turn the pipeline into an open-source framework, so that not everyone has to build it from scratch.

4. LegoFlow: dividing work among agents the way you would run a team

LegoFlow is an open-source, easy-to-use project for producing coding data. Building tasks, generating trajectories, training, and evaluation can each be launched with a single command through a built-in skill. Structurally, a coordinating node called Root schedules four modules called Blocks. Each Block does one kind of work and has its own output:

  • Curator: synthesizes tasks; outputs Tasks
  • Tracer: generates trajectories; outputs Trajectories
  • Trainer: trains; outputs Models
  • Evaluator: evaluates; outputs Scores

Design principle: divide the work the way a team does. Du said LegoFlow models the relationship between individual developers and team collaboration in real software development. A team has its own goals, each member has a role, and coordinating everyone has traditionally been a team-management problem. With coding agents, these roles can be filled by agents. So each Block corresponds to one role and one type of task, and its environments, logs, inputs, and outputs are all archived. Each Block’s role, behavior, inputs and outputs, and interaction boundaries are specified in a config.yaml file. In his view, clearly defined boundaries are the foundation of efficient teamwork. In addition, every code repository involved is treated as a black box and called only through agreed interfaces, so agents do not have to dig into or modify the large, complex code inside the repositories.

Root: coordinates the modules. It handles project-wide interaction with the user, installs and checks the environment, breaks the user’s goal into parts for different Blocks, and coordinates task assignment and resource allocation. Execution relies on plugin-style skills. For example, tell Root “I want to synthesize a batch of data,” and it calls the corresponding Curator skill. Each Block has skills such as setup, check, and run; running Curator’s run skill automatically executes the entire data pipeline and synthesizes tasks.

Curator: builds tasks. This is also what many data vendors care about:

  1. Collect high-quality repositories and PRs from GitHub. “High quality” has several dimensions: repositories can be judged by star counts and community activity, and the quality of PRs can be scored by an LLM or measured in other ways.
  2. Extract the issue from each merged PR, and split the change into a fix patch and a test patch for later validation and evaluation.
  3. Build an isolated Harbor environment that packages the entire task.
  4. Score and filter for quality with rubrics.
  5. Deliver tasks that pass a dual acceptance check: the unfixed code must fail the tests, and the tests must pass once the fix patch is applied.

The parameters for these steps are all set in Curator’s own config.yaml, such as which languages’ PRs to collect (Python, JavaScript, and so on), how many repositories to collect, and the maximum number of PRs per repository. Combined with plugin skills that run it with a single command, this greatly lowers the cost of what used to be a complex data synthesis pipeline. Curator’s skills include setup (one-time initialization), check (a pre-flight dry run), collect-prs (collect and filter PRs), create-tasks (generate Harbor tasks from PRs), and dashboard (score task difficulty and add labels), which Root calls as needed.

Note: Harbor was developed by the Terminal-Bench team and released together with Terminal-Bench 2.0. It runs and evaluates agents at scale in container environments and also supports SFT and RL data pipelines.

Tracer: generates trajectories. Once the tasks are ready, the next step is to obtain high-quality trajectories. Trajectories are the direct training data for SFT, and the trajectories a model is trained on directly determine what it can learn, which is why trajectories matter so much now. Tracer’s features:

  • It supports not only black-box harnesses such as Claude Code but also OpenCode, OpenHands, and others, so trajectories can be generated in many harnesses;
  • It adds a proxy layer between the harness and the model API that records the complete input and output of every turn, including the reasoning;
  • It executes and scores in real harnesses, then applies rubric-based quality filtering;
  • Anti-cheating: network access is cut off and the git history is cleared.

The final output is trajectories that can be used directly for training. Tracer’s command parameters are likewise set in config.yaml. Every Block is managed through a config.yaml in the same unified format, much as a team uses one template to assign everyone’s work.

Trainer and Evaluator: is this data actually usable? Trainer converts the data into LLaMA-Factory format, launches SFT training with a single command, and tunes the training parameters. Evaluator runs benchmark evaluations and reports the results back. Analyzing the results shows how good the data is and which direction the next iteration should take. The deliverables are a model and an evaluation report, which inform the next round of data synthesis.

Much current work on RSI (recursive self-improvement) aims to let a system improve itself. A data synthesis pipeline can do the same; the key is to make every step sufficiently precise.

Run dashboard: using conversion rates to find bottlenecks. From building tasks and generating trajectories to training and validation, the whole pipeline takes a long time and a lot of compute. The conversion rate is the share of items each stage keeps relative to the previous stage, and it directly affects the cost of each usable data point: the higher the conversion rate, the lower the unit cost and the higher the production efficiency. The team therefore monitors the conversion rate of each stage on a live dashboard. Here is a snapshot of one real Curator run: 822 repositories and 8,818 candidate PRs yielded 926 built tasks, of which 270 passed validation, an acceptance rate of 29.2%. For tasks that fail, the team traces back through the environment and tests according to the stage where they failed, fixes them, and runs acceptance again. Tasks that pass go to Tracer for trajectory generation.

5. LegoFlow-SWE: from 12 million PRs to 5,000 tasks

Using this pipeline, the team open-sourced a new set of SWE tasks and trajectories in September, LegoFlow-SWE:

  • Starting from more than 700,000 GitHub repositories, they obtained about 12 million candidate PRs;
  • PR filtering kept about 600,000 verifiable PRs, or 5.0%;
  • Task validation then left 5,000 tasks, a retention rate of only 0.0417% counted from the candidate PRs;
  • Using GLM-5.2 in two harnesses, OpenHands SDK and OpenCode, they generated 9,767 trajectories on these 5,000 tasks, of which 2,780 solved the task, a resolve rate of 28.46%.

With SFT on Qwen3.5-35B-A3B using only 1K LegoFlow-SWE trajectories, the model reaches 70.2, 48.8, and 57.0 on SWE-bench Verified, Pro, and Multilingual, respectively, which is 6.8, 10.6, and 5.3 points above the official Qwen3.5-35B-A3B-Instruct. It also outperforms SFT on trajectories that the same teacher model produced on SWE-rebench-v2 tasks (64.4 on Verified) and SFT on several external open-source data pools (StepFun 3.5, DeNovoSWE, Scale-SWE), reaching open-source SOTA for this base model. From standard SWE-bench to the multilingual benchmark, the results compare favorably across the board.

Note: SWE-rebench V2 is a set of SWE training tasks released by Nebius in March 2026. It contains more than 32,000 executable tasks with Docker environments, covering 20 programming languages.

Letting an agent run the entire pipeline on its own

This is not a one-off process. After training and evaluation, the results go back to Curator to plan the next round of data synthesis, which is how the overall quality of the data keeps improving.

The team also tried having an agent complete the entire pipeline end to end: crawling GitHub repositories from scratch, generating trajectories, training, evaluating, and then training a second time on the corrected data. The experiment used the base model Qwen3.5-35B-A3B-Base, which is weak on its own and scores only 7.6 on SWE-bench Verified:

  • After the first training run (data selected for coverage, 500 self-generated trajectories), it reached 56.1;
  • The first evaluation showed that the data needed deduplication, filtering of shallow reasoning, and standardized tool calls. After these fixes, the median reasoning length per turn grew from 140 to 959 characters, and the CoT quality improved;
  • After the second training run (data selected for reasoning depth, 512 trajectories), it reached 64.4, 8.3 points higher than the first run.

6. Four lessons from building SWE data

Lesson 1: Two metrics for judging data quality quickly

Task complexity correlates positively with training scores, which makes it an efficient way to judge task quality. The team computes it as a weighted score of five static signals: patch scope (30%), logic complexity (25%), context span (20%), test complexity (15%), and problem-statement complexity (10%). For comparison, LegoFlow-SWE has a complexity score of 6.27 and SWE-rebench V2 has 5.80; models trained with SFT on them reach 70.2 and 64.4 on SWE-bench Verified, respectively.

The number of turns is strongly correlated with the pass rate, with a Pearson correlation coefficient of −0.95: the more turns, the lower the resolve rate. Problems that are genuinely easy tend to be solved within a few turns.

Lesson 2: The stronger the model, the more it “cheats”

Peeking at git history and looking up answers online: the stronger the model, the better it is at both. According to the team’s observations, GLM-5.2, DeepSeek V4.0, and later models all show this behavior. In environments with internet access, the confirmed cheating rate rose from 3.6% for GLM-5 to 21.2% for GLM-5.2. Once the leakage was removed, both the teacher and the student scores dropped. The GLM-5.2 teacher fell from 86.8 to 78.8 on SWE-bench Verified, and the student model trained with SFT on 1K of its trajectories on SWE-rebench-v2 tasks fell from 74.4 to 64.4. Those 10 points came from cheating made possible by the leakage, not from the model’s real ability.

Du shared an example the team ran into recently while validating data. They had already deleted the git commits from the repository history, yet the model still cheated. The reason: after the git history was deleted, the original repository’s code still existed locally in the form of binary pack files, and the model, while hacking, unpacked those files and found the answer.

Note: (1) Git compresses historical objects into pack files stored under .git/objects/pack. Deleting only branches, tags, or other references to commits often leaves these objects on disk, where they can be recovered.

(2) Similar loopholes have appeared in public benchmarks. In September 2025, SWE-bench GitHub issue #465 reported that some agents used commands such as git log –all to view “future” commits in the evaluation repository, gaining hints for the fix or even seeing the reference patch; the affected submissions involved models including Claude 4 Sonnet, GLM-4.5, and Qwen3-Coder. The maintainers then updated the SWE-bench Verified images and removed the git history after the base commit.

Lesson 3: Filtering out shallow reasoning beats chasing the longest trajectories

While selecting data, the team found that as training performance improved, the average reasoning length per turn also increased. So the goal is deeper reasoning, not a pile of turns built on shallow reasoning.

In one experiment, the team split the same pool of GLM-5.2 / SWE-rebench-v2 trajectories into four groups by average reasoning tokens per turn and trained on each group separately. The four groups averaged 167, 368, 577, and 812 reasoning tokens per turn, and models trained on them with SFT scored 57.0%, 64.0%, 64.6%, and 65.0% on SWE-bench Verified. Going from the shallowest group to the second gained 7 points, while the last three groups differed by only 1 point. The gain therefore comes mainly from removing the shallowest group; selecting for even deeper reasoning adds little. The team’s approach is to remove shallow trajectories first, then spend the data budget on covering more tasks.

Lesson 4: For test-time scaling, add turns first, then rollouts

Given a compute budget, how can the pass rate be pushed higher? The team’s experience is to increase the number of interaction turns first, then the number of rollouts.

The team compared two ways to scale: increasing the number of interaction turns (sequential scaling) and increasing the number of parallel rollouts. With a single rollout, raising the number of turns from 20 to 100 lifts the resolve rate from 2.0% to 41.4%; with turns held at 20, going from 1 to 8 rollouts only reaches 6.0%. Beyond about 100 turns, more turns barely help: with a single rollout, going from 100 to 160 turns raises the resolve rate only from 41.4% to 43.8%. At that point, adding rollouts pays off more, reaching up to 47.4%.

7. Lego-RL: reinforcement learning inside real harnesses

Finally, Du briefly presented how this data performs in real RL training. Lego-RL is the team’s agentic RL training framework and part of LegoFlow, and it likewise supports Block skill plugins and a training dashboard. It supports training-inference consistency, fully asynchronous training, and mixed-harness training, with algorithms including SAO, GSPO, GRPO, and PPO. Supported training backends include FSDP, VeOmni, and Megatron; supported harnesses include OpenHands SDK, Claude Code, and OpenCode; and it supports multiple programming languages.

Why training-inference consistency matters so much

For agentic RL, the tokens obtained during rollout must line up exactly with the tokens at each position during training. The most error-prone point is that harnesses rewrite the context in their own ways, and each harness does it differently. For example, some harnesses remove failed tool calls and pass the pruned context back to the trainer as message history; others normalize tool arguments. Each such step introduces a mismatch. They fall into three typical cases:

  • Tool arguments are re-serialized: the id and name of the tool call stay the same, but the arguments are converted back and forth between a JSON string and a dict, so key order, default values, and blank lines can all change;
  • Reminders are inserted into the middle of the history: the harness injects system reminders into the message history;
  • Subagents share one session with the main thread: a subagent’s context is spliced back into the main context.

Two common, simple workarounds both have limitations. The first is to build a simplified harness into the trainer. With the same Qwen3.5-35B-A3B, changing only the harness moves SWE-bench Verified anywhere between 57.2 and 64.0, so the harness alone accounts for a difference of nearly 7 points; the context built by a simplified harness during training does not match the context the real harness builds in deployment. The second is to run the real harness and replay the archived conversation afterward. This requires re-tokenizing the text, which causes token drift: the same tool arguments are 13 tokens at sampling time but 16 tokens after re-tokenization, the logprobs deviate as a result, and the importance sampling ratio keeps drifting.

Lego-RL instead runs the real harness unmodified, uses a unified proxy to capture the actual tokens and logprobs at generation time, and replays these recorded tokens during training (see the third question in the Q&A at the end for details). The difference from the second workaround is that it replays the tokens recorded at generation time, not tokens obtained by re-tokenizing the conversation text afterward. Overall, Lego-RL’s training-inference alignment metric can exceed 0.999.

Note: Training-inference mismatch is not unique to agentic RL. In 2025, Feng Yao et al. pointed out in “On the Rollout-Training Mismatch in Modern RL Systems” that even with identical weights, the inference engine (such as vLLM) and the training engine (such as FSDP) can compute noticeably different probabilities for the same token, which effectively turns on-policy RL into off-policy RL; they proposed truncated importance sampling (TIS) as a correction. Agentic RL adds new problems on top of this, namely harnesses rewriting the context and inconsistent MoE routing, which are the cases discussed in this section.

MoE routing must be consistent too

For MoE models, if the sampling and training forward passes select different experts, a mismatch arises inherently. For example, the same token “def” is routed to experts {3, 17} during vLLM inference but to experts {3, 41} in the training forward pass. Lego-RL records the expert routing during rollout and reproduces the same routing in the training forward pass. With routing replay, the training-inference correlation coefficient rose from 0.9946 to 0.9993, and the token-level absolute difference in logp fell from 0.0062 to 0.0025.

Note: This practice of recording the routing during rollout and reusing it during training is known in the research community as Rollout Routing Replay (R3). It comes from a paper by Xiaomi’s MiMo team; see the References at the end.

How to select RL data

Early explorations were relatively simple: training only on task instances that the model gets “partially right” works better than skipping rollout filtering. The team ran a controlled comparison (951 samples per data pool). Starting from 36,884 samples, static filtering and selection based on rollout scores left 2,699. The “full difficulty band,” tasks solved 1 to 3 times out of 4 rollouts, achieved the highest average validation score (0.671). Taking only the upper half (solved 2–3 times) was essentially on par (0.670); taking only the lower half (solved 1–2 times) gained less (0.640); and random sampling brought no improvement: 72.7% of its tasks were always failed and 13.4% always solved, and because such tasks get the same score on every attempt, they provide no learning signal.

The drawback of this selection method is that it requires a large number of rollouts on the raw task instances, which makes inference costly.

Preventing reward hacking

A typical example: once the model has a shell, in turn 3 it uses git log to find the fix commit; in turn 4 it uses git show to view the diff, which is the reference answer; in turn 5 it copies the diff back into the source file; and in turn 6 it runs pytest, all tests pass, and it collects the full reward of 1.0, even though it never solved the task. The attack paths the team measured and the corresponding defenses:

Attack path Rate Defense in the environment and grading phases
Reading the answer from git history 4.6–20.5% Squash the repository into a single commit; restore the history at grading time
Modifying tests 2.4–19.4% Keep the tests invisible to the agent; restore the original tests before grading
Downloading the reference patch about 1.9% A privileged sidecar restricts outbound network access during the agent phase
Tampering with the grader about 2.5% Run the grader in a separate phase

Another principle is that infrastructure failures should not take part in policy updates. Trajectories produced by infrastructure problems such as timeouts or environment startup failures get a weight of 0: they are kept on record but produce no gradient. Trajectories that exceed the turn or context limit, by contrast, receive a score of 0 and are trained on as usual, because finishing the task within budget is the policy’s own responsibility.

Results

After separate training in each of the three harnesses, SWE-bench Verified (4800s setting) improved over the starting point in all of them, with the largest gain in OpenCode:

Model OpenHands SDK Claude Code OpenCode
Qwen3.5-35B-A3B-Instruct (starting point) 64.0 62.4 57.2
Qwen3.6-35B-A3B 67.4 63.4 60.6
KAT-Coder-V2.5-Dev 67.0 66.8 64.8
Lego-RL-Qwen3.5-35B-A3B 70.4 68.2 66.6
Change vs. starting point +6.4 +5.8 +9.4

Note: Each of the three Lego-RL scores comes from a model trained separately in the corresponding harness. The starting point in this table (64.0) and the Instruct baseline cited in Section 5 (63.4) come from two different evaluations with different settings.

With mixed-harness training, the reward signal also rose considerably. The team also measured how behavior changed before and after RL: the rate of re-reading a file after editing it rose from 73.6% to 98.1%; the number of files inspected before the first edit grew from 3.45 to 6.92; the rate of proactively running tests rose from 85.0% to 93.6%; and the tool-call format error rate fell from 1.07% to 0.15%. With 8 attempts per problem, pass@8 (at least one success, a measure of coverage) rose from 83.2% to 87.9%, up 4.7 percentage points, while pass^8 (all 8 attempts succeed, a measure of reliability) rose from 28.3% to 39.4%, up 11.1 percentage points, a larger gain. So the benefit of RL shows up mainly in reliability: relatively few new problems get solved, but problems the model could solve only occasionally before are now solved more consistently.

8. Q&A

Q: Two questions. First, when a model searches online, sometimes it is a legitimate need and sometimes it is cheating (for example, digging through git history for the answer). How do you tell them apart? Second, does a model trained in one harness lose performance when it is moved to another harness? How do you handle this in practice?

Yiming Du: Let me start with cheating. Our understanding of cheating behavior comes from two sources: what has been reported in published papers, and what has actually appeared in our own training. To identify it, we first look at anomalous signals, such as trajectory length or number of turns deviating clearly from the norm, and then inspect trajectories one by one to judge which behaviors are unreasonable. AI can help with this step. For example, we sample 100 trajectories and have an AI analyze them with several rounds of follow-up questions; when it finds an anomaly, we keep digging until we pin down the specific cheating method, then design safeguards accordingly and build them into the infrastructure. This is a process of continuous iteration.

Now cross-harness transfer. A model trained only on OpenHands SDK also improves on other harnesses, and we believe this comes from an improvement in the model’s own capability, namely learning which tool to use in which situation. But the transfer is not stable: train on A, and B may be slightly better while C improves only a little. So we later moved heavily to mixed-harness training, which gave fairly stable performance across all harnesses. It is also cheaper: training one model with multiple harnesses costs much less than training a separate model for each harness, which is also why many companies use multi-harness, multi-task training.

Q: How exactly do you handle the training-inference mismatch? Also, beyond Terminal-Bench, which new benchmarks are worth watching, such as coding benchmarks aimed at RSI?

Yiming Du: First, the training-inference mismatch. RL training first performs rollouts and then replays the same trajectory in the training phase. Mismatches arise mainly in three places: first, tool arguments are re-serialized, so the entire JSON string gets rewritten; second, the harness inserts content it considers necessary into the trajectory history; third, many harnesses support subagents, and a subagent’s context is spliced back into the main context. Our main approach is to use the prefix of each inference request to locate where a mismatch occurs, while recording the tokens generated at inference time. Whenever a mismatch appears, we replay with the recorded tokens, which guarantees training-inference consistency.

On benchmarks, harder RSI-related evaluations such as SWE-Marathon are worth watching. Also, our framework is not limited to coding agents: open-domain tasks that are closer to real-world scenarios, such as those in Claw-Eval, face the same kinds of problems.

Note: SWE-Marathon is an ultra-long-horizon software engineering benchmark released in 2026. It tests whether agents can keep planning and managing context and memory over hours of work to complete large tasks autonomously. Claw-Eval is a benchmark for autonomous agents on real-world tasks. It contains 300 human-verified tasks, and its primary metric is Pass^3, which requires all three independent attempts to pass.

Closing thoughts

The thread running through this talk is turning data building into engineering: tasks must be executable and verifiable; trajectories must be worth learning from, and the model must be kept from exploiting loopholes; and the results of training and evaluation in turn guide the next round of data. From filtering 12 million PRs down to 5,000 tasks, to training a 35B MoE model to open-source SOTA for that base model with 1K trajectories, to letting an agent run the whole pipeline on its own, LegoX demonstrates a way of producing data that is increasingly automated and can keep iterating.

The next recap will cover Xiaomeng Hu’s talk, “From Computer Use to Robot Use: Capability Boundaries and Data Needs of General Physical Agents,” moving from the digital world to the physical one.

References

LegoX projects and papers

  1. LegoX project homepage (with blog posts on LegoFlow, Lego-RL, and SWE-Lego)
  2. Code: LegoFlow · Lego-RL · SWE-Lego
  3. Data: LegoFlow-SWE (5,000 tasks, 9,767 trajectories) · Terminal-Lego-15k
  4. SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolving (arXiv:2601.01426)
  5. SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review (arXiv:2607.06065)
  6. LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents (arXiv:2608.17393)

Other work mentioned

  1. MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement (Xiaomi, September 2026)
  2. Harbor: the Terminal-Bench team’s framework for running and evaluating agents
  3. SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale (arXiv:2602.23866)
  4. SWE-bench issue #465: Repo State Loopholes During Agentic Evaluation
  5. On the Rollout-Training Mismatch in Modern RL Systems (Feng Yao et al., 2025)
  6. Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers (R3, arXiv:2510.11370)
  7. SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work? (arXiv:2606.07682)
  8. Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents (arXiv:2604.06132)