TL;DR Robot models like π0.5 show what they are about to do up to a second before they do it, and frontier LLMs can understand these signals. They can also edit, graft and combine the model’s weights directly to make it better, with no gradient training.

We gave Opus 10 manipulation tasks in RoboDojo. It built an algorithm that compares successes with failures to find where each failure lives, then edited just 0.004% of the weights. Success rose on half the tasks, including mahjong (23% to 36%), classifying objects (16% to 21%) and storing a laptop (7% to 12%). When we also let it train small models on simulated trajectories, it found the weight direction that helps folding and lifted folding from 51% to 75%.

Robotics has been built on expensive data collection, complex algorithms and finicky RL. We believe LLMs offer a new way to understand robot brains, optimize them, and eventually solve robotics.

01 · An LLM can read a robot’s brain#

Opus labelled all 73,728 neurons that write π0.5’s motion. In the animation at the top, grab neurons light up to a second before the gripper closes, and release neurons fire as it lets go. Failures show too: on mahjong, the neurons flag a failing grab with an AUROC of 0.81.

02 · Find the failure, flip it#

Opus compares successes with failures, finds the 128 neurons behind the failure, and flips them: 0.004% of the weights, no gradients.

1. Record48 stock attempts per task, good and failed.
2. CompareIn each layer, find the pattern that separates failed grabs from good ones. Check it on held-out attempts.
3. TargetPick the 2 layers where it is clearest and the 64 neurons in each that write it most.
4. FlipReverse the pattern in those 128 neurons. Score the edited robot.

The biggest win: a mahjong task where the robot lays out three matching tiles to claim four of a kind. Success went from 23% to 36%.

Same layout and seed, 1.2× speed. Stock knocks a tile loose; the edited robot finishes in 15 s.

03 · Where the flip helped#

Same recipe on all 10 tasks. Success rose on half of them.

Classify objects, 2.2× speed. Stock picks the wrong bins; the edited robot sorts them.

The clearer the failure signal, the bigger the gain (r = 0.69).

04 · Brain math: copy-paste what a robot learned#

Opus trained two small models on simulated trajectories, one with folding and one without. Their difference is the folding direction. Adding it to the original model lifted folding from 51% to 75%, beating the fine-tuned model it came from.

Layout 44. The edited robot folds in 12 s.

05 · Robots might get solved by LLMs writing weights#

Today an LLM can read a robot’s neurons, fix its failures and copy skills between brains, with no gradient training. Next, it writes the weights for a new task directly.

06 · Explore every neuron#

All 73,728 neurons in 3D across two recorded folds. Drag to turn, tap a dot to see what it does.

Open the viewer full screen ↗

Citation#

Please cite this work as:

@article{gupta2026robotweights,
  author  = {Pranay Gupta and Deepanshu Rohilla and Parjanya Prashant and Jiongli Zhu},
  title   = {{Can frontier LLMs fix robots by editing their weights?}},
  journal = {codnay.github.io},
  year    = {2026},
  month   = {November},
  url     = {https://codnay.github.io/blog/llms-fix-robots-by-editing-weights/},
}

All runs are RoboDojo simulation (Isaac Sim), seeds 0–2, scored by the simulator’s own success check. Related: task arithmetic (Ilharco et al., 2023), model soups (Wortsman et al., 2022), ExPO (Zheng et al., 2024), and Sparks of Weight-Space Programming.

Speeding up robots is a chicken-and-egg problem: you need deployments in the real world to collect data and iterate on models, but you need fast models in the first place to provide enough economic value to earn those deployments.

Several companies like Physical Intelligence^1,2, Skild AI, Generalist AI, Reward AI have shown impressive demos of their models working at impressive speeds, but the fundamental lever behind that is very fast demonstration data, which does not hold once you start scaling a fleet into the real world. We have spent the past few months deploying across various industries, and here is what we, along with several other companies actually deploying robots on the ground, have seen.

Why is deployment data slow?#

Operators in live deployments run two to four times slower than operators in demo setups, for a few key reasons:

1. Assistance has to be remote to scale. Sending an operator to every site breaks before you reach even a hundred deployments. Remote operation lets one office of trained operators serve many sites without downtime, which makes it the only viable path to human-assisted deployment at scale. Remote teleoperation is slower because it is hard to perceive depth intuitively from camera views; building a 3D environment inside a Meta Quest is a good technical solution, but nobody can use one for more than an hour without feeling nauseated.
2. The control interface differs. Many demo datasets use leader follower arms^3,4, which tie operator speed to physical arm speed. Deployments need dynamic speed adjustment like going slower near humans and fragile objects, for safety, etc. Most deployments we have seen use Meta Quest controllers, which track the operator’s hand and move the arm accordingly.
3. There is no predictability. Demo scenes are structured and learnable, so operators can act before they think. In a real deployment, an operator has to react to live cues, which is inherently slower.

Why is slow data a problem?#

Slow teleoperation hurts deployment and scaling in two ways:

1. Weak learning signal. Current robotics policies work by predicting the next chunk of actions. Data in which most of the motion is idle, or does not contain significant movement, carries poor information for a policy to learn from. We have seen this directly in our training runs: policies trained on such data are not only slower but usually less accurate, because each next-chunk prediction offers little progress toward completing the task.
2. Economics. When a policy or a teleoperator runs significantly slower than a human, it becomes impossible to create economic value for the end customer especially in environments such as kitchens and warehouses, which have been optimised around humans doing tasks quickly and efficiently.

Over the past six months we have run policies in live deployments and focused our engineering and research effort on making policies run faster than the teleoperation data they were trained on, so that we can deploy our robots at scale. We have found three levers that make policies faster while maintaining accuracy:

1. Adaptive speed-up of the training data#

The simplest approach you could think of is to speed up the training data and then train a model on it. Unfortunately, that works poorly in practice: you lose a lot of key information when you discard frames from the data you collect. Uniformly dropping every other frame (2x) took success from 94% to 53%, and at 4x uniform it collapsed to 9% for plate picking, handover between arms and release. The failures concentrated at the grasp and the release, exactly where the discarded frames carried the most information.

A better and more effective approach, also observed by many others in the field, is to treat the motion of the arms as two separate parts: free-space motion, where the arms are travelling toward their target region, and contact-rich motion, where the arms are interacting with the real world. Free-space motion contains far less information about manipulation than contact-rich motion, so it makes sense to speed up free-space motion while keeping contact-rich motion intact.

Prior works such as AutoSpeed⁵, SAIL⁶ and SRIL⁷ have consistently used an RL head to predict contact-rich motion and perform speed-ups around it. We were able to achieve similar or better results without training an RL head at all, by simply treating any motion within x seconds of a gripper event (gripper opening or closing) as contact-rich motion; trying everything from 0.1 to 2 seconds.

We ran hundreds of ablations at 2x, 3x, 4x, 5x and 10x, and across different contact-identification techniques, and saw the best results at 4x with actions within 0.5s of gripper events being classified as contact. Beyond 4x the arm still reaches the object, but it arrives with too much residual velocity to close the gripper cleanly, and success falls off sharply.

Contact window fixed at x = 0.5 s; each row is 500 real-robot runs on the same task and scene.

2. Selective chunk skipping at inference#

A VLA policy predicts several chunks of future actions at once. Skipping chunks makes the policy run faster by letting the arm jump to future states, with no retraining. Doing this naively causes two problems, and both of them push the policy out of distribution quickly:

1. Far-out predictions are less accurate. The further out a chunk is, the less likely it is to be accurate, because the world will have changed by the time the first few actions have been executed. The general practice is to query the model often and execute only near-term chunks.
2. Skipping causes jitter. When you skip chunks, the arm moves through a different distribution than the training data, and the video feed from the real world no longer corresponds to the training data, which takes the model out of distribution very quickly.

To solve these issues, our approach after running thousands of tests in the real world is to estimate the risk before skipping a step. We look at the motion predicted by the policy and skip only if both of the following conditions hold:

1. The predicted chunks sit close together in the real world. We measure the end-effector displacement between the last pose of the current chunk and the last pose of the chunk we would skip to, and require it to stay under x cm at 20 chunk executions per second. That threshold is not fixed: a VLM scores the current camera view for scene complexity i.e. clutter, number of movable objects, proximity of a human into 5 buckets, and we scale the allowed displacement from 2 cm in a clean, static scene down to 0.3 cm in a cluttered or human-occupied one. The intuition is that displacement is only safe if the world will still look the way the policy expects when the arm arrives.
2. There must be no gripper event inside the skipping window.

The second condition is essential. Every time we skipped chunks through a gripper event, the task fell apart. Restricting skips to slow, contact-free motion keeps the policy in distribution. With these techniques we achieve another 22 percent speed-up over our fastest policy.

3. DAgger-style intervention data#

Intervention data is the most useful and most unexpected lever for speeding up a model. We expected it to increase accuracy which it does, at roughly 30x the per-frame value of ordinary teleoperation data. But with no other changes, the policy also gained a 10–15 percent speed-up simply from adding intervention data, sped up using the same techniques described above.

The setup follows DAgger (Ross et al., 2011)⁸. The policy runs on the robot, and an operator takes control the moment it starts to fail. We then speed up that intervention data with the same gripper-event rule from Lever 1 and retrain.

For models trained at lower speeds, intervention data barely helps with speed, so there is effectively zero speed-up. For policies trained at a faster speed, sped-up intervention data helps them grow faster and faster.

Final results#

Combined, the three levers let us run our policies nearly four times (3.97x) faster than the data we collected from remote teleoperation. In the table below we report speed alongside accuracy as we add each lever:

Setup. All results are on a bimanual cell of two AgileX PiPER six-DoF arms with parallel-jaw grippers, with three RealSense cameras (one overhead, one per wrist) recorded at 20 fps, driven over CAN from a Jetson Orin NX 16 GB. The policy is π_0.5², with an action chunk of 30. Training data is 10 hours of remote teleoperation collected in the same deployment across seven days plus around one hour of usable teleoperation intervention data. Each number in the tables is 500 real-robot runs, with object positions re-randomised between runs.

Limitations#

These results come from a narrow set of tasks: picking up plates and dishes, and picking and placing objects. We have yet to test whether the gripper-event rule and the 4x optimum hold across a more diverse task set. We also have no explanation yet for why 4x outperforms both lower and higher factors.

What comes next?#

We expect our policies to soon run significantly faster than any human can control them, on specific tasks. To build superhuman policies, human data alone cannot get us there.

Human intervention data plays a role in robotics similar to the one RLHF⁹ played in the early days of LLM post-training. The field had to move past human feedback as models improved, because the models began to outperform expert annotators by too wide a margin. Robotics will hit the same wall in six to eight months.

Our next line of research is RL post-training with no humans in the loop. We plan to apply techniques in the GRPO¹⁰ and DAPO¹¹ family to robot policies, with the goal of pushing speed beyond human capability while holding accuracy at similar levels.

References#

Citation#

Please cite this work as:

@article{gupta2026speedingup,
  author  = {Pranay Gupta and Deepanshu Rohilla},
  title   = {{Scaling and speeding up robots in the real world}},
  journal = {codnay.github.io},
  year    = {2026},
  month   = {September},
  url     = {https://codnay.github.io/blog/scaling-and-speeding-up-robots-in-the-real-world/},
}