Praxis-1: From Video Pretraining to Robot Control

1.Introduction

Robotics deployments keep running into the same problem: real-world robot data is scarce, and collecting enough of it to train a generalist policy is expensive. Some labs respond with more teleoperation, others with more egocentric footage. We think the most scalable source is general video, the everyday footage people already record in huge volume. A model trained to predict that video learns how objects behave, how hands move and what a task looks like partway through. A robot can build on that knowledge.

With Praxis-1, we turn a 27B-parameter video foundation model into an open weight world model that predicts the future in short chunks. A small action expert reads the video model's internal state as it imagines what comes next. Each time the robot acts, Praxis-1 predicts both what the scene will look like and the motion that will get it there.

Praxis-1 fine-tuned on DROID is the top world action model and the top open-source robot foundation model on the RoboLab-120 leaderboard. On a real Franka arm it achieves a competitive success rate against the previous state-of-the-art world action model.

Praxis-1 can adapt to various embodiments, from single arms to bimanual robots and mobile manipulators. We showcase everyday tasks such as placing a mug upside down on a tray, putting an item in a box and closing the lid, clearing bottles into a bin or closing a drawer. The model can also adapt to embodiments that it hasn't seen in training data. We detail this with Noble Machines, Google's RT-1 and SO-101 below.

2.Architecture

Praxis-1 has three parts. A video backbone predicts what the scene will look like next, and an action expert reads the predicted video features to work out the motion that produces it. Thin per-robot projections translate each robot's poses and actions into a shared format, so one model can control different embodiments. On each call, the model takes an instruction, the latest camera frames and the robot's current pose. It returns a predicted chunk of future video along with 1.6 seconds of actions: 24 action rows at 15 Hz. The robot executes the chunk, sends back what its cameras saw and calls again.

2.1The video backbone

Praxis-1 is built on Runway's video foundation model: a diffusion transformer split into a condition encoder and a velocity decoder (a DDT layout, after Wang et al., 2025). It works in the latent space of a frozen Wan2.2 VAE and reads instructions through a frozen Qwen3.5-2B text encoder. 17 of its 20 feed-forward layers are mixture-of-experts (MoE); only the first two encoder blocks and the first decoder block stay dense.

Each MoE layer has one shared expert that every token uses, and 128 small routed experts, of which each token picks eight. The nine experts a token passes through add up to the width of one dense layer, so the compute per token stays about the same as a dense block while the layer holds 129 experts' worth of parameters. In total the backbone has 26.9B parameters, of which about 2.8B are active for any token. Experts are balanced as in DeepSeek-V3 (DeepSeek-AI, 2024): a routing bias that needs no auxiliary loss, plus a small sequence-level auxiliary loss.

The base model was trained to generate whole clips at once, with every frame able to see every other frame, including later ones. A robot can't see the future, so during mid-training we change the attention to be chunk-causal: the model works through time in 1.6-second chunks of six latent frames, and each chunk can only see the chunks before it. Each chunk is denoised from its own noise level while attending to itself and to clean copies of the earlier chunks. In training, those earlier chunks come from real video rather than the model's own predictions, a setup known as teacher forcing, which lets us train on four chunks at once in a single parallel pass. At inference, the model applies the same rules one chunk at a time. By default it reads one earlier chunk of real camera frames from the robot, so it sees the same kind of clean context it saw in training. The attention mask is shown below, alongside the action expert's.

2.2The action expert

The expert is a dense 16-layer transformer of width 1024, with about 314M parameters. It is closest in spirit to DeVA (Zhang et al., 2026), which also keeps a separate action expert that reads several depths of the video model. Its layers alternate between cross-attention into the backbone and self-attention. Each cross-attention layer reads one of eight backbone encoder blocks, spread from early to late layers, while a shared normalization and a 3072→1024 projection bring the backbone features to the expert's width.

Per chunk the expert holds one absolute pose token, eight learned "future" tokens and six action tokens of four rows each. Its queries for chunk k see the clean backbone frames before k and the backbone's noised frames of chunk k itself. The expert therefore reads the future as the backbone imagines it. The backbone never reads the expert, but the action loss still trains the backbone through those cross-attention reads.

We also tried putting action tokens inside the backbone (a mixture-of-transformers layout). On our offline proxy, a short bimanual fine-tune scored on action error, this layout showed no clear difference from the separate expert (+0.01 cm) and cost more. Thus we kept the separate action expert and leave other architectures as future work.

2.3Inputs and sampling

3.Training stages

Praxis-1 learns how the world works before it learns to act. Training runs in four stages, each starting from the weights of the one before, and video prediction stays part of the objective throughout. When actions are introduced, the model learns them alongside video rather than in place of it. Each stage reaches its effective batch size through gradient accumulation on top of the GPU count.

4.Mid-training Compute Scaling

We expect more mid-training compute to give fine-tuning a stronger base model. We tested this on two robots: Franka, which is seen in our DROID mid-training data mix, and RT-1, a new embodiment never seen in any pretraining or mid-training stage. Each point is a checkpoint from the same mid-training run, placed by the total compute spent to reach it, video pretraining included. Because later checkpoints revisit the same mid-training data, this measures more training compute rather than more data.

In our experiments on two embodiments, more mid-training did lead to better fine-tunes. We fine-tuned five checkpoints from the video and action mid-training run on DROID with the same schedule. On RoboLab-120, with the same sampler settings for every point, the first four scored progressively higher, 25.5%, 35.2%, 39.2% and 41.2%, and the fifth leveled off at 41.2%.

RT-1 is absent from the mid-training data, which allows us to test the same question on an unseen robot. We fine-tuned five mid-training checkpoints on RT-1 with the same recipe. SimplerEnv success rose from 56.3% for the earliest checkpoint to 80.0% for the latest, though not at every step (Figure 6). We see a trend of closed-loop improvement on both the seen and unseen embodiments, but the exact relationship needs further study.

5.Inference

Our model predicts actions chunk by chunk: the robot executes the whole chunk, sends back the frames it saw and calls again over a single WebSocket. Each policy call returns 24 action rows at 15 Hz, or 1.6 seconds of motion, denoised with the imagined video in two joint Euler steps. The first call of a context window returns 21 rows instead, because the first of its six latent frames covers only the current frame, not four. On a single H100 server, a call with the default chunked execution takes about 0.55-0.6 seconds round trip, about a third of the 1.6 seconds of motion it returns. By default, we run video classifier-free guidance (CFG) sequentially.

We have also explored real-time chunking (RTC): instead of waiting for a chunk to finish, the robot asks for its next chunk partway through the current one and keeps moving. The hard part is the junction. Blending the old and new plans made the arm oscillate, so the new plan instead continues from where the arm will actually be, using the model's own motion. On our bimanual station, this cut the time spent holding still about fivefold (from 32% to 5% in a session where each call took 0.8-0.9 seconds, and from 16% to 3% with a faster server at about 0.3 seconds per call). On a five-object pick-and-place into a bag, RTC reached a comparable success rate (24/30 vs 28/30 chunked) and finished faster (median 9.1 vs 15.2 seconds to success). RTC is opt-in in our release repo; chunked execution is the default. With RTC on, the server also precomputes the backbone while the robot is still executing, which cuts the round trip to about 0.4 seconds; the actions are bit-identical to RTC without it.

6.Results

Every result in this section comes from the same Praxis-1 checkpoint, fine-tuned on DROID using the recipe in Training Stages. We evaluate it in simulation on RoboLab-120, and on a real Franka FR3 arm operated by Positronic.

7.Everyday Tasks in Real-time

Topping a leaderboard is one thing; running on a real robot is another. Here Praxis-1 runs two real robots on everyday tasks: a bimanual YAM robot, whose two arms work together on tasks one arm cannot do alone, and a single Franka arm. Every clip below shows the policy in control, with no teleoperation, played at 1x.

The YAM fine-tunes follow the same recipe as DROID : they start from an action mid-training checkpoint and learn from roughly 1,400 to 2,600 hours of bimanual YAM recordings with three cameras.

7.1Two arms (YAM)

Collected on bimanual YAM stations at the Runway office, real speed; each clip is labeled with its instruction.

7.2Single arm (Franka, DROID)

Collected by Positronic on a Franka FR3 station, real speed; each clip is labeled with its instruction.

7.3Compare to a frontier LLM agent

Frontier multimodal models can now drive robot arms zero-shot, through tool or code interfaces. GPT-6 Astra placed above every public entry on RoboDojo when it was first evaluated, with 22.5% success (Zhang et al., 2026), and reaches 76.0% on RoboTwin 2.0 (Zhou et al., 2026). These agents are general: they solve a wide range of tasks zero-shot, and today they lead the RoboLab-120 leaderboard, where the top two entries are LLM agents (86.0% and 57.1% success). Their drawbacks are latency and cost. On the AA-Robotics benchmark, Astra averaged 30 minutes and $16.36 per tabletop task, against 20 seconds for a person (Artificial Analysis, 2026), and Zhou et al. report that its inference latency hinders practical deployment.

To measure this gap ourselves, we ran Praxis-1 and GPT-6 Astra on the same YAM teleoperation station, through the same console, on the same three tasks. On average, Astra took 15.6x longer to finish a task and cost 51x more per completed task (Figures 11 and 12).

Frontier LLM agents are advancing quickly, and the two approaches are complementary: an agent can plan a new task, and a fast policy like Praxis-1 can carry out each step in real time. We are studying how to pair them as future work (see Where we go from here).

8.Adapts to New Robots

8.1Noble Machines, from 29 minutes of data

We fine-tuned Praxis-1 for Noble Machines' robot on a single task, picking up pieces of wood and placing them in a container. The training data was 154 teleoperated episodes, about 29 minutes in total, with 26 minutes used for training and 3 held out for validation. After 4,000 fine-tuning steps (the last 1,000 with a decaying learning rate), about 3.6 hours on 16 GPUs, we tested the model in a simulation of Noble Machines' robot and task. On the two layouts it was trained on, it succeeded in all 15 trials (10/10 and 5/5). On a harder warehouse task not in its training data, clearing a pile of three planks into two bins, it fully succeeded in 6 of 10 episodes and placed 25 of 30 planks. We haven't yet run it on the physical robot, but the result is an early sign that a strong mid-trained base model can learn a new robot and task from a small amount of data.

8.2Google RT-1

To further test how Praxis-1 approaches a robot it hasn't seen before, we fine-tuned on RT-1, Google's mobile manipulator, whose recordings never appeared in our mid-training mix. From the latest mid-training checkpoint, we fine-tuned for 30k steps (about 28 hours on 32 H100s) on the full RT-1 train split: about 78k episodes and 95 hours, head camera only. We scored the model on SimplerEnv's official visual-matching grid (864 episodes over four tasks), the benchmark most RT-1 policies report.

Praxis-1 fine-tuned on RT-1 reaches 80.0% averaged over the four tasks, above RT-1 itself (52.4%). Comparatively, FLUX 3 Action reaches 39.7% when we fine-tune it on the same RT-1 data from its base checkpoint with Black Forest Labs' published DROID fine-tuning recipe.

8.3SO-101

SO-101 is a low-cost, open-source single arm built for hobbyists and researchers, and its community has uploaded thousands of small LeRobot datasets. We gathered 3,956 public SO-100/SO-101 datasets from 1,783 contributors, about 1,260 hours, mostly recorded at 30 fps and 640x480. Each has one to three scene cameras, and 2,438 of them (about 860 hours) also have a wrist camera.

Camera names in community datasets are unreliable: a camera called "phone" or "top" is often mounted on the wrist. So we labelled every camera from its video, and counted it as a wrist camera only when two checks agreed: motion (a fixed scene camera keeps most of its pixels still over 64 frames, a wrist camera does not) and appearance (a zero-shot CLIP model).

We held out 102 whole datasets (64 hours) from fine-tuning for validation, and fine-tuned Praxis-1 on the training episodes of the rest, about 1,070 hours from 3,814 datasets. Each sample uses one scene camera, plus the wrist camera when there is one; wrist datasets are also sampled with the scene camera alone, so one model runs with or without a wrist camera. The model predicts the arm's five joints (as offsets from the current state) and the gripper at 15 Hz, in 1.6-second chunks. Instructions are the clip's task or auto-captioned sub-steps that fall inside each training window. Each clip below shows the policy in control on a single instruction, played at 1x.

9.Finding the optimal guidance

Praxis-1 samples each action chunk with two denoising steps and applies classifier-free guidance to the instruction only: the unguided pass keeps the video history and the pose but swaps the instruction for a blank caption. The video guidance scale sets how strongly the predicted video follows the instruction, and a scale of 1 turns it off. The second knob is the flow shift, which warps the time grid of the two denoising steps. A larger shift leaves more of the denoising to the second step. Video and actions share one shift, which we tune for both together; giving each its own shift is left for future work.

We set both knobs with a step-wise search on a fixed set of simulated environments, starting from no video guidance at shift 8. Adding video guidance helped, and raising it together with the shift helped more, ending 4.0 points above the start; pushing either knob further scored lower. We ship video CFG 4 with shift 12.5 and two denoising steps. These guidance settings are embodiment-specific: they were tuned for the DROID Franka, and other robots need their own.

10.Offline evaluation

Alongside closed-loop benchmarks, we judge the model open loop in two ways: how accurate its actions are, and whether the future it imagines follows the instruction.

10.1Does the error come from the predicted video or the predicted actions?

We measure action error as the distance between the predicted and ground-truth absolute end-effector positions, on 32 held-out DROID windows. We use two conditioning modes to separate two sources of error: imagining the wrong future, and reading the wrong motion off the right one.

• Deployment: the model samples its own video and actions together, as it does on the robot.
• Oracle: we give the model the true future video, noised to the sampler's level, and sample only the actions, following the oracle test in mimic-video (Pai et al., 2025).

We observe that most of the remaining error comes from which future the video model imagines, not from the action expert, so improvements to future video prediction should pay off most. Given the real future, the error is about one-fifth as large. Note that deployment error is not a fully reliable metric: it combines future-video and future-action prediction, and several different futures can accomplish the same task. We discuss a better way to evaluate generated videos in the next section.

10.2Can the model picture the task before doing it?

To understand how the model plans and imagines the future, we give it a single first frame of a 3D-built scene instead of a real episode (only the background comes from a real image, with objects of various colors, sizes and positions added), the instruction and the robot's real starting pose. It then rolls itself out for 12.6 seconds, predicting video and actions together with no ground truth after the first frame. A vision-language judge watches each predicted video and counts it as following the instruction only if the right object is acted on, it goes to the right place and the action is correct. The scenes are built with look-alike objects, to test how robustly the model follows the prompt. Instruction following in imagination is a proxy for how well the model understands the task; how it relates to real-world success still needs further study.

Examples from the DROID prompt set: 12.6-second open-loop self-rollouts of Praxis-1 finetuned on DROID, starting from the first frame shown, with the judge's verdict.

The same imagined-rollout method also evaluates the mid-training base model, which is not fine-tuned on any specific embodiment. These long-horizon examples come from that base model on a bimanual YAM station, shown for video quality rather than task success. Each starts from a single first frame of a 3D-built scene instead of a real episode, and the model imagines all three cameras for up to 20 seconds with no robot and no actions. We also change the instruction during each rollout to see if the model can react to it in its video prediction, as listed above each example.

10.3Can the model predict whether a task will succeed?

We want to understand if the model's prediction of its own future also signals whether the task will succeed. In internal research on an earlier, unreleased version of the model fine-tuned for the Franka arm, we froze its weights and trained a small head on its representation of the next predicted chunk to estimate the chance that the episode succeeds from that point. Its only supervision was whole-episode outcomes, with no hand grading. On 272 real Franka episodes from sessions and policies it never saw, the score separated successes from failures after four chunks (about 6 seconds) with an AUROC of 0.77 (95% CI 0.70-0.83); elapsed time alone gets 0.64. Averaged per policy, it also ranked nine checkpoint and sampler combinations by their real success rate with a Spearman correlation of 0.79, though the interval is wide (0.19-1.00).

This is early evidence. The head doesn't yet read the instruction, so it gives a clean pick-and-place the same score whether or not it was the right task. Still, the result points to the direction we care about most, which is a model that learns from its own video predictions by using its imagined future to score and then improve its actions. We are actively researching this direction toward a truly self-improving world action model.

11.Quantization

Praxis-1 also holds up with its routed MoE experts rounded to 4-bit NVFP4. We fine-tuned the released checkpoint for 5k more steps with quantization-aware training, rounding the routed expert matrix multiplies to NVFP4 (E2M1 values in blocks of 16, E4M3 block scales and an FP32 global scale). We sampled 600 episodes from RoboLab-120 and ran them head-to-head on the two checkpoints: the NVFP4 model succeeds in 279/600 (46.5%) and the same fine-tuned checkpoint run in bf16, with the rounding switched off, in 287/600 (47.8%). The two are close. We repeated the test on the bimanual YAM, with a 5k-step quantization-aware fine-tune of a bimanual YAM checkpoint on a real station: on a five-object pick-and-place into a bag, the NVFP4 model succeeded in 26 of 30 rollouts (87%) against 28 of 30 (93%) for the bf16 checkpoint it was fine-tuned from, with grab quality 2.73 vs 2.40 (out of 3) and mean time to success 20.7 vs 20.9 seconds. None of these differences is significant at 30 rollouts per arm. We will release the NVFP4 checkpoints in our Hugging Face repository. Packing the routed experts in NVFP4 shrinks the policy weights from 54.4 GB to 17.5 GB.

12.Limitations of Praxis-1

• Look-alike objects. The most common real-robot failure is reaching for an object that resembles the target, and the wrong choice is usually already visible in the imagined video. The arm then carries out the plan it imagined, so the error starts in which future the video model predicts rather than in the action expert. This is consistent with Section 10.1, where most action error comes from the imagined future.
• Vague instructions. The policy does better when the instruction names the object's attribute (color, size); vague instructions are noticeably harder. When the instruction alone does not single out the target, the model has to guess among the plausible objects in the scene, and in a scene with look-alike objects it can pick the wrong one.
• Long tasks. Multi-step tasks work when each step gets its own instruction, but chaining several steps from a single prompt remains hard. The model has to keep track of which steps are already done and move on to the next one without being told. Pairing it with a language or vision-language model that breaks a task into steps is one of the directions in Section 13.

13.Where we go from here

We are releasing three checkpoints on Hugging Face under the Apache 2.0 license: Praxis-1 fine-tuned on DROID, Praxis-1 fine-tuned on the bimanual YAM and the base mid-trained model. They come with the model code in our GitHub repository, a policy server, a MuJoCo simulation of the same setup and code for both full and LoRA fine-tuning. See more details in our launch post.

We'll continuously invest in world action model (WAM) research, especially focusing on long-term reasoning, agentic planning and self-improvement loops.

We are excited to deploy our policy with partners on real tasks. If you'd like to work with us on this, get in touch.