Render the Robot, Generate the World

One video world model across embodiments

Yash Jangir1 Karan Mirakhor2 Renos Zabounidis2 Muhammad Zubair Irshad3 Sergey Zakharov3 Yonatan Bisk2 Homanga Bharadhwaj1

1Johns Hopkins University2Carnegie Mellon University3Toyota Research Institute

An initial image and an action trajectory. The body is rendered in 3D along the actions, and Doppel-8 generates the video: a long-horizon rollout on a held-out OakInk2 episode.

Abstract

Video world models usually learn how a robot moves from raw action vectors. Doppel instead renders the robot along its planned motion, from the camera's viewpoint, and conditions one shared video model on these renders together with the actions. Camera poses missing from robot datasets are recovered from the videos. One model trained on eight datasets (Doppel-8; about 600 hours of robot arms, a bimanual robot, a humanoid, and human hands and bodies) improves PSNR over action-only baselines on the three robot-arm datasets by 1.00 dB on unseen trajectories and 1.20 dB out of distribution. Rendering also makes learning more efficient: trained on the three robot datasets, Doppel achieves higher average PSNR and SCS than a shared action-only baseline with 63.8% fewer training draws.

Actions alone, or the rendered robot

Same first frame, same actions. The usual model sees only the action vectors; Doppel also sees the robot rendered along those actions.

First frame
Action-conditionedAction-only expert, trained on UR5e data only
Actions, 8 per frame
Prediction
Render-conditionedDoppel
Rendered robot + actions
Prediction
Ground truth
Frame 00 / 16

Videos

Same first frame, same actions. The baseline is an action-only expert: the same backbone, trained on that robot alone.

The action-only expert moves the arm but leaves the object in place. On the held-out toy kitchen, Doppel-8 raises moving-object PSNR from 21.50 to 24.85 dB.

Clip

Ground truth
Action-only expert
Doppel
Frame 0 / 16

Long horizon

Rollouts from a single frame on held-out test episodes. Each step continues from the model's own last frame, so the model never sees the ground truth again. Errors accumulate over the rollout.

Ground truth
Rendered motion
Doppel-8

Camera extrinsics

Rendering the robot needs the camera pose. RoboMIND, Bridge and ABC don't provide it, so we recover it from the videos: render the robot at its logged joint angles and optimize the camera until the render lines up with the video.

One run, step by step

A WidowX clip calibrated from a perturbed start. Blue: the robot rendered from the current camera estimate.

Recovered extrinsics

Recovered camera extrinsics per dataset.
EmbodimentDemosExtrinsicsOne per
UR5e RoboMIND17,58849recording session
WidowX Bridge14,82456recording session
YAM ABC12,93312,933video
Total45,34513,038

ABC cameras move between recordings, so each ABC video gets its own extrinsics. We will release the recovered extrinsics with the calibration code.

Method

Doppel conditions Wan 2.1 I2V 14B on render-enriched dense representations, injected as per-token timestep embeddings.

Data

Eight datasets: about 600 hours of training video and 803,602 training clips, with one shared action encoder and no embodiment label. Each dataset has 200 validation clips (unseen trajectories) and 200 test clips (the listed shift).

Results

Doppel beats action-only conditioning and fine-tuned OSCAR on all three robot datasets, on unseen trajectories and under distribution shift. The cards show Doppel-8; click one for Doppel's training curve against the expert.

Ground truth beside Doppel-8 on held-out test clips: a new lab, a new kitchen, unseen objects.

The five added datasets

Doppel-8, the same model as the cards above.

Trained on the three robot datasets, Doppel achieves higher average PSNR and SCS than a shared action-only baseline with 63.8% fewer training draws.

Region metrics

Scores on the robot and on moving objects only, at the final checkpoints, as in the table.

Full whole-frame results table

Final checkpoints, 200 clips per split. Highlighted: Doppel-8. Best in bold. OSCAR is skeleton-conditioned, fine-tuned per dataset.

PSNR, SSIM, LPIPS and SCS by dataset and method on the selected split.
DatasetMethodTrained onPSNR ↑SSIM ↑LPIPS ↓SCS ↑PSNR vs action-only

Metrics over the 16 predicted frames. SCS: overlap of SAM 2 object masks between the predicted and ground-truth videos.

Policy evaluation

A policy runs closed-loop inside Doppel: its actions are rendered on the robot, Doppel predicts the scene, and the prediction is the policy's next input. Here Doppel is fine-tuned on this robot's demonstrations.

Bimanual YAM, MolmoAct2 fine-tuned on this robot. From the real start frame, the policy completes the task inside Doppel.

In the wild

Extreme out of distribution: a phone video of a climber. From one real frame and the input motion, Doppel-8 rolls out the rest.

Limitations

Geometry and a camera

Building the renderings needs a URDF, accurate robot geometry and a calibrated camera. Unknown or rapidly changing viewpoints remain a limitation.

Configuration trajectories

Doppel conditions on joint-space trajectories. Imperfect geometry or lower-level control commands would require modeling uncertainty in execution.

Seen embodiments, one camera

Evaluation covers embodiments seen in training and one camera per trajectory. Transfer to unseen embodiments and multi-view prediction are future work.

Long horizons

Long rollouts accumulate error: PSNR falls by about 5 dB from start to end.

Citation

@misc{jangir2026doppel,
  title  = {Render the Robot, Generate the World:
            One Video World Model Across Embodiments},
  author = {Jangir, Yash and Mirakhor, Karan and Zabounidis, Renos and
            Irshad, Muhammad Zubair and Zakharov, Sergey and
            Bisk, Yonatan and Bharadhwaj, Homanga},
  year   = {2026}
}