WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining

TL;DR

Imagine the Future. Predict Its Motion.

WorldFlow jointly generates future video and explicit 3D motion, combining large-scale visual pretraining with geometric supervision for more generalizable motion prediction.

Abstract

Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics. To circumvent these limitations, we introduce WorldFlow, a generative world model that jointly predicts future video and query-based 3D trajectories. By parameterizing 3D motion as continuous displacement tokens aligned with the video grid, a unified Mixture-of-Transformers (MoT) backbone jointly denoises both modalities under a rectified-flow objective. This formulation couples explicit geometric dynamics with visual evolution, effectively unlocking the physical commonsense embedded within pretrained video models. To empirically scale and validate this unified architecture, we train WorldFlow on a heterogeneous mixture of 1.44 million object-level samples spanning simulated and real-world domains. We further introduce WorldFlowBench, comprising 1,678 evaluation samples from 11 data sources across simulated and real-world interactions, including spatial reasoning tasks. Across all three evaluation sets, WorldFlow achieves the lowest average displacement error (ADE) and final displacement error (FDE) among the evaluated learning-based methods. Notably, on the challenging SpatialCog set, WorldFlow achieves an ADE of 0.060 m, a 54.2% reduction compared with the strongest external baseline. It also reduces ADE and FDE by 55.9% and 68.5%, respectively, compared with its flow-only variant.

Motivation

Point 1What aspects of future states do different representation spaces capture?

Pixels capture appearance. Actions describe an agent's commands. We choose motion space: explicit 3D motion over time as the core motion prediction space.

Pixel space

What will the world look like?

Future images and video

Limitation: Pixel predictions leave 3D dynamics implicit, requiring additional geometric grounding for manipulation planning and control.

Action space

What should this agent do?

Embodiment-specific commands

Limitation: Embodiment-specific action spaces make models trained on one robot difficult to transfer directly to another.

Motion space (our focus)

How will the world move?

3D flow over time

A shared geometric description of physical change, beyond a particular robot's action interface.

f_t = p_t − p_t−13D displacement at each query point, over time3D + time = 4D dynamics

All 3D positions and flow vectors are expressed in the first frame’s camera coordinate system.

Point 2Is flow-only generative training enough to generalize?

Evolution of motion modeling: (a) Tracking models recover dense scene flow from observed video. (b) Motion prediction models forecast future 3D flow from an image and text, using sparse scene flow or query-conditioned formulations. (c) WorldFlow is conditioned on an image, text, and 3D query points, and jointly generates future video and query-conditioned 3D flow.

(a) Tracking

Observed video

Tracking
model

Dense scene flow

(b) Motion prediction

Image

Text

opt*Query points

Motion prediction
model

Sparse scene flow

opt*Query-conditioned flow

(c) WorldFlow (ours)

Image

Text

Query points

World model

Future video

Query-conditioned flow

* Optional pair: query points are used only for query-conditioned flow prediction.

Why Joint Video-Flow Prediction?

Geometry alone does not constitute a full world model. While 3D trajectories explicitly track positions over time, they compress away the core physical constraints—such as object boundaries, surface textures, and environmental affordances—that define real-world interaction. Consequently, a motion model trained strictly on sparse 3D coordinates inevitably suffers from generalization collapse when encountering novel scenes or tasks requiring intricate spatial reasoning.

Joint video-flow prediction bridges this cognitive gap. Pixels provide the dense, scalable signal necessary to unlock the rich physical and visual priors accumulated through large-scale video pretraining. Instead of treating motion as isolated geometric vectors, WorldFlow forces the model to synthesize the scene's appearance and its underlying 3D trajectories simultaneously, embedding explicit motion within a global physical and semantic context.

Supported by our flow-only comparison: SpatialCog ADE 0.136 → 0.060 m (55.9% lower) · Core Argument 1

Framework

WorldFlow architecture: text, an observed frame, and query points condition a pretrained reasoner and a joint generator with shared multimodal attention to predict future video and cumulative 3D flow. — **Overview of WORLDFLOW.** Conditioned on text tokens `t_j` through the pretrained reasoner, clean observed-frame latents `v₀`, and query-point geometry, the generator jointly denoises video and flow from noise `ϵ_j` and `δ_k` through shared multimodal attention. Generated video latents `v_j` are decoded by the VAE, while generated 3D flow `f_t` is rescaled and accumulated from the initial query positions to recover metric trajectories. The 2D flow overlay shows generated 3D trajectories projected onto generated video frames. The 3D flow overlay shows generated 3D trajectories overlaid on a scene point cloud reconstructed from the initial frame’s ground-truth depth.

Experimental Results

Qualitative comparison

SpatialCog

11 cases

Input frame

Text prompt

put the card set to the left of the Monkey King building-block box

Flow on the observed frame

WorldFlow · generated RGB + flow

Ground truth · video + flow

0 / 48

WorldFlowBench Results

ADE / FDE ↓ Lower is better · PWT ↑ Higher is better · Scroll horizontally to view all columns

3D motion forecasting on WorldFlowBench.
Model	Sim7			Real3			SpatialCog
Model	ADE ↓	FDE ↓	PWT ↑	ADE ↓	FDE ↓	PWT ↑	ADE ↓	FDE ↓	PWT ↑
Non-parametric reference
Static	0.096	0.201	0.573	0.062	0.086	0.464	0.143	0.278	0.412
Video generation + 3D tracking
NovaFlow	0.196	0.318	0.289	0.095	0.129	0.293	0.165	0.295	0.320
Cosmos3-Nano + T4W	0.101	0.202	0.451	0.088	0.133	0.310	0.141	0.257	0.398
Cosmos3-Nano + 4RC	0.096	0.196	0.531	0.085	0.122	0.332	0.141	0.263	0.401
LingBot-Video-MoE + T4W	0.105	0.207	0.425	0.097	0.145	0.267	0.134	0.239	0.364
LingBot-Video-MoE + 4RC	0.104	0.207	0.492	0.092	0.134	0.295	0.131	0.232	0.376
Direct flow generation
MolmoMotion-AR-H1	0.176	0.323	0.280	0.185	0.335	0.222	0.282	0.533	0.173
μ₀	0.186	0.307	0.069	0.116	0.156	0.148	0.213	0.270	0.055
EgoWAM	0.198	0.336	0.020	0.130	0.190	0.056	0.249	0.383	0.080
Ablations
WorldFlow (flow-only)	0.099	0.206	0.495	0.101	0.152	0.320	0.136	0.248	0.372
WorldFlow + T4W	0.088	0.174	0.424	0.100	0.154	0.263	0.083	0.138	0.403
WorldFlow + 4RC	0.088	0.170	0.483	0.092	0.132	0.291	0.098	0.175	0.411
WorldFlow w/o auxiliary loss	0.063	0.116	0.527	0.083	0.115	0.299	0.062	0.084	0.436
Joint video + 3D flow (ours)
WorldFlow	0.051	0.094	0.564	0.081	0.113	0.311	0.060	0.078	0.449

ADE and FDE are position errors in meters. PWT is the mean point–frame hit rate at 1, 2, and 5 cm. Bold and underline indicate the best and second-best non-static results, including ablations. Static is a reference excluded from ranking. T4W denotes Track4World. WorldFlow + T4W / 4RC extracts trajectories from WorldFlow-generated videos using the corresponding tracker.

Core Argument 1: Joint generation outperforms flow-only training

The flow-only variant shares WorldFlow’s DiT Transformer backbone, training data, and optimization settings. The variant predicts only 3D flow, whereas the full model jointly generates video and flow. Joint generation reduces ADE by 48.5%, 19.8%, and 55.9% on Sim7, Real3, and SpatialCog, respectively, and lowers FDE on all three subsets—including a 68.5% FDE reduction on SpatialCog (0.248 → 0.078 m)—supporting the value of complementary visual supervision for accurate 3D motion.

Flow-onlyWorldFlow · joint generation

ADE (m) ↓Error reduction vs. flow-only

FDE (m) ↓Error reduction vs. flow-only

Core Argument 2: Direct flow generation outperforms post-hoc tracking

Using the same WorldFlow checkpoint and conditioning inputs, directly generated trajectories achieve lower ADE and FDE and higher PWT than Track4World and 4RC applied to its generated videos on all three subsets. This supports explicit 3D flow as a generative output rather than recovering motion through a separate tracking stage.

WorldFlow video + T4WWorldFlow video + 4RCWorldFlow · direct flow generation

ADE (m) ↓Error reduction vs. T4W / 4RC, respectively

FDE (m) ↓Error reduction vs. T4W / 4RC, respectively

Tracking failure cases

Tracking model

Text prompt

put the plastic gold coin in front of the pink food-shop building-block box

WorldFlow video + T4W

ADE 0.111 m · FDE 0.217 m

WorldFlow video + flow

ADE 0.041 m · FDE 0.036 m

GT video + GT flow

Ground truth

0 / 48

Pixel-dependent trackers can fail on small objects and during object flipping, where visibility and appearance change substantially. Through large-scale flow training, WorldFlow generates more physically plausible trajectories than even the GT flow annotations in these examples. The first two columns use the same generated video, contrasting post-hoc tracking with directly generated 3D flow; the third shows ground truth. T4W trails retain segments with confidence ≥ 0.5 at both endpoints. ADE/FDE are computed over all query points, without confidence filtering.

Real-Robot Experiments

Flow prediction success rate

WorldFlow97.2%35 / 36

μ₀25.0%9 / 36

Robot execution success rate

WorldFlow66.7%24 / 36

μ₀11.1%4 / 36

Flow prediction and robot execution outcomes are labeled separately for each trial. For visualization, trajectories whose initial query points lie far from the object point cloud are filtered using the same rule for both methods; success labels are unchanged. Point-cloud colors come from the observed image.

Per-task flow prediction and end-to-end robot execution success.
Task	Flow prediction		Robot execution
Task	WorldFlow	μ₀	WorldFlow	μ₀
Pick and Place
Place the watermelon doll into the small pink plate.	3/3	0/3	3/3	0/3
Place the oolong tea into the large blue tray.	3/3	1/3	2/3	0/3
Place the potato chips into the dusty pink basket.	3/3	1/3	2/3	0/3
Place the yellow toy inside the red circle.	3/3	1/3	2/3	1/3
Stacking
Place the green biscuit box on the translucent box.	3/3	1/3	3/3	0/3
Place the can of Red Bull on the translucent box.	3/3	1/3	2/3	1/3
Place the beef curry on the paper towel.	3/3	1/3	1/3	0/3
Place the yellow toy on the paper towel.	3/3	2/3	0/3	0/3
Relative Placement
Place the sunscreen in front of the tissue.	2/3	0/3	1/3	0/3
Place the potato chips behind the Oreo box.	3/3	0/3	2/3	1/3
Place the watermelon doll to the left of the cola.	3/3	1/3	3/3	1/3
Place the oolong tea to the right of the small blue toy car.	3/3	0/3	3/3	0/3
Overall success rate	97.2% 35/36	25.0% 9/36	66.7% 24/36	11.1% 4/36

Each entry reports successes out of three trials (36 trials per method). Flow prediction and end-to-end robot execution are annotated independently. The drop from flow prediction success to end-to-end execution success mainly stems from grasp failures by T(R,O) Grasp, planning failures in trajectory optimization, and unstable placement during stacking. Both methods use matched initial layouts and the same grasping, planning, and control settings.

Future Work

We aim to scale both the volume and diversity of flow data, with particular emphasis on reliable real-world annotations and broader coverage of motion patterns, interactions, and camera configurations. We also plan to extend joint generation to include robot actions alongside video and 3D flow. By progressively establishing reliable performance and generalization for each modality, we seek to understand whether jointly modeling video, flow, and action yields synergistic benefits in predicting future world evolution and guiding manipulation.

Citation

@misc{lu2026worldflowjointvideo3d,
  title={WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining},
  author={Yuhao Lu and Jiongxu Chen and Hexiang Wei and Jiajin Wu and Juelin Li and Mingjian Liang and Fan Yang and Zhuowei Liang and Zihao Liu and Zhennan Wang and Sheng Wang and Lin Shao},
  year={2026},
  eprint={2610.39278},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2610.39278},
}

WorldFlowJoint Video and 3D Motion Prediction
via Large-Scale Pretraining