World-model action ranking benchmark

A world model can plan by imagining the result of each candidate action and picking the one that lands closest to a goal image. This benchmark checks whether those picks match what really happens. It has 3,499 robot-arm states in RoboCasa kitchens, 64 candidate actions per state, and the true result of every action from replaying it in the MuJoCo simulator.

Main result for Meta's released RoboJEPA models: ranking improves from 22M to about 1B parameters and then stays flat up to 8B, while the error of the imagined future keeps falling. Among the 32 goal-directed candidates the best models reach a Spearman correlation of about .42 and pick the single best action 5% of the time.

Data, metric code and submission instructions: Ryukijano/robojepa-rankability. Simulation only, no physical robot. Licence CC BY-NC-SA 4.0, non-commercial research use.

Released RoboJEPA models

Final released checkpoint for each size.

Models we trained or fine-tuned

Trained on RoboCasa simulator data with replayed outcomes, evaluated on the 499 states from the 10 held-out kitchens. ± is the standard deviation over seeds.

What the columns mean

Overall ρ
Spearman correlation between the model's scores and the true outcome over all 64 candidates, averaged over states. 1 means the order is perfect, 0 means chance.
Fine ρ
The same, over only the 32 goal-directed candidates, which end close to each other. This is the hard part.
Random ρ
The same, over only the 32 random candidates.
AUROC@5cm
How well the score separates actions that end within 5 cm of the goal from those that don't (0.5 is chance).
Regret
How much further from the goal the model's top pick ends than the best available candidate.
Hit@1
How often the model's top pick is the best candidate.
Fine pairwise
For two goal-directed candidates, how often the model orders them correctly (0.5 is chance).
ECE
Calibration error of the score turned into a probability of ending within 5 cm, fitted on half the states and tested on the other half. Lower is better.
Imag. err
How far the imagined next frame is from the real one, in the encoder's feature space (mean absolute difference). Lower is better.

To add a model: run evaluate.py and open a pull request on the dataset with results/<name>.json. We add a row once we can reproduce the result.