Scaling Action-Free Video Latent Prediction for Transferable Representations in End-to-End Autonomous Driving

Xingtai Gui1,*† Xubin Zhou2,* Yucheng Zhou1,* Dongqian Guo1 Jiahao Gong3 Feiyang Tan3 Jianbing Shen1,‡

1 SKL-IOTSC, Department of AI, University of Macau 2 Harbin Institute of Technology 3 Afari Intelligent Drive

* Equal contribution.† Work done during the internship at Afari Intelligent Drive.‡ Corresponding author.

Abstract

End-to-end autonomous driving policies require effective visual representations that capture both spatial scene structure and temporal dynamics. Video latent prediction has emerged as an effective paradigm for learning spatiotemporal visual representations, yet it remains unclear whether scaling this objective with action-free driving videos can yield representations that benefit end-to-end planning. We study action-free video latent prediction as a scalable pretraining objective and introduce ScaleDrive, a driving-oriented framework pretrained exclusively on large-scale internet open-world driving videos, without action annotations or benchmark-specific data. We keep the pretrained video encoder frozen during downstream end-to-end planning training to isolate the transferability of representations learned from internet driving videos. Ego-motion and depth probes show that scaling action-free video pretraining progressively strengthens the temporal and geometric information encoded in the frozen representation. We systematically study the visual representation transfer across different end-to-end driving policies and closed-loop benchmarks, where ScaleDrive consistently improves downstream performance and demonstrates that scaling action-free video latent prediction yields representations that generalize across both planning policy architectures and evaluation scenarios and protocols.

Overview of ScaleDrive: action-free video latent prediction followed by frozen encoder transfer to driving policies
ScaleDrive learns from action-free internet driving videos and transfers a frozen video representation to downstream planning policies.

Results comparison

NAVSIMv1

NAVSIMv1 planning comparison table; ScaleDrive reaches 93.8 PDMS

NAVSIMv2

NAVSIMv2 planning comparison table; ScaleDrive reaches 90.5 EPDMS

Bench2Drive

Bench2Drive comparison table of TFv6 with BEV, V-JEPA2, and ScaleDrive representations

HUGSIM

HUGSIM comparison table showing Route Completion and HD-Score across scenarios

Transfer across planners

NAVSIMv1 PDMS; each ScaleDrive result is compared with the corresponding planner baseline.

PlannerBaselineWith ScaleDriveGain
LTF83.886.2+2.4
DiffusionDrive86.688.1+1.5
DrivoR93.193.4+0.3
ChainFlow92.693.2+0.6

Closed-loop videos

Paired clips compare ScaleDrive with the V-JEPA2 baseline in the same scenes.

Scene 001

ScaleDrive
V-JEPA2 baseline

Scene 002

ScaleDrive
V-JEPA2 baseline

Scene 003

ScaleDrive
V-JEPA2 baseline

Scene 004

ScaleDrive
V-JEPA2 baseline

Code

Coming soon.