Joint video-action models learn a shared representation to predict future video frames and robot actions, with decoupled heads that allow the video head to be dropped during inference for real-time policy execution. However, this decoupling means the two heads are trained independently against ground truth, with no mechanism for one head’s predictions to inform the other. We observe that the video head produces structurally incorrect robot poses, especially during dynamic motion. We introduce Kinematic Forcing (KF), a simple training objective that grounds this shared representation in the robot’s physical structure, using the action head’s own predicted pose as the source of that structure. From the action head’s predicted pose, we recover joint angles via inverse kinematics, render the robot with its URDF model, and composite it into the scene with an inpainter. We encode this kinematic composite using a frozen DINOv2 encoder and force the backbone’s shared representation to align with the features at robot-token positions using a physics-grounded, action-derived target. The entire rendering and alignment pipeline is training-only and removed at inference, leaving the deployed model identical to the baseline. Across simulated (WidowX, Google Robot) and real-world (xArm7) tasks, KF improves video prediction quality at robot regions, with the largest gains under fast motion, while maintaining downstream policy success.
... Read more