Within Reach
Enabling Embodiment-Aware Affordance Prediction Through Synthetic Dataset Generation
C.J. de Dood (TU Delft - Mechanical Engineering)
Javier Alonso-Mora – Mentor (TU Delft - Mechanical Engineering)
Albert Gassol Puigjaner – Mentor
J.F.P. Kooij – Graduation committee member (TU Delft - Mechanical Engineering)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Affordance is a relational property that depends on both an object and the agent acting on it, yet most 3D affordance detection methods predict a single geometry-only heatmap that ignores which robot will act. We make embodiment an explicit conditioning variable for affordance prediction. Using Isaac Sim, we record contact between three morphologically distinct robots, the ANYmal-C quadruped, the Ridgeback-Franka mobile manipulator, and the Unitree H1 humanoid, and a set of household objects, and aggregate the raw contacts into per-point, per-body-part maps annotated with the displacement and rotation each contact creates. On this dataset we train a PointNet++-based model that conditions on object geometry and a natural-language description of the robot through cross-attention, producing different affordance maps for the same object depending on the acting morphology. The model recovers the reachability mask, body-part class, and interaction pose across embodiments, with near-convex objects such as bottles and vases the most predictable and geometrically varied chairs and lamps the hardest. The results show that embodiment is a learnable conditioning variable for affordance prediction, and provide a dataset and baseline for the task.
help_outline