Current visuomotor manipulators jointly train their perception-planning-action components simultaneously using an end-to-end framework to avoid hand-engineering components. Despite this, methods for human-to-robot object handover tasks require a perception component that segments the hand from the object, which can introduce error propagation. For this reason, this study investigates the applicability of an end-to-end framework that eliminates the need for hand-object segmentation in a simulated human-to-robot object handover task using HandoverSim.

To address this, a behavior cloning agent is used to convert camera input into RGB-D voxel space and output discretized 6-DoF manipulation to directly discover features for the handover task. This study introduces a framework that combines the behavior cloning agent with the HandoverSim, which allows experimenting with various training configurations. These configurations consist of experiments with: 1) expert demonstration data; 2) optimal camera setup; 3) handover objects; and 4) voxel-based RGB augmentation techniques.

The trained model is evaluated on its generalization to diverse handover conditions in the HandoverSim Benchmark. The results demonstrate that the behavior cloning agent can learn features for the handover task without requiring a perception component. The model learns the grasp-object relation whilst minimizing contact with the hand. Despite this, performance is limited by sparse training data and grasping accuracy.

With mobile robotics being applied for more and more complex applications, their autonomy should be preserved. While a lot of research is performed into the direction of failure prediction for autonomous processes or systems, the field of mobile robots has received less attention. Proactive failure prediction for mobile robots is a useful tool to prevent unwanted downtime and undesired damages. This work attempts to fill this research gap by showing the applicability of anomaly detection methods for failure prediction in the field of mobile robots. Specifically, we employ an unsupervised Variational Autoencoder to predict failures in the operational data from the Discovery Collector, a manure cleaning robot developed by Lely Industries. We elaborately showcase the feature engineering steps which yield the best performance, provide the performance of three general datasets, and state promising next steps for root cause classification which is enabled by accurate failure prediction. All in all, our work shows that the use of feature offsets, calculated from desired values compared to actual values, enhances the model performance tremendously. The provided datasets showcase F1-scores ranging from 0.64-0.76, showing the proposed solution is able to solve the failure prediction problem in the field of mobile robots, while highlighting the encountered limitations for future improvement.

Large Language Models (LLMs) possess significant semantic knowledge about the world, making them valuable for high level control for robots through Vision-Language-Action (VLA) models. These models integrate an LLM to deduce semantic knowledge from a robot’s vision and natural language inputs, facilitating real-world actions. Despite their potential, VLA models are a relatively new research area, with applications mostly limited to simulations or household tasks and insufficient validation in broader contexts. This study aims to develop a LLM-Enhanced Robotic Affordance and Control System (LERACS) for robotic manipulation in applied cases such as the management and maintenance of electrical grid infrastructure. LERACS is designed to visually ground manipulable objects and decompose tasks based on user instructions within a human-robot interaction chat interface, using ChatGPT. A system validation and an AI user experiment were conducted to evaluate its effectiveness in interpreting and performing actions based on pre-made and synthetically generated user instructions. These assessed LERACS’ performance across various settings and instructions. Results indicate high success rates in environmental interpretation and task execution, with robust labeling accuracy, especially in complex settings. Feedback from the AI user experiment highlighted LERACS’ adaptability, identified areas for improvement, and demonstrated its practical utility across diverse settings and task complexities.