Musical NAO

Musical NAO: Instrument Playing Humanoid Using Monocular Vision

Papakonstantinopoulos, Ioannis (TU Delft Electrical Engineering, Mathematics and Computer Science)

Hindriks, Koen (mentor)

Delft University of Technology

2018-11-28

Humanoid robots are becoming part of modern societies mainly involving the education and entertainment fields. Therefore, providing techniques that enable a humanoid to autonomously play instruments like the metallophone we enrich its usefulness.

In this work we present an approach for making humanoid robot NAO play the metallophone. Specifically, we provide an application which can be used by everyone owned a NAO robot. In our approach, no markers are necessary to be attached on the robot, making the procedure of playing music, seems natural. We propose an efficient real-time approach for recognizing and estimate the metallophone in 3D space using a single camera. Our vision approach performs well in camera displacements, illumination and aspect changes.

For accurate playing, the robot automatically validates the computed IK configurations based on a vision-based robot control approach and adapts its arm configurations if it is necessary. To achieve this, we rely on the estimated pose of the musical instrument and the detection of the beaters in 2D, based on natural features and then we apply inverse kinematics. Concerning kinematics, we use singularity-robust approach by pre-computing forward kinematics which are used to efficiently solve inverse kinematics (IK) problems.

In our approach we limited on moving only NAO's arms for playing the metallophone. Therefore, we developed a human-robot interaction phase in which the robot uses its perception about the position of the instrument in 3D space and it guides a human in order to place it in a reachable for the robot position. Moreover, in our system we provide a simple way to add songs, which NAO can immediately play.

Humanoid
NAO
Monocular Vision
NAO kinematics
Monocular Visual-Servoing

http://resolver.tudelft.nl/uuid:03e04cdc-34cd-42a5-93b5-a26457c07d03

2018-11-29

Student theses

master thesis

© 2018 Ioannis Papakonstantinopoulos