As technology continues to evolve at a rapid pace, robots are becoming an increasingly common sight in our daily lives.
Robots that work with humans need to adapt to a variety of users and tasks, and learn to optimise their behaviour. For non-specialist users to interact with such robots, the robot's learning process needs to be transparent through its behaviour. Reinforcement Learning (RL) is a promising learning method to achieve this adaptability. However, the behaviour generated by RL is not inherently transparent because of the exploration/exploitation trade-off that is needed to optimise a policy for a specific task.

A RL algorithm is Temporal Difference (TD) learning. In TD learning, the algorithm updates a Q-table to keep track of Q-values. Q-values represent the expected future rewards that the agent (the actor that decides what action to take) can receive by taking a specific action in a certain state. Calculating the Q-values involves a value called the Temporal Difference, which is the difference between the current Q-value with the received reward added and the Q-value for the future state and chosen action.

Emotions are a natural way of communicating intent and situational appraisal for humans. In this study, emotional expressions based on Temporal Differences were implemented as a means to increase the transparency of a robot's learning progress. The effects on the robot's learning progress, learning result, and user experience were analysed.

A between-subject experiment with 61 participants on the following three robot modes was performed: no emotions, simulated emotions, and simulated emotions with matching attribution (see Table \ref{table:robotModes}). The simulated emotions are hope, fear, joy, and distress, which were expressed by a humanoid robot. The robot mode with simulated emotions and matching attributions would explain for what task it was feeling hope or fear. The task was a simple task where a human teacher had to help a humanoid robot to learn to express three different colours based on human commands.

The results demonstrate minimal differences between these three conditions. This means that for simple tasks, emotional expressions grounded in RL do not have a significant effect, and thus do not help nor hurt. The findings are discussed, and it is proposed that emotion simulation is beneficial for tasks that are more complex, afford some robot autonomy, and for which the emotion is informative about how the user should influence the robot's actions to the benefit of the robot's policy.

In recent years there has been an increasing interest in the development of social robots for educational purposes. A presumed advantage is their physical presence in the learner’s referential world. However, it remains an open question whether this embodied presence of a robot can be utilized to support learning with robots’ gestures. The aim of our study was to program a social robot that tutors mathematics through verbal explanations supported by deictic, beat, and iconic gestures that align with explanations, and to analyse its effectiveness on learning outcomes. The learning outcomes entail cognitive outcomes and affective outcomes. A between-subject experiment was performed to research the effect of the robot’s use of supporting gestures on cognitive and affective learning outcomes compared to a robot that does not perform gestures or performs random gestures. In total, 78 children (Mage= 8.3 years) from Dutch primary schools participated. Results showed that the social robot effectively taught children mathematical equivalence by a significant increase in test scores between the pre- and post-test. No differences were found in test scores between participants in the supportive gesturing condition, the no gesturing condition, and the random gesturing condition. Participants showed high task and tutor engagement scores and head direction estimations revealed that participants were attentive towards the robot, but no differences were found between the three conditions. However, the gestures ‘helpfulness’ was rated significantly higher by participants in the supportive gesture condition than the random gesture condition, suggesting that children felt the gestures mattered.