Current developments in Artificial Intelligence (AI) led to a resurgence of Explainable AI (XAI). New methods are being researched to obtain information from AI systems in order to generate explanations for their output. However, there is an overall lack of valid and reliable evaluations of the effects on users' experience of, and behavior in response to explanations. New XAI methods are often based on an intuitive notion what an effective explanation should be. Rule- and example-based contrastive explanations are two exemplary explanation styles. In this study we evaluate the effects of these two explanation styles on system understanding, persuasive power and task performance in the context of decision support in diabetes self-management. Furthermore, we provide three sets of recommendations based on our experience designing this evaluation to help improve future evaluations. Our results show that rule-based explanations have a small positive effect on system understanding, whereas both rule- and example-based explanations seem to persuade users in following the advice even when incorrect. Neither explanation improves task performance compared to no explanation. This can be explained by the fact that both explanation styles only provide details relevant for a single decision, not the underlying rational or causality. These results show the importance of user evaluations in assessing the current assumptions and intuitions on effective explanations.@en

People of low literacy could benefit from automated support when learning about societal participation. We design an Embodied Conversational Agent (ECA) ‘coach’ that can provide effective learning support to low-literate learners, develop a prototype virtual learning environment, and evaluate this prototype with low-literate end users. First, we inventory the learning support benefits of ECA coaching. Second, we update existing requirements to better specify functional demands for the coach ECA. Third, we write use cases and develop the prototype. Finally, we evaluate the prototype with low-literate users in a mixed-method within-subjects experiment. Results show that the coach influences the subjective learning experience: Participants report higher positive affect, higher user-system engagement, and increased self-efficacy regarding online banking. These results particularly apply to the domain of challenging information skills exercises. Caveats apply: One of four exercises was significantly more difficult than the other three; and coach support rules were not clearly formalized.@en