Data-informed design experiments

Abstract

Our society is facing a number of great challenges which will require all of us to significantly change our lifestyle in the coming years. To support people in those transitions, next to systemic changes, new design interventions have to be crafted that intentionally aim to redirect behaviour for the common good. As changing behaviour intentionally comes with great responsibility, social and behavioural design calls for sound and deliberate design and evaluation. However, changing behaviour is something that takes a long time to materialise durably and thus conventional qualitative user-centered approaches to evaluation may not be the most suitable. On the other hand, quantitative approaches measuring only the outcomes of the behaviour do not provide detailed insight into the performance of the intervention. This thesis investigated how integrating various sources of qualitative and quantitative data on a behavioural situation during evaluation contributes to critically assessing and anticipating the effectiveness and appropriateness of an intervention aimed at changing behaviour. Studies that investigate the effectiveness of interventions often find that the effects induced in the short-term were not sustained (Abrahamse et al., 2005). A theoretical model was developed that conceptualises the underlying mechanism of this observation as the transition of the design being efficacious (works when people receive an intervention) to being effective (works when people are offered an intervention), which is influenced by the appropriateness of the intervention to its context. This appropriateness can be further operationalised into three types: aesthetic, moral and systemic appropriateness. The relations between the effectiveness and appropriateness were experimentally explored through deploying research artefacts in the context of the end-user. In this experiment an interactive bedlight and a chatbot were evaluated on their effectiveness and appropriateness in achieving the intended effect, ‘adopting regular sleep and wake times’, while at the same time understanding their performance in relation to ‘sleeping better’ and ‘balancing sleep and other practices’. In the study several perspectives on the situation were collected and integrated: sensor data from an ecology of instrumented things, data from interviews with the participant before and after using the intervention, and data generated through the interaction with the interventions. Integrating these perspectives resulted in concurrent insight into the performance of the intervention as it is now and potential elements for improvement. Although some perspectives are more attuned to the efficacy, and others more to the appropriateness—in general the integration of perspectives contribute to a holistic understanding of the situation as the individual perspectives filled in each other’s blind spots. Through assessing the efficacy and appropriateness of the intervention the long-term effectiveness can be anticipated. This could nurture new methods for evaluation where data is informing the design process in order to assess how the mechanism in the intervention performs to decide what the right level of persuasive influence of the intervention is and assess whether the intervention is proportionate.