In recent years, a large variety of online communication tools have emerged, including social Virtual Reality (VR) platforms for interacting in a virtual world with participants being represented as virtual avatars. Given their popularity, an active area of research focuses on improving the user experience in these virtual experiences. To enable experimentation at large scale on online platforms, it is however essential to collect behavioural data (e.g. movements and audio information). In this work, we present a toolchain that enables the running of experiments using a modified version of the social VR platform Mozilla Hubs. Specifically, our toolkit enables collection and tracking or user positions and movements at a central location, enabling fine-grained analysis of user behaviour during a social VR experience.

Eliciting and capturing drivers' affective responses in a realistic outdoor setting with pedestrians poses a challenge when designing in-vehicle, empathic interfaces. To address this, we designed a controlled, outdoor car driving circuit where drivers (N=27) drove and encountered pedestrian confederates who performed non-verbal positive or non-positive road crossing actions towards them. Our findings reveal that drivers reported higher valence upon observing positive, non-verbal crossing actions, and higher arousal upon observing non-positive crossing actions. Drivers' heart signals (BVP, IBI and BPM), skin conductance and facial expressions (brow lowering, eyelid tightening, nose wrinkling, and lip stretching) all varied significantly when observing positive and non-positive actions. Our car driving study, by drawing on realistic driving conditions, further contributes to the development of in-vehicle empathic interfaces that leverage behavioural and physiological sensing. Through automatic inference of driver affect resulting from pedestrian actions, our work can enable novel empathic interfaces for supporting driver emotion self-regulation.

Capturing drivers’ affective responses given driving context and driver-pedestrian interactions remains a challenge for designing in-vehicle, empathic interfaces. To address this, we conducted two lab-based studies using camera and physiological sensors. Our first study collected participants’ (N = 21) emotion self-reports and physiological signals (including facial temperatures) toward non-verbal, pedestrian crossing videos from the Joint Attention for Autonomous Driving dataset. Our second study increased realism by employing a hybrid driving simulator setup to capture participants’ affective responses (N = 24) toward enacted, non-verbal pedestrian crossing actions. Key findings showed: (a) non-positive actions in videos elicited higher arousal ratings, whereas different in-video pedestrian crossing actions significantly influenced participants’ physiological signals. (b) Non-verbal pedestrian interactions in the hybrid simulator setup significantly influenced participants’ facial expressions, but not their physiological signals. We contribute to the development of in-vehicle empathic interfaces that draw on behavioral and physiological sensing to in-situ infer driver affective responses during non-verbal pedestrian interactions.

Artistic co-creation empowers communities to shape their narratives, however HCI research does not support this multifaceted discussion and reflection process. In the context of community opera, we consider how to support co-creation through the design, implementation, and initial evaluation of the Co-Creation Space (CCS) to help community artists 1) generate raw artistic ideas, and 2) discuss and reflect on the shared meaning of those ideas. This work describes our user-centered process to gather requirements and design the tool, and validates its' usability with 6 community opera participants. Our findings support the value of our tool for group discussion and personal reflection during the creative process.

This work presents the Co-Creation Space, a multilingual platform for professional and community artists to 1) generate raw artistic ideas, and 2) discuss and reflect on the shared meaning of those ideas. The paper describes the architecture and the technology behind the platform, and how it was used to facilitate the communication process during several user trials. By supporting ideation sessions around media items guided by a facilitator and allowing users to express themselves and be part of the creation of an artistic product, participants were enabled to access new cultural spaces and be part of the creative process.

Automatically inferring drivers' emotions during driver-pedestrian interactions to improve road safety remains a challenge for designing in-vehicle, empathic interfaces. To that end, we carried out a lab-based study using a combination of camera and physiological sensors. We collected participants' (N=21) real-time, affective (emotion self-reports, heart rate, pupil diameter, skin conductance, and facial temperatures) responses towards non-verbal, pedestrian crossing videos from the Joint Attention for Autonomous Driving (JAAD) dataset. Our findings reveal that positive, non-verbal, pedestrian crossing actions in the videos elicit higher valence ratings from participants, while non-positive actions elicit higher arousal. Different pedestrian crossing actions in the videos also have a significant influence on participants' physiological signals (heart rate, pupil diameter, skin conductance) and facial temperatures. Our findings provide a first step toward enabling in-car empathic interfaces that draw on behavioural and physiological sensing to in situ infer driver emotions during non-verbal pedestrian interactions.

Voice is a rich modality for conveying emotions, however emotional prosody production can be situationally or medically impaired. Since thermal displays have been shown to evoke emotions, we explore how thermal stimulation can augment perception of neutrally-spoken voice messages with affect. We designed ThermalWear, a wearable on-chest thermal display, then tested in a controlled study (N=12) the effects of fabric, thermal intensity, and direction of change. Thereafter, we synthesized 12 neutrally-spoken voice messages, validated (N=7) them, then tested (N=12) if thermal stimuli can augment their perception with affect. We found warm and cool stimuli (a) can be perceived on the chest, and quickly without fabric (4.7-5s) (b) do not incur discomfort (c) generally increase arousal of voice messages and (d) increase / decrease message valence, respectively. We discuss how thermal displays can augment voice perception, which can enhance voice assistants and support individuals with emotional prosody impairments.

While traditional live-broadcasting is typically comprised of a handful of well-defined workflows, these become insufficient when targeting multiple screens and interactive companion devices on the viewer side. In this case study, we describe the development of an end-to-end system enabling immersive and interactive experiences using an object-based broadcasting approach. We detail the deployment of this system during the live broadcast of the FA Cup Final at Wembley Stadium in London in May 2018. We also describe the trials and interviews we ran in the run-up to this event, the infrastructure we used, the final software developed for controlling and rendering on-screen graphics and the system for generating and configuring the live broadcast-objects. In this process, we learned about the workflows inside an OB truck during live productions through an ethnographic study and the challenges involved in running an object-based broadcast over the Internet, which we discuss alongside other gained insights.

Millions of photos are shared online daily, but the richness of interaction compared with face-to-face (F2F) sharing is still missing. While this may change with social Virtual Reality (socialVR), we still lack tools to measure such immersive and interactive experiences. In this paper, we investigate photo sharing experiences in immersive environments, focusing on socialVR. Running context mapping (N=10), an expert creative session (N=6), and an online experience clustering questionnaire (N=20), we develop and statistically evaluate a questionnaire to measure photo sharing experiences. We then ran a controlled, within-subject study (N=26 pairs) to compare photo sharing under F2F, Skype, and Facebook Spaces. Using interviews, audio analysis, and our questionnaire, we found that socialVR can closely approximate F2F sharing. We contribute empirical findings on the immersiveness differences between digital communication media, and propose a socialVR questionnaire that can in the future generalize beyond photo sharing.

Multiscreen TV viewing refers to a spectrum of media productions that can be watched on TV screens and companion screens (e.g., smartphones and tablets). TV production companies are now promoting an interactive and engaging way of viewing TV by offering tailored applications for TV programs. However, viewers are demotivated to install dozens of applications and switch between them. This is one of the obstacles that hinder companion screen applications from reaching mass audiences. To solve this, TV production companies need a standard process for producing multiscreen content, allowing viewers to follow all kinds of programs in one single application. This paper proposes a new object-based production platform for authoring programs for multiscreen. The platform consists of two parts: the preproduction tool and the live editing tool. To evaluate whether the proposed workflow is appropriate, validation interviews were conducted with professionals in the TV broadcasting industry. The professionals were positive about the proposed new workflow, indicating that the platform allows for preparations at the preproduction stage and reduces the workload during the live broadcasting. They see as well its potential to adapt to the current production workflow.

Multiscreen TV viewing refers to a spectrum of media productions that can be watched using TV and companion screens such as smartphones and tablets. In the last several years, companies are creating companion applications to enrich the TV viewing experience, but viewers are demotivated to consume them because they have to download dozens of second screen applications. This paper proposes to integrate the creation of companion screen content in a single object-based preproduction tool. It identifies, from the perspective of TV production professionals, the best paradigm and the needed features to support content authoring for multiscreen viewing experiences.

This paper evaluates the influence of an additional visual aesthetic layer on the experience of concert goers during a live event. The additional visual layer incorporates musical features as well as bio-sensing data collected during the concert, which is coordinated by our audience engagement monitoring technology. This technology was used during a real Jazz concert. The collected measurements were used in an experiment with 32 participants, where two different forms of visualization were compared: one factoring in music amplitude, audience engagement collected by the sensors and the dynamic atmosphere of the event, the other one purely relying on the beat of the music. The findings indicate that the visual layer could add value to the experience if used during a live concert, providing a higher level of immersion and feeling of togetherness among the audience.

The CWI-ADE2016 Dataset is a collection of more than 40 million Bluetooth Low Energy (BLE) packets and of 14 million accelerometer and temperature samples generated by wristbands that people wore in a nightclub. The data was gathered during Amsterdam Dance Event 2016 in an exclusive club experience curated around human senses, which leveraged technology as a bridge between the club and the guests. Each guest was handed a custom-made wristband with a BLE-enabled device that broadcast movement, temperature and other sensor readings. A network of Raspberry Pi receivers deployed for the occasion captured broadcast packets from wristbands and any other BLE device in the environment. This data provides a full picture of the performance of the real life deployment of a sensing infrastructure and gives insights to designing sensing platforms, understanding networks and crowds behaviour or studying opportunistic sensing. This paper describes an analysis of this dataset and some examples of usage.

This article presents an end-to-end system for capturing physiological sensor data and visualising it on a real-time graphic dashboard and as part of an art installation. More specifically, it describes an event where the level of engagement of the audience was measured by means of Galvanic Skin Response (GSR) sensors and of the presenter through a sweater fitted with GSR, ECG and acceleration sensors. The gathered data was presented in real-time through a visualisation projected onto a screen and a physical electro-mechanical installation, which would change the height of helium-filled balloons depending on the atmosphere in the auditorium. Thereby trying to create a tangible way of making the invisible visible.

Measuring the experience of audience of arts events is essential in the experience economy of this day and age, but it is a difficult task. The value of such information goes beyond evaluating the impact of the arts, as it can provide insights and feedback to enhance the work of artists and the experiences of other audience members. Through in-depth understanding of the needs of the providers and consumers of the arts, we progressively developed a biosensor infrastructure that was deployed in theaters. Over the years, we identified the challenges and issues related to developing and deploying a biosensor infrastructure in theaters. These collective experiences and identified issues were categorized into three main areas: processes, data, and system. A total of seven heuristics are developed across the three main areas. Processes place the stakeholders and audiences at the core of the research; data provides guidelines for data validity, collecting a variety of data, and supporting real-time data gathering; and systems covers the concurrency, scalability, deployment and feedback of the infrastructure. We believe that this set of heuristics forms the foundation for an adequate infrastructure to measure audience experience in the wild and it is a valuable source of guideline for future work.