Extracting concepts and understanding relationships from videos is essential in Video-Based Design (VBD), where videos serve as a primary medium for exploration but require significant effort in managing meta-information. Mind maps, with their ability to visually organize complex data, offer a promising approach for structuring and analysing video content. Recent advancements in Large Language Models (LLMs) provide new opportunities for meta-information processing and visual understanding in VBD, yet their application remains underexplored. This study recruited 28 VBD practitioners to investigate the use of prompt-tuned LLMs for generating mind maps from ethnographic videos. Comparing LLM-generated mind maps with those created by professional designers, we evaluated rated scores, design effectiveness, and user experience across two contexts. Findings reveal that LLMs effectively capture central concepts but struggle with hierarchical organization and contextual grounding. We discuss trust, customization, and workflow integration as key factors to guide future research on LLM-supported information mapping in VBD.

The rise of large language models for client-facing conversational search in healthcare necessitates evaluation frameworks that enable the assessment and comparison of these tools. Most such frameworks centre around the automated calculation of performance-related metrics and benchmarks. Though necessary, this focus fails to account for the human factors that impact the development, use, and adoption of these systems, as well as the factors specific to the healthcare context. Human evaluation frameworks attempt to address these drawbacks, but few such frameworks have been developed so far, and even fewer are those based on expert insight. In this work, we conduct semi-structured interviews with eleven healthcare professionals in health lifestyle care. From these interviews, we contribute a two-part healthcare domain expert evaluation framework, (K) Knowledge and (I) Interaction, which organises seven evaluation metrics. Our results reveal key understudied metrics for evaluation like (I1) Context-Seeking, (I2) Empathy, and (I3) Trustworthiness.

Ideation is a critical component of video-based design (VBD), where videos serve as the primary medium for design exploration and inspiration. The emergence of generative AI offers considerable potential to enhance this process by streamlining video analysis and facilitating idea generation. In this paper, we present DesignMinds, a prototype that integrates a state-of-the-art Vision-Language Model (VLM) with a context-enhanced Large Language Model (LLM) to support ideation in VBD. To evaluate DesignMinds, we conducted a between-subject study with 35 design practitioners, comparing its performance to a baseline condition. Our results demonstrate that DesignMinds significantly enhances the flexibility and originality of ideation, while also increasing task engagement. Importantly, the introduction of this technology did not negatively impact user experience, technology acceptance, or usability.

The fast evolving large language models (LLMs), powered by generative pre-trained transformers (GPTs), have shown revolutionary potential to transform many fields beyond natural language processing. They generate new opportunities for innovation in engineering design and manufacturing. Design and manufacturing represent critical research domains where knowledge has always been deeply embedded in engineering designers, manufacturing engineers, and technicians. LLMs can analyze, retrieve, and generate vast amount of knowledge and concepts, presenting exciting possibilities for automating tasks such as design ideation, knowledge extraction, and specification generation in design and manufacturing. [...]

Ideation is a critical component of video-based design (VBD), where videos serve as the primary medium for design exploration and inspiration. The emergence of generative AI offers considerable potential to enhance this process by streamlining video analysis and facilitating idea generation. In this paper, we present DesignMinds, a prototype that integrates a state-of-the-art Vision-Language Model (VLM) with a context-enhanced Large Language Model (LLM) to support ideation in VBD. To evaluate DesignMinds, we conducted a between-subject study with 35 design practitioners, comparing its performance to a baseline condition. Our results demonstrate that DesignMinds significantly enhances the flexibility and originality of ideation, while also increasing task engagement. Importantly, the introduction of this technology did not negatively impact user experience, technology acceptance, or usability.

Appropriate trust, trust which aligns with system trustworthiness, in Artificial Intelligence (AI) systems has become an important area of research. However, there remains debate in the community about how to design for appropriate trust. This debate is a result of the complex nature of trust in AI, which can be difficult to understand and evaluate, as well as the lack of holistic approaches to trust. In this paper, we aim to clarify some of this debate by operationalising appropriate trust within the context of the Human-Centred AI Design (HCD) process. To do so, we organised three workshops with 13 participants total from design and development backgrounds. We carried out design activities to stimulate discussion on appropriate trust in the HCD process. This paper aims to help researchers and practitioners understand appropriate trust in AI through a design lens by illustrating how it interacts with the HCD process.

Open hardware and the need for ecologically valid measurements drive the Electroencephalography (EEG) democratization movement—EEG has been steadily transcending the boundaries of clinical research, making its way into interdisciplinary fields. In Human-Computer Interaction (HCI), EEG is used to measure cognitive workload and infer cognitive processes for building cognition-aware systems. We describe and evaluate our BCIglass prototype where EEG electrodes are embedded in the frame of a mainstream Head-Mounted Display (HMD) to create a skull-peripheral topology. We devised a lab study with 34 participants who completed seven established cognitive tasks. Then, we conducted a pilot field study with one participant to test BCIglass in everyday-life settings. Our findings demonstrate that BCIglass captures EEG activity in a manner comparable to a research-grade EEG-cap system. Our topology infers the cognitive task at hand, and the underlying cognitive process(es) by proxy, with an accuracy of ∼80% and only three electrodes at the skull periphery. Embedding EEG electrodes in lightweight HMDs represents a promising approach in the quest to achieve ubiquitous brain-computer interfacing in real-world settings.

In the shift towards human-centered manufacturing, our two-year longitudinal study investigates the real-world impact of deploying Cognitive Assistants (CAs) in factories. The CAs were designed to facilitate knowledge sharing among factory operators. Our investigation focused on smartphone-based voice assistants and LLM-powered chatbots, examining their usability and utility in a real-world factory setting. Based on the qualitative feedback we collected during the deployments of CAs at the factories, we conducted a thematic analysis to investigate the perceptions, challenges, and overall impact on workflow and knowledge sharing.

Our results indicate that while CAs have the potential to significantly improve efficiency through knowledge sharing and quicker resolution of production issues, they also introduce concerns around workplace surveillance, the types of knowledge that can be shared, and shortcomings compared to human-to-human knowledge sharing. Additionally, our findings stress the importance of addressing privacy, knowledge contribution burdens, and tensions between factory operators and their managers.

Video-Based Design (VBD) is a design methodology that utilizes video as a primary tool for understanding user interactions, prototyping, and conducting research to enhance the design process. Artificial Intelligence (AI) can be instrumental in video-based design by analyzing and interpreting visual data from videos to enhance user interaction, automate design processes, and improve product functionality. In this study, we explore how AI can enhance professional video-based design with a State-of-the-Art (SOTA) deep learning model. We developed a prototype annotation platform (MarkupLens) and conducted a between-subjects eye-tracking study with 36 designers, annotating videos with three levels of AI assistance. Our findings indicate that MarkupLens improved design annotation quality and productivity. Additionally, it reduced the cognitive load that designers exhibited and enhanced their User Experience (UX). We believe that designer-AI collaboration can greatly enhance the process of eliciting insights in video-based design.

Recent advances in natural language processing enable more intelligent ways to support knowledge sharing in factories. In manufacturing, operating production lines has become increasingly knowledge-intensive, putting strain on a factory's capacity to train and support new operators. This paper introduces a Large Language Model (LLM)-based system designed to retrieve information from the extensive knowledge contained in factory documentation and knowledge shared by expert operators. The system aims to efficiently answer queries from operators and facilitate the sharing of new knowledge. We conducted a user study at a factory to assess its potential impact and adoption, eliciting several perceived benefits, namely, enabling quicker information retrieval and more efficient resolution of issues. However, the study also highlighted a preference for learning from a human expert when such an option is available. Furthermore, we benchmarked several commercial and open-sourced LLMs for this system. The current state-of-the-art model, GPT-4, consistently outperformed its counterparts, with open-source models trailing closely, presenting an attractive option given their data privacy and customization benefits. In summary, this work offers preliminary insights and a system design for factories considering using LLM tools for knowledge management.

Emoji have become an essential part of modern communication, helping to convey emotions and tone quickly and concisely. Emoji used by humans and Intelligent Agents (IA) have been shown to affect people’s decision making intentions, suggesting they could be used to manipulate users to follow their advice. We present a mixed-methods crowdsourcing study (N = 194) that shows that adherence to an IA’s recommendation and user experience are not affected by emoji when used in a positive, collaborative way. However, we demonstrate that explanations provided by an IA do increase adherence to its recommendation.

Many industries face the challenge of capturing workers' knowledge to share it, particularly tacit knowledge. The operation of complex systems such as a manufacturing line is knowledge-intensive. Considering this knowledge's breadth and dynamic nature, existing knowledge-sharing solutions are inefficient and resource intensive. Conversational user interfaces are an efficient way to convey information that mimics how humans share knowledge; however, we know little about how to design them specifically for knowledge sharing, especially regarding tacit knowledge. In this work, we present an intelligent assistant that we have developed to support the elicitation of tacit knowledge from workers through systematic reflection. The system can interact with workers by voice or text and generate visualizations of shop floor data to support reflective prompts.

As agile manufacturing expands and workforce mobility increases, the importance of efficient knowledge transfer among factory workers grows. Cognitive Assistants (CAs) with Large Language Models (LLMs), like GPT-3.5, can bridge knowledge gaps and improve worker performance in manufacturing settings. This study investigates the opportunities, risks, and user acceptance of LLM-powered CAs in two factory contexts: textile and detergent production. Several opportunities and risks are identified through a literature review, proof-of-concept implementation, and focus group sessions. Factory representatives raise concerns regarding data security, privacy, and the reliability of LLMs in high-stake environments. By following design guidelines regarding persistent memory, real-time data integration, security, privacy, and ethical concerns, LLM-powered CAs can become valuable assets in manufacturing settings and other industries.

Virtual Reality (VR) technology provides the elderly, and people with dementia, the opportunity to reminisce by exploring places outside their (care) home, free from age-related (physical) restrictions. However, the elderly are particularly vulnerable to overstimulation. Irresponsible VR design can cause stress and anxiety, potentially even exacerbating cognitive decline, and diminishing well-being. We present an electromyography (EMG) driven stress- and emotion-adaptive VR environment for the elderly that provides an immersive but controlled experience targeted at preventing negative emotions. We report our results and insights from a pilot study with elderly participants (N=3). Our system detects and mitigates signs of stress and negative emotions while promoting pleasant recollections.

Operating a complex and dynamic system, such as an agile manufacturing line, is a knowledge-intensive task. It imposes a steep learning curve on novice operators and prompts experienced operators to continuously discover new knowledge, share it, and retain it. In practice, training novices is resource-intensive, and the knowledge discovered by experts is not shared effectively. To tackle these challenges, we developed an AI-powered pervasive system that provides cognitive augmentation to users of complex systems. We present an AI cognitive assistant that provides on-the-job training to novices while acquiring and sharing (tacit) knowledge from experts. Cognitive support is provided as dialectic recommendations for standard work instructions, decision-making, training material, and knowledge acquisition. These recommendations are adjusted to the user and context to minimize interruption and maximize relevance. In this article, we describe how we implemented the cognitive assistant, how it interacts with users, its usage scenarios, and the challenges and opportunities.

Learning to operate a complex system, such as an agile production line, can be a daunting task. The high variability in products and frequent reconfigurations make it difficult to keep documentation up-to-date and share new knowledge amongst factory workers. We introduce CLAICA, a Continuously Learning AI Cognitive Assistant that supports workers in the aforementioned scenario. CLAICA learns from (experienced) workers, formalizes new knowledge, stores it in a knowledge base, along with contextual information, and shares it when relevant. We conducted a user study with 83 participants who performed eight knowledge exchange tasks with CLAICA, completed a survey, and provided qualitative feedback. Our results provide a deeper understanding of how prior training, context expertise, and interaction modality affect the user experience of cognitive assistants. We draw on our results to elicit design and evaluation guidelines for cognitive assistants that support knowledge exchange in fast-paced and demanding environments, such as an agile production line.

The contemporary digitalization of the sports experience brought new challenges for the HCI community. HCI researchers started exploring how mobile and wearable devices could support the physical, social, and environmental aspects of sports, while technological transformations like the metaverse, inbodied technologies, and AI have recently paved the way for augmented humans, esports, new forms of sociality, and new ways to engage the sports audience. In this preface, we present the papers accepted to the workshop Net Trends in HCI and Sports, held in conjunction with MobileHCI ‘22, which precisely attempted to deal with the recent advancements in technology used in the sports domain.

Conversational agents (CA) occasionally fail to understand the user's intention or respond inappropriately due to natural language complexity. These conversational breakdowns can happen because of low intent and entity prediction confidence scores. A promising repair strategy in such cases is that the CA proposes to users likely alternatives to proceed. If one of these options matches the user's intention, the breakdown is repaired successfully. We propose that successful repairs should be followed by a learning mechanism to minimize future breakdowns. After a successful repair, the CA, user, or both can learn each other's specific phrasing. This prevents similar phrasings from causing reoccurring breakdowns. We compared user preferences for these learning mechanisms in a scenario-based study with manufacturing workers (). Our result showed that users first prefer to share the learning burden with the CA (61.3%), followed by entirely outsourcing the learning burden to the CA (60.7%) as opposed to themselves.

Maintaining a complex system, such as a modern production line, is a knowledge-intensive task. Many firms use maintenance reports as a decision support tool. However, reports are often poor quality and tedious to compile. A Conversational User Interface (CUI) could streamline the reporting process by validating the user's input, eliciting more valuable information, and reducing the time needed. In this paper, we use a Technology Probe to explore the potential of a CUI to create instructional maintenance reports. We conducted a between-groups study (N = 24) in which participants had to replace the inner tube of a bicycle tire. One group documented the procedure using a CUI while replacing the inner tube, whereas the other group compiled a paper report afterward. The CUI was enacted by a researcher according to a set of rules. Our results indicate that using a CUI for maintenance reports saves a significant amount of time, is no more cognitively demanding than writing a report, and results in maintenance reports of higher quality.

Over the last 15 years, we have witnessed a digitalization of the sports experience, i.e., many sports have been enhanced by digital and wearable devices. The centrality of the human body and the different contexts where sports can be practiced have led HCI research to explore how mobile and wearable devices could support the physical, social, and environmental aspects of sports disciplines. Yet, the field of HCI & sports continues to evolve under the push of new technological developments and events affecting people worldwide, such as the Covid-19 pandemic and climate change. Technological advancements like the metaverse, inbodied technologies, and AI have paved the way for augmented humans, esports, new forms of sociality, and new ways to engage the audience. Likewise, contextual factors push sports trends in two opposite directions simultaneously: on the one hand, they foster the indoorisation and individualization of sports; on the other hand, they encourage practicing sports outdoors and taking advantage of the "restorative environment"of nature. With this workshop, we would like to invite the MobileHCI community to discuss the current trends in portable technologies for sports and trace future directions for HCI research in this field.