WB
W.P. Brinkman
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
75 records found
1
Comparing Local LLM-Based Extraction of Stakeholder Values for Value Model Visualization in Deliberations
How effectively can local large language models extract information required to generate visualizations of stakeholder value models?
In stakeholder deliberation, it can be useful to give insights into stakeholders' values (such as privacy, safety, and fairness). Previously, conversational agents were built to give insight into such a value model. Yet, the question of whether visualizations can aid in the understanding of a value model still remains. As a first step, this paper proposes two visualizations of value models based on existing literature: radar charts and value cards. This paper argues why these visualizations can aid in the understanding of value models, but to create them, data needs to be extracted from transcripts. Therefore, we also compare three consumer-grade local LLMs (≤ 35B parameters) - Gemma4:e4b, Phi4-reasoning:14b, and Qwen3.6:35b - on their ability to extract data from deliberative transcripts necessary to generate these visualizations. Using local LLMs for this task can be beneficial, as using cloud-provided LLMs can lead to value profiles being built. The evaluated LLMs are found to have strong agreement (Cohen's κ ≥ 0.808) with human coders on extracting the values included in the transcripts; however, they have mixed agreement when ranking the values or assigning codes to their importance. When comparing textual justifications for value rankings and assigned importance codes, justifications between LLMs and humans may differ, but the textual justifications generally do not disagree on whether a value is important or not. When the local LLMs have to give a summary of the meaning of values, they are generally roughly similar to human summaries or those provided by the other LLMs, and in 38 to 46\% of cases, LLM summaries are nearly equivalent. These findings suggest that consumer-grade local LLMs are effective at identifying human values present in text, but struggle to code their importance, ordinal rank, and summarize their meaning. This makes them currently unsuitable to replace human coders without oversight.
...
In stakeholder deliberation, it can be useful to give insights into stakeholders' values (such as privacy, safety, and fairness). Previously, conversational agents were built to give insight into such a value model. Yet, the question of whether visualizations can aid in the understanding of a value model still remains. As a first step, this paper proposes two visualizations of value models based on existing literature: radar charts and value cards. This paper argues why these visualizations can aid in the understanding of value models, but to create them, data needs to be extracted from transcripts. Therefore, we also compare three consumer-grade local LLMs (≤ 35B parameters) - Gemma4:e4b, Phi4-reasoning:14b, and Qwen3.6:35b - on their ability to extract data from deliberative transcripts necessary to generate these visualizations. Using local LLMs for this task can be beneficial, as using cloud-provided LLMs can lead to value profiles being built. The evaluated LLMs are found to have strong agreement (Cohen's κ ≥ 0.808) with human coders on extracting the values included in the transcripts; however, they have mixed agreement when ranking the values or assigning codes to their importance. When comparing textual justifications for value rankings and assigned importance codes, justifications between LLMs and humans may differ, but the textual justifications generally do not disagree on whether a value is important or not. When the local LLMs have to give a summary of the meaning of values, they are generally roughly similar to human summaries or those provided by the other LLMs, and in 38 to 46\% of cases, LLM summaries are nearly equivalent. These findings suggest that consumer-grade local LLMs are effective at identifying human values present in text, but struggle to code their importance, ordinal rank, and summarize their meaning. This makes them currently unsuitable to replace human coders without oversight.
Extracting Value, Value Tension, and Points of Agreement From Deliberation
Can LLMs identify value, value tensions, and consensus points from multi-stakeholder deliberation transcripts?
Deliberation is a process in which people come together to discuss and find solutions to complex topics. Human moderators are expensive to employ, time-consuming, and can introduce their own biases into the moderation and summary of deliberative conversations. Large language models (LLMs), on the other hand, avoid obscuring minority opinions and bring groups together rather than dividing them. In this paper, we examine whether LLMs can extract values, value tensions, and consensus points from deliberative transcripts. We experiment with 3 different prompting strategies, zero-shot, few-shot, and chain of thought prompting, and 3 different LLM models, Gemma 2, Qwen, and Mistral. We evaluate the results using ground truth annotations and an LLM as a judge study. We found that all LLMs were capable of extracting the basic constructs by providing valid outputs to the prompts given. Mistral slightly outperformed the other models according to LLM-as-a-judge, whereas Gemma 2 achieved the highest F1 score. Chain of thought prompting outperformed the others, according to the LLM-as-a-judge, with few-shot prompting achieving the highest overall F1 score. We found that the interaction between the model and prompting strategy is highly dependent on the evaluation criteria with the correlation in results between LLM-as-a-judge and metric evaluation tending to be slightly negative.
...
Deliberation is a process in which people come together to discuss and find solutions to complex topics. Human moderators are expensive to employ, time-consuming, and can introduce their own biases into the moderation and summary of deliberative conversations. Large language models (LLMs), on the other hand, avoid obscuring minority opinions and bring groups together rather than dividing them. In this paper, we examine whether LLMs can extract values, value tensions, and consensus points from deliberative transcripts. We experiment with 3 different prompting strategies, zero-shot, few-shot, and chain of thought prompting, and 3 different LLM models, Gemma 2, Qwen, and Mistral. We evaluate the results using ground truth annotations and an LLM as a judge study. We found that all LLMs were capable of extracting the basic constructs by providing valid outputs to the prompts given. Mistral slightly outperformed the other models according to LLM-as-a-judge, whereas Gemma 2 achieved the highest F1 score. Chain of thought prompting outperformed the others, according to the LLM-as-a-judge, with few-shot prompting achieving the highest overall F1 score. We found that the interaction between the model and prompting strategy is highly dependent on the evaluation criteria with the correlation in results between LLM-as-a-judge and metric evaluation tending to be slightly negative.
How to design post-reflection dialogue from transcripts using the identified values, value tensions, and consensus points?
A deliberative approach to modeling retrospection ex post facto in multi-stakeholder decision-making scenarios
Large Language Models (LLMs) excel at natural language tasks, yet most contemporary systems and tool prioritize providing an answer over fostering reflection and deliberation. This research investigated whether LLM-based tools can generate post-reflection dialogue in multi-stakeholder decision making scenarios by using identified value tensions and points of contention found in transcripts.
A Deliberative AI approach was developed using publicly available transcripts and several open-sources LLMs. The generated reflective dialogue was subsequently evaluated through Synthetic Personae evaluators according to the five metrics established: safety, privacy, autonomy, societal well-being, and points of contention. Two different prompting strategies: single-turn and multi-turn were deployed to see if there were meaningful differences between the two.
The results indicated that the methodology can produce reflective dialogues that are perceived positively, exceeding the predefined success threshold. Furthermore, the iterative multi-turn interactions were found to improve perceived satisfaction compared to the single-turn approach on average.
Although limited to English language deliberations, the findings demonstrate the feasibility of using Deliberative AI to support reflection and, rather than proposing a universal solutions, this work provides a reproducible proof of concept that can be adapted based on future models, transcript contexts, and languages, motivating the development of more LLM-based deliberative systems.
...
A Deliberative AI approach was developed using publicly available transcripts and several open-sources LLMs. The generated reflective dialogue was subsequently evaluated through Synthetic Personae evaluators according to the five metrics established: safety, privacy, autonomy, societal well-being, and points of contention. Two different prompting strategies: single-turn and multi-turn were deployed to see if there were meaningful differences between the two.
The results indicated that the methodology can produce reflective dialogues that are perceived positively, exceeding the predefined success threshold. Furthermore, the iterative multi-turn interactions were found to improve perceived satisfaction compared to the single-turn approach on average.
Although limited to English language deliberations, the findings demonstrate the feasibility of using Deliberative AI to support reflection and, rather than proposing a universal solutions, this work provides a reproducible proof of concept that can be adapted based on future models, transcript contexts, and languages, motivating the development of more LLM-based deliberative systems.
...
Large Language Models (LLMs) excel at natural language tasks, yet most contemporary systems and tool prioritize providing an answer over fostering reflection and deliberation. This research investigated whether LLM-based tools can generate post-reflection dialogue in multi-stakeholder decision making scenarios by using identified value tensions and points of contention found in transcripts.
A Deliberative AI approach was developed using publicly available transcripts and several open-sources LLMs. The generated reflective dialogue was subsequently evaluated through Synthetic Personae evaluators according to the five metrics established: safety, privacy, autonomy, societal well-being, and points of contention. Two different prompting strategies: single-turn and multi-turn were deployed to see if there were meaningful differences between the two.
The results indicated that the methodology can produce reflective dialogues that are perceived positively, exceeding the predefined success threshold. Furthermore, the iterative multi-turn interactions were found to improve perceived satisfaction compared to the single-turn approach on average.
Although limited to English language deliberations, the findings demonstrate the feasibility of using Deliberative AI to support reflection and, rather than proposing a universal solutions, this work provides a reproducible proof of concept that can be adapted based on future models, transcript contexts, and languages, motivating the development of more LLM-based deliberative systems.
A Deliberative AI approach was developed using publicly available transcripts and several open-sources LLMs. The generated reflective dialogue was subsequently evaluated through Synthetic Personae evaluators according to the five metrics established: safety, privacy, autonomy, societal well-being, and points of contention. Two different prompting strategies: single-turn and multi-turn were deployed to see if there were meaningful differences between the two.
The results indicated that the methodology can produce reflective dialogues that are perceived positively, exceeding the predefined success threshold. Furthermore, the iterative multi-turn interactions were found to improve perceived satisfaction compared to the single-turn approach on average.
Although limited to English language deliberations, the findings demonstrate the feasibility of using Deliberative AI to support reflection and, rather than proposing a universal solutions, this work provides a reproducible proof of concept that can be adapted based on future models, transcript contexts, and languages, motivating the development of more LLM-based deliberative systems.
Scaling Deliberative-Quality Measurement with Large Language Models
Validating a Four-Model LLM Ensemble as a Coder for a Theory-Justified Three-Dimension Discourse Quality Index Sub-Codebook on UK House of Commons Debate
Evaluating the quality of public deliberation is a prerequisite for governing it on evidence rather than impression, yet the bottleneck is measurement. The Discourse Quality Index (DQI), the standard instrument in the field, requires trained human coders — slow, costly, and applied inconsistently across research teams — which means most empirical deliberation studies contain only a few thousand speeches. Scaling that measurement is the problem this paper addresses. Large language models (LLMs) can apply a structured rubric rapidly and uniformly, so the central question is whether an LLM coder is reliable enough on theory-grounded deliberative constructs to stand in for a trained human and raise the data-size ceiling. We make
two contributions. First, we construct a theory-justified three-dimension DQI sub- codebook — Level of Justification, Respect for Groups, and Counterarguments — defending each inclusion and exclusion from deliberative-democracy theory. Second,
we benchmark a four-model LLM-as-judge ensemble against two trained coders on
200 UK House of Commons public-safety debate acts. We report Gwet’s linear- weighted AC1: trained coders reach AC1 = 0.78, 0.94, and 0.48 on the three dimensions, and the ensemble reaches 0.74, 0.89, and 0.48 against the lead coder.
We also tested whether an LLM can substitute for a human coder, framed as an equivalence test: whether the paired difference in agreement falls within a ±0.10 band of the human–human baseline, judged by where its confidence interval lies rather than by a test against zero. The difference falls within 0.05 of the baseline
on every dimension, and its confidence interval lies inside the band on all three.
The results suggest that on justification and recognition the ensemble can serve as reliable independent measurement, and that on Counterarguments it can serve as a second coder in a doubly-coded design — where the binding constraint appears to
be the codebook anchor language rather than the model. ...
two contributions. First, we construct a theory-justified three-dimension DQI sub- codebook — Level of Justification, Respect for Groups, and Counterarguments — defending each inclusion and exclusion from deliberative-democracy theory. Second,
we benchmark a four-model LLM-as-judge ensemble against two trained coders on
200 UK House of Commons public-safety debate acts. We report Gwet’s linear- weighted AC1: trained coders reach AC1 = 0.78, 0.94, and 0.48 on the three dimensions, and the ensemble reaches 0.74, 0.89, and 0.48 against the lead coder.
We also tested whether an LLM can substitute for a human coder, framed as an equivalence test: whether the paired difference in agreement falls within a ±0.10 band of the human–human baseline, judged by where its confidence interval lies rather than by a test against zero. The difference falls within 0.05 of the baseline
on every dimension, and its confidence interval lies inside the band on all three.
The results suggest that on justification and recognition the ensemble can serve as reliable independent measurement, and that on Counterarguments it can serve as a second coder in a doubly-coded design — where the binding constraint appears to
be the codebook anchor language rather than the model. ...
Evaluating the quality of public deliberation is a prerequisite for governing it on evidence rather than impression, yet the bottleneck is measurement. The Discourse Quality Index (DQI), the standard instrument in the field, requires trained human coders — slow, costly, and applied inconsistently across research teams — which means most empirical deliberation studies contain only a few thousand speeches. Scaling that measurement is the problem this paper addresses. Large language models (LLMs) can apply a structured rubric rapidly and uniformly, so the central question is whether an LLM coder is reliable enough on theory-grounded deliberative constructs to stand in for a trained human and raise the data-size ceiling. We make
two contributions. First, we construct a theory-justified three-dimension DQI sub- codebook — Level of Justification, Respect for Groups, and Counterarguments — defending each inclusion and exclusion from deliberative-democracy theory. Second,
we benchmark a four-model LLM-as-judge ensemble against two trained coders on
200 UK House of Commons public-safety debate acts. We report Gwet’s linear- weighted AC1: trained coders reach AC1 = 0.78, 0.94, and 0.48 on the three dimensions, and the ensemble reaches 0.74, 0.89, and 0.48 against the lead coder.
We also tested whether an LLM can substitute for a human coder, framed as an equivalence test: whether the paired difference in agreement falls within a ±0.10 band of the human–human baseline, judged by where its confidence interval lies rather than by a test against zero. The difference falls within 0.05 of the baseline
on every dimension, and its confidence interval lies inside the band on all three.
The results suggest that on justification and recognition the ensemble can serve as reliable independent measurement, and that on Counterarguments it can serve as a second coder in a doubly-coded design — where the binding constraint appears to
be the codebook anchor language rather than the model.
two contributions. First, we construct a theory-justified three-dimension DQI sub- codebook — Level of Justification, Respect for Groups, and Counterarguments — defending each inclusion and exclusion from deliberative-democracy theory. Second,
we benchmark a four-model LLM-as-judge ensemble against two trained coders on
200 UK House of Commons public-safety debate acts. We report Gwet’s linear- weighted AC1: trained coders reach AC1 = 0.78, 0.94, and 0.48 on the three dimensions, and the ensemble reaches 0.74, 0.89, and 0.48 against the lead coder.
We also tested whether an LLM can substitute for a human coder, framed as an equivalence test: whether the paired difference in agreement falls within a ±0.10 band of the human–human baseline, judged by where its confidence interval lies rather than by a test against zero. The difference falls within 0.05 of the baseline
on every dimension, and its confidence interval lies inside the band on all three.
The results suggest that on justification and recognition the ensemble can serve as reliable independent measurement, and that on Counterarguments it can serve as a second coder in a doubly-coded design — where the binding constraint appears to
be the codebook anchor language rather than the model.
Children around the world contact children’s helplines when facing emotional, social, or psychological difficulties. These helplines provide confidential support via phone or textbased conversations, where children can share concerns ranging from everyday worries to serious safety issues. Helplines rely on skilled volunteer counsellors who can empathise, structure conversations, and help children find solutions. These helplines train a large number of volunteer counsellors annually to keep up with the volume of conversations they receive. For example, De Kindertelefoon in the Netherlands handled on average around 900 conversations per day and trained 300 new volunteers in 2024. Traditional training methods, such as role-playing, are valuable but resource-intensive, time-consuming, and dependent on the availability of trainers. To address these challenges, interactive simulation-based agents offer a promising extension to existing training practices by enabling scalable, safe, and consistent training. Such agents can simulate a virtual child with whom trainees can practise counselling skills without involving real children. However, current solutions mainly focus on observable interaction behaviour, while paying less attention to clarifying the motivations underlying the child’s actions.....
...
Children around the world contact children’s helplines when facing emotional, social, or psychological difficulties. These helplines provide confidential support via phone or textbased conversations, where children can share concerns ranging from everyday worries to serious safety issues. Helplines rely on skilled volunteer counsellors who can empathise, structure conversations, and help children find solutions. These helplines train a large number of volunteer counsellors annually to keep up with the volume of conversations they receive. For example, De Kindertelefoon in the Netherlands handled on average around 900 conversations per day and trained 300 new volunteers in 2024. Traditional training methods, such as role-playing, are valuable but resource-intensive, time-consuming, and dependent on the availability of trainers. To address these challenges, interactive simulation-based agents offer a promising extension to existing training practices by enabling scalable, safe, and consistent training. Such agents can simulate a virtual child with whom trainees can practise counselling skills without involving real children. However, current solutions mainly focus on observable interaction behaviour, while paying less attention to clarifying the motivations underlying the child’s actions.....
Understanding mental resilience-how individuals recover from stressors-is a critical focus area for mental health practitioners. To support this, the Experience Sampling Method (ESM) can be utilized, which offers rich, real-time data for tracking emotional dynamics. However, visualizing these data in an informative and actionable way remains a challenge. This study presents a set of visualizations designed to depict emotional recovery trajectories by focusing on mood, emotions, types of stressors and the influence of coping activities. To assess whether the developed visuals are interpretable and useful to mental health professionals, a survey-based evaluation was conducted. Afterwards, thematic analysis was applied in order to analyze and present the results. What was highlighted in the results is that easy-to-interpret visuals, such as line graphs and heatmaps, were more likely to be considered intuitive and clinically relevant. Moreover, visualizations that incorporated contextual details, like specific emotional responses or coping strategies, were regarded as more insightful and valuable for guiding therapy.
...
Understanding mental resilience-how individuals recover from stressors-is a critical focus area for mental health practitioners. To support this, the Experience Sampling Method (ESM) can be utilized, which offers rich, real-time data for tracking emotional dynamics. However, visualizing these data in an informative and actionable way remains a challenge. This study presents a set of visualizations designed to depict emotional recovery trajectories by focusing on mood, emotions, types of stressors and the influence of coping activities. To assess whether the developed visuals are interpretable and useful to mental health professionals, a survey-based evaluation was conducted. Afterwards, thematic analysis was applied in order to analyze and present the results. What was highlighted in the results is that easy-to-interpret visuals, such as line graphs and heatmaps, were more likely to be considered intuitive and clinically relevant. Moreover, visualizations that incorporated contextual details, like specific emotional responses or coping strategies, were regarded as more insightful and valuable for guiding therapy.
Human Insight vs. Artificial Intelligence: A Thematic Analysis
Comparing Manual and LLM Approaches to Understanding How Smokers Experience Preparatory Activities in a Digital Cessation Intervention
Smoking remains a leading cause of preventable death, making effective cessation support a global health priority. While conversational agents (chatbots) offer a scalable solution, their success depends on understanding the user’s experience. This study addresses two interconnected challenges: first, understanding the subjective experience of smokers with preparatory activities proposed by a chatbot, and second, evaluating the efficacy of Large Language Models (LLMs) in analyzing this qualitative feedback. This research employs a comparative design. A manual thematic analysis of smokers’ written reflections first established a baseline coding scheme. This scheme was then compared against the outputs of three LLMs, which were tasked with both generating themes independently and applying the predefined manual scheme. The accuracy of the LLMs’ application was measured against the human baseline using Cohen’s Kappa. The manual analysis revealed that smokers’ experiences were predominantly positive, showing strong motivation and a sense that the activities helped reinforce their quitting goals. This was concurrently challenged by expressions of skepticism about the activities’ effectiveness and mentions of personal barriers to quitting. The comparative analysis demonstrated that while LLMs could identify these broad positive and negative topics, they failed to capture more subtle, attitude-based concepts, such as a user’s willingness to engage with an activity despite their personal doubts. Furthermore, the models’ accuracy in applying a predefined coding scheme was substantially lower than the human baseline. This work makes two primary contributions. For digital health, the findings show that cessation aids must be designed to personalize activities to address specific user barriers and skepticism. Methodologically, the study provides a clear verdict on the current role of LLMs in this context: while LLMs show potential as an exploratory aid in theme generation, they are not yet a viable tool for applying a predefined coding scheme, making human analytical oversight essential for ensuring the depth and validity of qualitative research.
...
Smoking remains a leading cause of preventable death, making effective cessation support a global health priority. While conversational agents (chatbots) offer a scalable solution, their success depends on understanding the user’s experience. This study addresses two interconnected challenges: first, understanding the subjective experience of smokers with preparatory activities proposed by a chatbot, and second, evaluating the efficacy of Large Language Models (LLMs) in analyzing this qualitative feedback. This research employs a comparative design. A manual thematic analysis of smokers’ written reflections first established a baseline coding scheme. This scheme was then compared against the outputs of three LLMs, which were tasked with both generating themes independently and applying the predefined manual scheme. The accuracy of the LLMs’ application was measured against the human baseline using Cohen’s Kappa. The manual analysis revealed that smokers’ experiences were predominantly positive, showing strong motivation and a sense that the activities helped reinforce their quitting goals. This was concurrently challenged by expressions of skepticism about the activities’ effectiveness and mentions of personal barriers to quitting. The comparative analysis demonstrated that while LLMs could identify these broad positive and negative topics, they failed to capture more subtle, attitude-based concepts, such as a user’s willingness to engage with an activity despite their personal doubts. Furthermore, the models’ accuracy in applying a predefined coding scheme was substantially lower than the human baseline. This work makes two primary contributions. For digital health, the findings show that cessation aids must be designed to personalize activities to address specific user barriers and skepticism. Methodologically, the study provides a clear verdict on the current role of LLMs in this context: while LLMs show potential as an exploratory aid in theme generation, they are not yet a viable tool for applying a predefined coding scheme, making human analytical oversight essential for ensuring the depth and validity of qualitative research.
Interaction with Artificial Social Agents
A thematic analysis of people’s experiences
The use of Artificial Social Agents (ASAs) is rapidly expanding across society. As these agents become more integrated into our interactions, understanding the user experience of them becomes increasingly necessary to ensure their design aligns with user needs, promotes trust, and supports meaningful engagement. This study aims to investigate how users experience interactions with ASAs, focusing on using thematic analysis to identify recurring themes in user-reported experiences with ASAs. In addition, it also explores the reliability of locally hosted Large Language Models (LLMs) in identifying those experiences. We conducted a manual -peer validated- thematic analysis, resulting in a total of 31 themes. Afterwards, we conducted two experiments with LLMs, namely giving hem an unguided prompt (i.e. the LLM discovers and groups themes independently) and a guided prompt (i.e. the LLM matches predefined themes to responses) and measured their agreements with the manual analysis both intuitively and analytically. From our findings, it became clear that users experience ASAs through a balance of practical utility and emotional engagement. Themes covering the agent's helpfulness, sociability, enjoyability and perceived intelligence played a central role in shaping user experience. Most users responded positively to ASAs that felt intuitive, responsive, and human-like, though perceptions of human-likeness varied, sometimes enhancing the experience and other times creating discomfort. Our evaluation of LLMs showed that while they are capable of uncovering broad thematic patterns through unguided analysis, they fall short when tasked with consistently identifying and labeling predefined themes at the individual response level. This suggests that current LLMs, while useful as supplementary tools, are not yet reliable replacements for human-led thematic analysis in capturing the full nuance of user experiences at a detailed level. The conclusions reinforce the continued value and need of human-led thematic analysis, particularly when aiming to capture subtle, context-dependent insights that automated models may overlook.
...
The use of Artificial Social Agents (ASAs) is rapidly expanding across society. As these agents become more integrated into our interactions, understanding the user experience of them becomes increasingly necessary to ensure their design aligns with user needs, promotes trust, and supports meaningful engagement. This study aims to investigate how users experience interactions with ASAs, focusing on using thematic analysis to identify recurring themes in user-reported experiences with ASAs. In addition, it also explores the reliability of locally hosted Large Language Models (LLMs) in identifying those experiences. We conducted a manual -peer validated- thematic analysis, resulting in a total of 31 themes. Afterwards, we conducted two experiments with LLMs, namely giving hem an unguided prompt (i.e. the LLM discovers and groups themes independently) and a guided prompt (i.e. the LLM matches predefined themes to responses) and measured their agreements with the manual analysis both intuitively and analytically. From our findings, it became clear that users experience ASAs through a balance of practical utility and emotional engagement. Themes covering the agent's helpfulness, sociability, enjoyability and perceived intelligence played a central role in shaping user experience. Most users responded positively to ASAs that felt intuitive, responsive, and human-like, though perceptions of human-likeness varied, sometimes enhancing the experience and other times creating discomfort. Our evaluation of LLMs showed that while they are capable of uncovering broad thematic patterns through unguided analysis, they fall short when tasked with consistently identifying and labeling predefined themes at the individual response level. This suggests that current LLMs, while useful as supplementary tools, are not yet reliable replacements for human-led thematic analysis in capturing the full nuance of user experiences at a detailed level. The conclusions reinforce the continued value and need of human-led thematic analysis, particularly when aiming to capture subtle, context-dependent insights that automated models may overlook.
Smoking cessation remains a persistent public health challenge, with digital interventions increasingly adopted to support behavior change. This study explores how smokers plan and engage with preparatory activities suggested by conversational agents in online cessation programs. It addresses three main questions: the factors that influence engagement with these activities, the ability of large language models to identify smokers’ articulated plans, and the role of conditional "if-then" formulations in expressing coping strategies. A thematic analysis was conducted on qualitative user responses using both manual coding and automated labeling by local large language models. The findings show that smokers create context-sensitive plans shaped by emotional states, routines, and perceived usefulness of the suggestions, often expressed through conditional intentions. While manual analysis produced consistent and detailed themes, the large language model showed low agreement, highlighting limitations in current AI driven qualitative analysis. These results inform the design of more adaptive digital cessation tools and contribute to the understanding of AI’s role in supporting thematic research.
...
Smoking cessation remains a persistent public health challenge, with digital interventions increasingly adopted to support behavior change. This study explores how smokers plan and engage with preparatory activities suggested by conversational agents in online cessation programs. It addresses three main questions: the factors that influence engagement with these activities, the ability of large language models to identify smokers’ articulated plans, and the role of conditional "if-then" formulations in expressing coping strategies. A thematic analysis was conducted on qualitative user responses using both manual coding and automated labeling by local large language models. The findings show that smokers create context-sensitive plans shaped by emotional states, routines, and perceived usefulness of the suggestions, often expressed through conditional intentions. While manual analysis produced consistent and detailed themes, the large language model showed low agreement, highlighting limitations in current AI driven qualitative analysis. These results inform the design of more adaptive digital cessation tools and contribute to the understanding of AI’s role in supporting thematic research.
Digital smoking cessation tools increasingly use chatbots to recommend preparatory activities intended to support behavior change. However, little is known about how users actually experience these activities and whether large language models (LLMs) can support the qualitative analysis of such experiences. This study explores how smokers and vapers described their experiences with chatbot-suggested activities in an online intervention and investigates whether an LLM can assist with analyzing their responses.
We conducted a reflexive thematic analysis on 650 filtered user responses and examined whether a local version of LLaMA 3.3 (8B) could support the generation of common patterns and themes from qualitative data, and the categorization of such data into predefined themes. Our findings show that participants had mixed experiences. Some felt encouraged and supported by the activities, while others found them confusing, irrelevant to smoking cessation, or difficult to complete due to contextual or environmental barriers.
While the model was able to generate reasonable themes that overlapped with human interpretations, it was unable to reliably label data points using a predefined coding schema, achieving a Cohen’s Kappa of just 0.003. These findings suggest that LLMs may be useful in the early, exploratory stages of qualitative analysis, but currently lack the accuracy needed for theme application in complex, nuanced data. ...
We conducted a reflexive thematic analysis on 650 filtered user responses and examined whether a local version of LLaMA 3.3 (8B) could support the generation of common patterns and themes from qualitative data, and the categorization of such data into predefined themes. Our findings show that participants had mixed experiences. Some felt encouraged and supported by the activities, while others found them confusing, irrelevant to smoking cessation, or difficult to complete due to contextual or environmental barriers.
While the model was able to generate reasonable themes that overlapped with human interpretations, it was unable to reliably label data points using a predefined coding schema, achieving a Cohen’s Kappa of just 0.003. These findings suggest that LLMs may be useful in the early, exploratory stages of qualitative analysis, but currently lack the accuracy needed for theme application in complex, nuanced data. ...
Digital smoking cessation tools increasingly use chatbots to recommend preparatory activities intended to support behavior change. However, little is known about how users actually experience these activities and whether large language models (LLMs) can support the qualitative analysis of such experiences. This study explores how smokers and vapers described their experiences with chatbot-suggested activities in an online intervention and investigates whether an LLM can assist with analyzing their responses.
We conducted a reflexive thematic analysis on 650 filtered user responses and examined whether a local version of LLaMA 3.3 (8B) could support the generation of common patterns and themes from qualitative data, and the categorization of such data into predefined themes. Our findings show that participants had mixed experiences. Some felt encouraged and supported by the activities, while others found them confusing, irrelevant to smoking cessation, or difficult to complete due to contextual or environmental barriers.
While the model was able to generate reasonable themes that overlapped with human interpretations, it was unable to reliably label data points using a predefined coding schema, achieving a Cohen’s Kappa of just 0.003. These findings suggest that LLMs may be useful in the early, exploratory stages of qualitative analysis, but currently lack the accuracy needed for theme application in complex, nuanced data.
We conducted a reflexive thematic analysis on 650 filtered user responses and examined whether a local version of LLaMA 3.3 (8B) could support the generation of common patterns and themes from qualitative data, and the categorization of such data into predefined themes. Our findings show that participants had mixed experiences. Some felt encouraged and supported by the activities, while others found them confusing, irrelevant to smoking cessation, or difficult to complete due to contextual or environmental barriers.
While the model was able to generate reasonable themes that overlapped with human interpretations, it was unable to reliably label data points using a predefined coding schema, achieving a Cohen’s Kappa of just 0.003. These findings suggest that LLMs may be useful in the early, exploratory stages of qualitative analysis, but currently lack the accuracy needed for theme application in complex, nuanced data.
Smoking cessation interventions sometimes involve the use of both technology and human support to increase effectiveness. Nevertheless, little is known about user preferences for allocating human support and whether large language models (LLMs) can support qualitative analysis to better understand these preferences. This study analyzes how smokers’ ethical perspectives shape their preferences for time allocation mechanisms in online smoking cessation programs, while also evaluating how can LLMs support this analysis.
We conducted a deductive thematic analysis of open-ended responses from users who completed a questionnaire after participating in a smoking cessation program with a virtual coach. Additionally, we employed the LLaMA large language model to identify patterns in the responses and to assign the ethical themes discovered during the analysis.
The findings indicate that some users valued fairness and preferred scheduled, randomized interventions or no feedback. Others emphasized autonomy, wanting users to request feedback themselves. Some suggested prioritizing motivated or advanced users. A common view was that interventions should focus on those in need, users at risk of disengagement or health problems, those making little progress, experiencing emotional difficulties, or lacking clarity. The large language model was successful in identifying themes but not in accurately allocating themes, reflected by a low Cohen’s Kappa of 0.05.
The results presenting user preferences can guide the design of interventions that need to be effective and ethically sound. The findings suggest that while large language models can identify themes, they are not yet suitable for allocating those themes. ...
We conducted a deductive thematic analysis of open-ended responses from users who completed a questionnaire after participating in a smoking cessation program with a virtual coach. Additionally, we employed the LLaMA large language model to identify patterns in the responses and to assign the ethical themes discovered during the analysis.
The findings indicate that some users valued fairness and preferred scheduled, randomized interventions or no feedback. Others emphasized autonomy, wanting users to request feedback themselves. Some suggested prioritizing motivated or advanced users. A common view was that interventions should focus on those in need, users at risk of disengagement or health problems, those making little progress, experiencing emotional difficulties, or lacking clarity. The large language model was successful in identifying themes but not in accurately allocating themes, reflected by a low Cohen’s Kappa of 0.05.
The results presenting user preferences can guide the design of interventions that need to be effective and ethically sound. The findings suggest that while large language models can identify themes, they are not yet suitable for allocating those themes. ...
Smoking cessation interventions sometimes involve the use of both technology and human support to increase effectiveness. Nevertheless, little is known about user preferences for allocating human support and whether large language models (LLMs) can support qualitative analysis to better understand these preferences. This study analyzes how smokers’ ethical perspectives shape their preferences for time allocation mechanisms in online smoking cessation programs, while also evaluating how can LLMs support this analysis.
We conducted a deductive thematic analysis of open-ended responses from users who completed a questionnaire after participating in a smoking cessation program with a virtual coach. Additionally, we employed the LLaMA large language model to identify patterns in the responses and to assign the ethical themes discovered during the analysis.
The findings indicate that some users valued fairness and preferred scheduled, randomized interventions or no feedback. Others emphasized autonomy, wanting users to request feedback themselves. Some suggested prioritizing motivated or advanced users. A common view was that interventions should focus on those in need, users at risk of disengagement or health problems, those making little progress, experiencing emotional difficulties, or lacking clarity. The large language model was successful in identifying themes but not in accurately allocating themes, reflected by a low Cohen’s Kappa of 0.05.
The results presenting user preferences can guide the design of interventions that need to be effective and ethically sound. The findings suggest that while large language models can identify themes, they are not yet suitable for allocating those themes.
We conducted a deductive thematic analysis of open-ended responses from users who completed a questionnaire after participating in a smoking cessation program with a virtual coach. Additionally, we employed the LLaMA large language model to identify patterns in the responses and to assign the ethical themes discovered during the analysis.
The findings indicate that some users valued fairness and preferred scheduled, randomized interventions or no feedback. Others emphasized autonomy, wanting users to request feedback themselves. Some suggested prioritizing motivated or advanced users. A common view was that interventions should focus on those in need, users at risk of disengagement or health problems, those making little progress, experiencing emotional difficulties, or lacking clarity. The large language model was successful in identifying themes but not in accurately allocating themes, reflected by a low Cohen’s Kappa of 0.05.
The results presenting user preferences can guide the design of interventions that need to be effective and ethically sound. The findings suggest that while large language models can identify themes, they are not yet suitable for allocating those themes.
Child helplines provide a platform for children facing serious challenges, allowing them to share their stories and receive emotional support and guidance through counselling sessions. To ensure that volunteers are well prepared for these interactions, effective training programmes are essential. Recently, a BDI-based conversational agent was developed to train counsellors through role-play simulations, with the agent taking on the role of the child. However, this simulation training does not provide guidance to the trainees that caused a decrease in the trainees’ self-efficacy. In this research, we aim to enhance the effectiveness of counselling training through role-play simulation by integrating a pedagogical agent. We integrated an adaptive pedagogical agent by applying the scaffolding technique, where we taught a set of skills divided into three modules. The pedagogical agent also provided feedback and hints as additional guidance methods. We evaluated this design through a mixed study involving 22 participants, comparing an intervention group trained with the pedagogical agent to a control group using a standard training approach. We measured the participants’ performance, self-efficacy, and perceived usefulness of the system. While the intervention group showed higher mean scores across all measures compared to the control group, the differences were not statistically significant, indicating a possible underpowered experiment. This study contributes to the integration of pedagogical agents in simulation training systems for child helplines by proposing a framework that combines scaffolding, adaptivity, and structured learning.
...
Child helplines provide a platform for children facing serious challenges, allowing them to share their stories and receive emotional support and guidance through counselling sessions. To ensure that volunteers are well prepared for these interactions, effective training programmes are essential. Recently, a BDI-based conversational agent was developed to train counsellors through role-play simulations, with the agent taking on the role of the child. However, this simulation training does not provide guidance to the trainees that caused a decrease in the trainees’ self-efficacy. In this research, we aim to enhance the effectiveness of counselling training through role-play simulation by integrating a pedagogical agent. We integrated an adaptive pedagogical agent by applying the scaffolding technique, where we taught a set of skills divided into three modules. The pedagogical agent also provided feedback and hints as additional guidance methods. We evaluated this design through a mixed study involving 22 participants, comparing an intervention group trained with the pedagogical agent to a control group using a standard training approach. We measured the participants’ performance, self-efficacy, and perceived usefulness of the system. While the intervention group showed higher mean scores across all measures compared to the control group, the differences were not statistically significant, indicating a possible underpowered experiment. This study contributes to the integration of pedagogical agents in simulation training systems for child helplines by proposing a framework that combines scaffolding, adaptivity, and structured learning.
Child helpline counsellors require various skills and strategies to achieve lasting change in children who require assistance. Typical training methods such as role-play are resource intensive, leading to the development of computer simulation-based training systems where learners counsel the computer which assumes the role of a child requiring assistance. Such systems are limited in their understanding and responses, causing them to appear unrealistic and repetitive. In this paper, we built upon one such rule-based agent through the integration of Large Language Models (LLMs) to vastly expand both the understanding and responses of the agent. We conducted a within-subject experiment with 37 participants who we recruited online through Prolific, where they interacted with both systems, assuming the role of a counsellor. Our results indicate that participants find the integrated system to be human-like in its behaviour, have a more positive attitude towards it, and have a better impression of their overall experience with it. Our thematic analysis revealed that the integrated system felt more adaptive, and engaging, and allowed them to focus more on applying the conversational strategy, while the rule-based system felt scripted and boring. Our work provides an integrated system for effectively training child helpline counsellors and a method by which LLMs and rule-based systems can be integrated in general.
...
Child helpline counsellors require various skills and strategies to achieve lasting change in children who require assistance. Typical training methods such as role-play are resource intensive, leading to the development of computer simulation-based training systems where learners counsel the computer which assumes the role of a child requiring assistance. Such systems are limited in their understanding and responses, causing them to appear unrealistic and repetitive. In this paper, we built upon one such rule-based agent through the integration of Large Language Models (LLMs) to vastly expand both the understanding and responses of the agent. We conducted a within-subject experiment with 37 participants who we recruited online through Prolific, where they interacted with both systems, assuming the role of a counsellor. Our results indicate that participants find the integrated system to be human-like in its behaviour, have a more positive attitude towards it, and have a better impression of their overall experience with it. Our thematic analysis revealed that the integrated system felt more adaptive, and engaging, and allowed them to focus more on applying the conversational strategy, while the rule-based system felt scripted and boring. Our work provides an integrated system for effectively training child helpline counsellors and a method by which LLMs and rule-based systems can be integrated in general.
This thesis investigates how Reinforcement Learning (RL) can increase support effectiveness in virtual coach-based smoking cessation interventions. Such interventions have shown promise in helping people change behaviors such as smoking. However, personalizing the support they provide by accounting for people's current and future states might further increase their effectiveness. States thereby refer to people's relatively stable conditions at certain moments in time, capturing aspects such as motivation, knowledge, or the presence of personal reminders. After deriving general user needs for the support provided by a virtual coach-based smoking cessation intervention from a study with 671 daily smokers, we thus used RL to adapt the support to people's current and future states. Specifically, using data collected from three crowdsourcing studies with each more than 500 participants, we assessed the effectiveness of different RL model components in adapting 1) how people are persuaded, 2) what they are asked to do, and 3) who they are supported by. Our findings suggest that considering current and future states increases the effort smokers spend on smoking cessation activities and helps them build quitting-related competencies over time. Given that model components were derived from behavior change theories, this shows the potential of using psychology-informed RL to create smoking cessation support that is effective in the long run.
...
This thesis investigates how Reinforcement Learning (RL) can increase support effectiveness in virtual coach-based smoking cessation interventions. Such interventions have shown promise in helping people change behaviors such as smoking. However, personalizing the support they provide by accounting for people's current and future states might further increase their effectiveness. States thereby refer to people's relatively stable conditions at certain moments in time, capturing aspects such as motivation, knowledge, or the presence of personal reminders. After deriving general user needs for the support provided by a virtual coach-based smoking cessation intervention from a study with 671 daily smokers, we thus used RL to adapt the support to people's current and future states. Specifically, using data collected from three crowdsourcing studies with each more than 500 participants, we assessed the effectiveness of different RL model components in adapting 1) how people are persuaded, 2) what they are asked to do, and 3) who they are supported by. Our findings suggest that considering current and future states increases the effort smokers spend on smoking cessation activities and helps them build quitting-related competencies over time. Given that model components were derived from behavior change theories, this shows the potential of using psychology-informed RL to create smoking cessation support that is effective in the long run.
Analyzing users’ introductions to human coaches
Insights from eHealth applications introductions
This study analyzed the self-introductions of participants in a smoking and vaping cessation program to understand their motivations, barriers, and support needs. Through thematic analysis of 787 participant introductions, six main themes were identified: Motivations for Quitting, Previous Attempts to Quit, Barriers to Quitting, Desired Support, Usage Patterns, and Identity. Health concerns emerged as the most common motivation for quitting, while psychological and social challenges were the primary barriers. The study found no significant correlation between the length of introductions and participant engagement or satisfaction, suggesting that while introductions provide valuable qualitative insights, they do not relate to other factors. The findings describe the contents of introductions to human coaches, although the direct impact on outcomes requires further exploration. Future research should employ a controlled design to evaluate the effectiveness of detailed participant introductions in enhancing eHealth support and consider the complexity of dual usage in smoking and vaping cessation efforts.
...
This study analyzed the self-introductions of participants in a smoking and vaping cessation program to understand their motivations, barriers, and support needs. Through thematic analysis of 787 participant introductions, six main themes were identified: Motivations for Quitting, Previous Attempts to Quit, Barriers to Quitting, Desired Support, Usage Patterns, and Identity. Health concerns emerged as the most common motivation for quitting, while psychological and social challenges were the primary barriers. The study found no significant correlation between the length of introductions and participant engagement or satisfaction, suggesting that while introductions provide valuable qualitative insights, they do not relate to other factors. The findings describe the contents of introductions to human coaches, although the direct impact on outcomes requires further exploration. Future research should employ a controlled design to evaluate the effectiveness of detailed participant introductions in enhancing eHealth support and consider the complexity of dual usage in smoking and vaping cessation efforts.
Smoking and vaping cessation remains a significant public health challenge despite the availability of numerous aids and eHealth applications. This study explores the reasons behind users' preference for human feedback when preparing to quit smoking or vaping, aiming to address a gap in existing literature on the integration of human elements in eHealth platforms. The research involved 479 participants interacting with a virtual coach, with some receiving human feedback. We conducted a thematic analysis of two open-text questions with 265 responses each from post-questionnaires, and key themes such as emotional connection, personalized advice, effectiveness, motivation, and accountability were identified. Through using quantitative data and previously published research, these findings were further explained. The results from the quantitative analysis show that incorporating human elements in eHealth applications can enhance smoking cessation support. This research provides insights into the main reasons how the human support in eHealth applications should be designed. Key recommendations include designing human feedback to offer empathy and validation, tailoring feedback to individual needs, incorporating interactive elements to maintain engagement, providing constant encouragement, and establishing accountability mechanisms.
...
Smoking and vaping cessation remains a significant public health challenge despite the availability of numerous aids and eHealth applications. This study explores the reasons behind users' preference for human feedback when preparing to quit smoking or vaping, aiming to address a gap in existing literature on the integration of human elements in eHealth platforms. The research involved 479 participants interacting with a virtual coach, with some receiving human feedback. We conducted a thematic analysis of two open-text questions with 265 responses each from post-questionnaires, and key themes such as emotional connection, personalized advice, effectiveness, motivation, and accountability were identified. Through using quantitative data and previously published research, these findings were further explained. The results from the quantitative analysis show that incorporating human elements in eHealth applications can enhance smoking cessation support. This research provides insights into the main reasons how the human support in eHealth applications should be designed. Key recommendations include designing human feedback to offer empathy and validation, tailoring feedback to individual needs, incorporating interactive elements to maintain engagement, providing constant encouragement, and establishing accountability mechanisms.
Background. Quitting smoking is a challenge nowadays. Virtual coaches offer autonomous, personalized guidance for smoking cessation. However, such systems cannot replace human coaches completely. In situations, when human coaches cannot provide help to everyone - a virtual coach could follow a set of ethical principles to decide on who should get the feedback from a human.
Objective. Our study aims to identify users’ preferences on ethical principles that a virtual coach should follow to decide when to allocate human feedback to individuals preparing to quit smoking.
Methods. Our research was based on pre-gathered data, that included participants’ responses to open and closed questions regarding feedback allocation principles. Thematic analysis was conducted on these responses. Triangulation was performed using a qualitative literature review and quantitative data analysis.
Results. Four main themes were identified: (1) Struggling the Most (63.75%), (2) Increasing Chances of Success the Most (13.75%), (3) Equal Treatment (11.25%), and (4) Appreciating the Most (11.25%). Participants prioritized support for those experiencing the greatest difficulty in smoking cessation. The triangulation supported the validity of these themes.
Conclusions. Our study highlights the importance of integrating user-preferred ethical principles in virtual coaching systems for smoking cessation. Prioritization of users who struggle the most can increase the effectiveness and fairness of such systems, potentially increasing success rates. Future research should explore additional ethical principles, combining several principles into systems, and real-world application of these findings to further refine virtual coaching in healthcare. ...
Objective. Our study aims to identify users’ preferences on ethical principles that a virtual coach should follow to decide when to allocate human feedback to individuals preparing to quit smoking.
Methods. Our research was based on pre-gathered data, that included participants’ responses to open and closed questions regarding feedback allocation principles. Thematic analysis was conducted on these responses. Triangulation was performed using a qualitative literature review and quantitative data analysis.
Results. Four main themes were identified: (1) Struggling the Most (63.75%), (2) Increasing Chances of Success the Most (13.75%), (3) Equal Treatment (11.25%), and (4) Appreciating the Most (11.25%). Participants prioritized support for those experiencing the greatest difficulty in smoking cessation. The triangulation supported the validity of these themes.
Conclusions. Our study highlights the importance of integrating user-preferred ethical principles in virtual coaching systems for smoking cessation. Prioritization of users who struggle the most can increase the effectiveness and fairness of such systems, potentially increasing success rates. Future research should explore additional ethical principles, combining several principles into systems, and real-world application of these findings to further refine virtual coaching in healthcare. ...
Background. Quitting smoking is a challenge nowadays. Virtual coaches offer autonomous, personalized guidance for smoking cessation. However, such systems cannot replace human coaches completely. In situations, when human coaches cannot provide help to everyone - a virtual coach could follow a set of ethical principles to decide on who should get the feedback from a human.
Objective. Our study aims to identify users’ preferences on ethical principles that a virtual coach should follow to decide when to allocate human feedback to individuals preparing to quit smoking.
Methods. Our research was based on pre-gathered data, that included participants’ responses to open and closed questions regarding feedback allocation principles. Thematic analysis was conducted on these responses. Triangulation was performed using a qualitative literature review and quantitative data analysis.
Results. Four main themes were identified: (1) Struggling the Most (63.75%), (2) Increasing Chances of Success the Most (13.75%), (3) Equal Treatment (11.25%), and (4) Appreciating the Most (11.25%). Participants prioritized support for those experiencing the greatest difficulty in smoking cessation. The triangulation supported the validity of these themes.
Conclusions. Our study highlights the importance of integrating user-preferred ethical principles in virtual coaching systems for smoking cessation. Prioritization of users who struggle the most can increase the effectiveness and fairness of such systems, potentially increasing success rates. Future research should explore additional ethical principles, combining several principles into systems, and real-world application of these findings to further refine virtual coaching in healthcare.
Objective. Our study aims to identify users’ preferences on ethical principles that a virtual coach should follow to decide when to allocate human feedback to individuals preparing to quit smoking.
Methods. Our research was based on pre-gathered data, that included participants’ responses to open and closed questions regarding feedback allocation principles. Thematic analysis was conducted on these responses. Triangulation was performed using a qualitative literature review and quantitative data analysis.
Results. Four main themes were identified: (1) Struggling the Most (63.75%), (2) Increasing Chances of Success the Most (13.75%), (3) Equal Treatment (11.25%), and (4) Appreciating the Most (11.25%). Participants prioritized support for those experiencing the greatest difficulty in smoking cessation. The triangulation supported the validity of these themes.
Conclusions. Our study highlights the importance of integrating user-preferred ethical principles in virtual coaching systems for smoking cessation. Prioritization of users who struggle the most can increase the effectiveness and fairness of such systems, potentially increasing success rates. Future research should explore additional ethical principles, combining several principles into systems, and real-world application of these findings to further refine virtual coaching in healthcare.
Smoking remains one of the largest health concerns worldwide, which is why eHealth applications with virtual coaches have been developed to assist smokers with quitting. Providing additional feedback from human coaches during such smoking cessation programs can further improve the effectiveness of the intervention. However, due to budgetary constraints and the limited availability of human coaches, it is important to make informed decisions about when someone gets human support to optimize the effectiveness. This research investigates the use of reinforcement learning (RL) to determine when to provide human feedback in quitting smoking with a virtual coach. Using data from a longitudinal study, we implemented an RL model that decides when to involve a human coach based on users' appreciation for human support and their self-efficacy, optimizing the effort that people spend on preparatory activities and their likelihood of returning to the program. Results show that the model is effective in allocating human support, increasing users' effort and return likelihood while considering the cost of human coaches. These findings support using RL to help with determining when to provide human support in smoking cessation programs.
...
Smoking remains one of the largest health concerns worldwide, which is why eHealth applications with virtual coaches have been developed to assist smokers with quitting. Providing additional feedback from human coaches during such smoking cessation programs can further improve the effectiveness of the intervention. However, due to budgetary constraints and the limited availability of human coaches, it is important to make informed decisions about when someone gets human support to optimize the effectiveness. This research investigates the use of reinforcement learning (RL) to determine when to provide human feedback in quitting smoking with a virtual coach. Using data from a longitudinal study, we implemented an RL model that decides when to involve a human coach based on users' appreciation for human support and their self-efficacy, optimizing the effort that people spend on preparatory activities and their likelihood of returning to the program. Results show that the model is effective in allocating human support, increasing users' effort and return likelihood while considering the cost of human coaches. These findings support using RL to help with determining when to provide human support in smoking cessation programs.
Examining the Efficacy of Persuasive eHealth Applications in Facilitating Smoking Cessation
An Analysis of Competency Based Activities
To assist smokers in potentially quitting their habit, this paper investigates digital eHealth applications. Based on a dataset provided by research into an eHealth application, it aims to determine if persuasive activities can convince users of the usefulness of competencies determined to assist in quitting smoking. A thematic analysis was used on the participant's qualitative responses to the activities. Using this and the quantitative data derived, insights into the efficacy of persuasion were determined.
It was determined that engagement with the optional qualitative aspect of the data produced similar utility perspectives on the competencies to those who did not comment. It was noted that the general perspective of the competencies rose after completing the activity, however not to a significant degree. Additionally, no notable correlations between age, gender or educational level and increased perception of the competency arose. Several interesting remarks from participants were analysed to offer considerations for any future research in this field. ...
It was determined that engagement with the optional qualitative aspect of the data produced similar utility perspectives on the competencies to those who did not comment. It was noted that the general perspective of the competencies rose after completing the activity, however not to a significant degree. Additionally, no notable correlations between age, gender or educational level and increased perception of the competency arose. Several interesting remarks from participants were analysed to offer considerations for any future research in this field. ...
To assist smokers in potentially quitting their habit, this paper investigates digital eHealth applications. Based on a dataset provided by research into an eHealth application, it aims to determine if persuasive activities can convince users of the usefulness of competencies determined to assist in quitting smoking. A thematic analysis was used on the participant's qualitative responses to the activities. Using this and the quantitative data derived, insights into the efficacy of persuasion were determined.
It was determined that engagement with the optional qualitative aspect of the data produced similar utility perspectives on the competencies to those who did not comment. It was noted that the general perspective of the competencies rose after completing the activity, however not to a significant degree. Additionally, no notable correlations between age, gender or educational level and increased perception of the competency arose. Several interesting remarks from participants were analysed to offer considerations for any future research in this field.
It was determined that engagement with the optional qualitative aspect of the data produced similar utility perspectives on the competencies to those who did not comment. It was noted that the general perspective of the competencies rose after completing the activity, however not to a significant degree. Additionally, no notable correlations between age, gender or educational level and increased perception of the competency arose. Several interesting remarks from participants were analysed to offer considerations for any future research in this field.
Smoking has been one of the great threats to health in recent years, being strongly correlated with multiple negative health consequences, including lung cancer. Recent research suggests that artificial intelligence chatbots can be effective in persuading healthy behavior change. However, these chatbots usually rely on persuasive techniques to achieve their goal. Such techniques depend on identifying and meeting users' needs to be effective. To help improve understanding of the domain of healthy behavior change, we proposed a study which analysed the needs of daily smokers as they emerged from their interactions with a chatbot specifically designed to help them quit. The study performed a thematic analysis on users' free-text feedback, from which a set of 8 themes that directly correspond to 8 different needs were observed. The user needs were correlated with their genders, ages and highest completed education levels. While most of the results indicate that there is no significant correlation between the needs and user characteristics, which suggests that user needs are evenly distributed, certain correlations were highlighted for further analysis.
...
Smoking has been one of the great threats to health in recent years, being strongly correlated with multiple negative health consequences, including lung cancer. Recent research suggests that artificial intelligence chatbots can be effective in persuading healthy behavior change. However, these chatbots usually rely on persuasive techniques to achieve their goal. Such techniques depend on identifying and meeting users' needs to be effective. To help improve understanding of the domain of healthy behavior change, we proposed a study which analysed the needs of daily smokers as they emerged from their interactions with a chatbot specifically designed to help them quit. The study performed a thematic analysis on users' free-text feedback, from which a set of 8 themes that directly correspond to 8 different needs were observed. The user needs were correlated with their genders, ages and highest completed education levels. While most of the results indicate that there is no significant correlation between the needs and user characteristics, which suggests that user needs are evenly distributed, certain correlations were highlighted for further analysis.