Digital smoking cessation tools increasingly use chatbots to recommend preparatory activities intended to support behavior change. However, little is known about how users actually experience these activities and whether large language models (LLMs) can support the qualitative analysis of such experiences. This study explores how smokers and vapers described their experiences with chatbot-suggested activities in an online intervention and investigates whether an LLM can assist with analyzing their responses.

We conducted a reflexive thematic analysis on 650 filtered user responses and examined whether a local version of LLaMA 3.3 (8B) could support the generation of common patterns and themes from qualitative data, and the categorization of such data into predefined themes. Our findings show that participants had mixed experiences. Some felt encouraged and supported by the activities, while others found them confusing, irrelevant to smoking cessation, or difficult to complete due to contextual or environmental barriers.

While the model was able to generate reasonable themes that overlapped with human interpretations, it was unable to reliably label data points using a predefined coding schema, achieving a Cohen’s Kappa of just 0.003. These findings suggest that LLMs may be useful in the early, exploratory stages of qualitative analysis, but currently lack the accuracy needed for theme application in complex, nuanced data.