Comparing Local LLM-Based Extraction of Stakeholder Values for Value Model Visualization in Deliberations

Abstract

In stakeholder deliberation, it can be useful to give insights into stakeholders' values (such as privacy, safety, and fairness). Previously, conversational agents were built to give insight into such a value model. Yet, the question of whether visualizations can aid in the understanding of a value model still remains. As a first step, this paper proposes two visualizations of value models based on existing literature: radar charts and value cards. This paper argues why these visualizations can aid in the understanding of value models, but to create them, data needs to be extracted from transcripts. Therefore, we also compare three consumer-grade local LLMs (≤ 35B parameters) - Gemma4:e4b, Phi4-reasoning:14b, and Qwen3.6:35b - on their ability to extract data from deliberative transcripts necessary to generate these visualizations. Using local LLMs for this task can be beneficial, as using cloud-provided LLMs can lead to value profiles being built. The evaluated LLMs are found to have strong agreement (Cohen's κ ≥ 0.808) with human coders on extracting the values included in the transcripts; however, they have mixed agreement when ranking the values or assigning codes to their importance. When comparing textual justifications for value rankings and assigned importance codes, justifications between LLMs and humans may differ, but the textual justifications generally do not disagree on whether a value is important or not. When the local LLMs have to give a summary of the meaning of values, they are generally roughly similar to human summaries or those provided by the other LLMs, and in 38 to 46\% of cases, LLM summaries are nearly equivalent. These findings suggest that consumer-grade local LLMs are effective at identifying human values present in text, but struggle to code their importance, ordinal rank, and summarize their meaning. This makes them currently unsuitable to replace human coders without oversight.