Extracting Value, Value Tension, and Points of Agreement From Deliberation

Abstract

Deliberation is a process in which people come together to discuss and find solutions to complex topics. Human moderators are expensive to employ, time-consuming, and can introduce their own biases into the moderation and summary of deliberative conversations. Large language models (LLMs), on the other hand, avoid obscuring minority opinions and bring groups together rather than dividing them. In this paper, we examine whether LLMs can extract values, value tensions, and consensus points from deliberative transcripts. We experiment with 3 different prompting strategies, zero-shot, few-shot, and chain of thought prompting, and 3 different LLM models, Gemma 2, Qwen, and Mistral. We evaluate the results using ground truth annotations and an LLM as a judge study. We found that all LLMs were capable of extracting the basic constructs by providing valid outputs to the prompts given. Mistral slightly outperformed the other models according to LLM-as-a-judge, whereas Gemma 2 achieved the highest F1 score. Chain of thought prompting outperformed the others, according to the LLM-as-a-judge, with few-shot prompting achieving the highest overall F1 score. We found that the interaction between the model and prompting strategy is highly dependent on the evaluation criteria with the correlation in results between LLM-as-a-judge and metric evaluation tending to be slightly negative.