How to design post-reflection dialogue from transcripts using the identified values, value tensions, and consensus points?

Abstract

Large Language Models (LLMs) excel at natural language tasks, yet most contemporary systems and tool prioritize providing an answer over fostering reflection and deliberation. This research investigated whether LLM-based tools can generate post-reflection dialogue in multi-stakeholder decision making scenarios by using identified value tensions and points of contention found in transcripts.

A Deliberative AI approach was developed using publicly available transcripts and several open-sources LLMs. The generated reflective dialogue was subsequently evaluated through Synthetic Personae evaluators according to the five metrics established: safety, privacy, autonomy, societal well-being, and points of contention. Two different prompting strategies: single-turn and multi-turn were deployed to see if there were meaningful differences between the two.

The results indicated that the methodology can produce reflective dialogues that are perceived positively, exceeding the predefined success threshold. Furthermore, the iterative multi-turn interactions were found to improve perceived satisfaction compared to the single-turn approach on average.

Although limited to English language deliberations, the findings demonstrate the feasibility of using Deliberative AI to support reflection and, rather than proposing a universal solutions, this work provides a reproducible proof of concept that can be adapted based on future models, transcript contexts, and languages, motivating the development of more LLM-based deliberative systems.