Navigating Nutritional Nuance: A PICO-Based Approach to Validating Nutritional Health Claims Using Retrieval-Augmented Generation

Evidence-based lifestyle practices are effective in preventing and treating cardiovascular disease. However, the growing body of scientific literature and the prevalence of conflicting studies makes it challenging for healthcare practitioners to stay informed. Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG), offer potential for automated fact-checking, where much work has been done in areas like politics, limited research has explored their application to nutritional health claims, which are more nuanced and demand rigorous evaluation of interventional studies for scientific validation. To fill this gap, this study investigates how effectively a RAG-based LLM can verify nuanced nutritional health claims. We develop a five-module framework, introducing an inclusion criteria-based approach and SMaPS Sequential Mapping of PICO-based Synthesis to enhance literature selection and evidence synthesis. Our findings indicate that while our Advanced RAG-LLM model shows potential in verifying nuanced health claims, it still faces significant limitations in accuracy.  Although the inclusion criteria-based filter and SMaPS approach help balance predictions, the model often defaults to neutral outcomes when evidence is unclear. The problem of overgeneralization, the inclusion of irrelevant studies, and the difficulty of synthesizing precise numerical data undermines the model's reliability to verify nuanced health claims.