BH
B.S. Han
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
2 records found
1
Evidence-based lifestyle practices are effective in preventing and treating cardiovascular disease. However, the growing body of scientific literature and the prevalence of conflicting studies makes it challenging for healthcare practitioners to stay informed. Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG), offer potential for automated fact-checking, where much work has been done in areas like politics, limited research has explored their application to nutritional health claims, which are more nuanced and demand rigorous evaluation of interventional studies for scientific validation. To fill this gap, this study investigates how effectively a RAG-based LLM can verify nuanced nutritional health claims. We develop a five-module framework, introducing an inclusion criteria-based approach and SMaPS Sequential Mapping of PICO-based Synthesis to enhance literature selection and evidence synthesis. Our findings indicate that while our Advanced RAG-LLM model shows potential in verifying nuanced health claims, it still faces significant limitations in accuracy. Although the inclusion criteria-based filter and SMaPS approach help balance predictions, the model often defaults to neutral outcomes when evidence is unclear. The problem of overgeneralization, the inclusion of irrelevant studies, and the difficulty of synthesizing precise numerical data undermines the model's reliability to verify nuanced health claims.
Evidence-based lifestyle practices are effective in preventing and treating cardiovascular disease. However, the growing body of scientific literature and the prevalence of conflicting studies makes it challenging for healthcare practitioners to stay informed. Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG), offer potential for automated fact-checking, where much work has been done in areas like politics, limited research has explored their application to nutritional health claims, which are more nuanced and demand rigorous evaluation of interventional studies for scientific validation. To fill this gap, this study investigates how effectively a RAG-based LLM can verify nuanced nutritional health claims. We develop a five-module framework, introducing an inclusion criteria-based approach and SMaPS Sequential Mapping of PICO-based Synthesis to enhance literature selection and evidence synthesis. Our findings indicate that while our Advanced RAG-LLM model shows potential in verifying nuanced health claims, it still faces significant limitations in accuracy. Although the inclusion criteria-based filter and SMaPS approach help balance predictions, the model often defaults to neutral outcomes when evidence is unclear. The problem of overgeneralization, the inclusion of irrelevant studies, and the difficulty of synthesizing precise numerical data undermines the model's reliability to verify nuanced health claims.
Personality disorders affect 1 in 7 adults reducing their quality of life. Schema-focused therapy (SFT) has become very popular in Psychotherapy in the treatment of personality disorders (PD), unfortunately there is still in increasing societal need for such mental healthcare. Automation in the assessment of SFT allows for Ecological Momentary Assessments (EMA). Resulting in a dynamic assessment of schema-modes and making the treatment more socially available. Automation is realised by Allaart in the form of a conversational agent (CA), but needs a better schema classification algorithm to improve its efficacy. The goal of this study is to evaluate the k-Nearest Neighbour (kNN) algorithm along with Allaart’s dataset. The main question of the study is as follows: How well can a schema be automatically classified from a text using KNN? The method comprises of an experimentalpipeline consisting of 4 stages: Labeling of dataset; pre-processing of the data; schema classification; and evaluation. kNN performed satisfactory in multi-label binary classification with a mean accuracy of 71% and a mean weighted f1-score of 0.62. kNN did not outperform other classification algorithm and performed inadequate in ordinal classification. Results indicate a contrast between majority and minority classes and found a recall of 100% on one of the majority classes. Hence, the data set is concluded to be imbalanced. Due to limitations on the dataset and the CA no reliable conclusion can be made on the performance of kNN in automated schema classification. This study proposed future research to conduct a field experiment
such that the CA and its ability to perform EMA is evaluated and reliable data is produced. ...
such that the CA and its ability to perform EMA is evaluated and reliable data is produced. ...
Personality disorders affect 1 in 7 adults reducing their quality of life. Schema-focused therapy (SFT) has become very popular in Psychotherapy in the treatment of personality disorders (PD), unfortunately there is still in increasing societal need for such mental healthcare. Automation in the assessment of SFT allows for Ecological Momentary Assessments (EMA). Resulting in a dynamic assessment of schema-modes and making the treatment more socially available. Automation is realised by Allaart in the form of a conversational agent (CA), but needs a better schema classification algorithm to improve its efficacy. The goal of this study is to evaluate the k-Nearest Neighbour (kNN) algorithm along with Allaart’s dataset. The main question of the study is as follows: How well can a schema be automatically classified from a text using KNN? The method comprises of an experimentalpipeline consisting of 4 stages: Labeling of dataset; pre-processing of the data; schema classification; and evaluation. kNN performed satisfactory in multi-label binary classification with a mean accuracy of 71% and a mean weighted f1-score of 0.62. kNN did not outperform other classification algorithm and performed inadequate in ordinal classification. Results indicate a contrast between majority and minority classes and found a recall of 100% on one of the majority classes. Hence, the data set is concluded to be imbalanced. Due to limitations on the dataset and the CA no reliable conclusion can be made on the performance of kNN in automated schema classification. This study proposed future research to conduct a field experiment
such that the CA and its ability to perform EMA is evaluated and reliable data is produced.
such that the CA and its ability to perform EMA is evaluated and reliable data is produced.