AK
A.S. Kuiper
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
2 records found
1
Applying Large Language Models (LLMs) to high-stakes classification tasks like systematic review screening is challenged by prompt sensitivity and a lack of transparency. We introduce IMAPR (Iterative Multi-signal Adaptive Prompt Refinement), a novel framework where a single LLM uses its own internal signals to iteratively refine its prompts, improving classification robustness and reliability. Unlike black-box optimizers that tune the prompts using only external scores, IMAPR is a white-box approach that diagnoses why a prediction failed using three internal signals: model confidence, a rationale, and a knowledge alignment score that checks whether the evidence cited in the rationale actually covers the user-defined inclusion criteria. We evaluate IMAPR on a real-world biomedical screening task, comparing it against strong baselines including GPO and StraGo. IMAPR outperforms the best baseline (GPO) by 8.8% in Macro-F1 while maintaining high, stable recall across runs. Across seven LLMs, IMAPR yields an average 9.2% improvement in Macro-F1 An ablation shows that knowledge-alignment acts as a recall safeguard: removing it leaves Macro-F1 similar but degrades recall, reducing reliability for screening. These results suggest that diagnostic, signal-driven prompt refinement is a practical alternative to black-box optimization for transparent, dependable LLM screening systems.
...
Applying Large Language Models (LLMs) to high-stakes classification tasks like systematic review screening is challenged by prompt sensitivity and a lack of transparency. We introduce IMAPR (Iterative Multi-signal Adaptive Prompt Refinement), a novel framework where a single LLM uses its own internal signals to iteratively refine its prompts, improving classification robustness and reliability. Unlike black-box optimizers that tune the prompts using only external scores, IMAPR is a white-box approach that diagnoses why a prediction failed using three internal signals: model confidence, a rationale, and a knowledge alignment score that checks whether the evidence cited in the rationale actually covers the user-defined inclusion criteria. We evaluate IMAPR on a real-world biomedical screening task, comparing it against strong baselines including GPO and StraGo. IMAPR outperforms the best baseline (GPO) by 8.8% in Macro-F1 while maintaining high, stable recall across runs. Across seven LLMs, IMAPR yields an average 9.2% improvement in Macro-F1 An ablation shows that knowledge-alignment acts as a recall safeguard: removing it leaves Macro-F1 similar but degrades recall, reducing reliability for screening. These results suggest that diagnostic, signal-driven prompt refinement is a practical alternative to black-box optimization for transparent, dependable LLM screening systems.
Commonsense knowledge is the key of human intelligence in generalizing their knowledge to deal with complex tasks. Over the past years, a lot of research has been done in both natural language processing (NLP) and computer vision (CV) on leveraging commonsense knowledge to improve AI models. However, no systematic comparisons of existing work have been made between the two domains. Therefore this survey aims to provide an overview of how commonsense knowledge is used within NLP and CV and how research varies between these two domains and what future challenges it may hold. An observation made from this survey is that leveraging commonsense is more difficult in CV than NLP, as commonsense is mostly incorporated textually and datasets need to be filtered to make them more relevant for visual commonsense. We hope to promote further research and create a better understanding of commonsense knowledge and its applications with this survey.
...
Commonsense knowledge is the key of human intelligence in generalizing their knowledge to deal with complex tasks. Over the past years, a lot of research has been done in both natural language processing (NLP) and computer vision (CV) on leveraging commonsense knowledge to improve AI models. However, no systematic comparisons of existing work have been made between the two domains. Therefore this survey aims to provide an overview of how commonsense knowledge is used within NLP and CV and how research varies between these two domains and what future challenges it may hold. An observation made from this survey is that leveraging commonsense is more difficult in CV than NLP, as commonsense is mostly incorporated textually and datasets need to be filtered to make them more relevant for visual commonsense. We hope to promote further research and create a better understanding of commonsense knowledge and its applications with this survey.