Performance Enhancement Techniques for Information Extraction with Small Language Models
P.S. Deatc (TU Delft - Electrical Engineering, Mathematics and Computer Science)
A. Katsifodimos – Mentor (TU Delft - Electrical Engineering, Mathematics and Computer Science)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Information extraction from long and complex documents is an important task in domains such as healthcare and law. This study investigates how performance-enhancing techniques can improve information extraction in smaller language models.
We develop a language-model-based extraction pipeline that combines retrieval-augmented generation, structured extraction using schema-driven prompting, and post-extraction filtering. The system is evaluated on a legal dataset (Kleister NDA) and a real-world Dutch medical dataset consisting of general practitioner referral letters. We study the impact of retrieval depth, context size, chunk ordering, and a post-processing large language model (LLM) filter across multiple open and closed-source models.
The results show that retrieval is highly effective, with full coverage achieved using only a small number of retrieved chunks. However, increasing context size does not consistently improve performance, as it introduces a trade-off between precision and recall. Intermediate context sizes typically yield the best balance. Chunk ordering has a smaller but model-dependent effect, while different models show distinct behaviors ranging from conservative extraction to strong sensitivity to context composition.
Finally, a lightweight post-extraction filtering step significantly improves precision while preserving recall, substantially improving overall extraction performance. However, a performance gap remains compared to a state-of-the-art large model benchmark.
Overall, the study shows that smaller language models can achieve strong extraction performance when supported by carefully designed pipeline components, context design, and post-processing filters.