PD
P.S. Deatc
info
Please Note
<p>This page displays the records of the person named above and is not linked to a unique person identifier. This record may need to be merged to a profile.</p>
2 records found
1
Information extraction from long and complex documents is an important task in domains such as healthcare and law. This study investigates how performance-enhancing techniques can improve information extraction in smaller language models.
We develop a language-model-based extraction pipeline that combines retrieval-augmented generation, structured extraction using schema-driven prompting, and post-extraction filtering. The system is evaluated on a legal dataset (Kleister NDA) and a real-world Dutch medical dataset consisting of general practitioner referral letters. We study the impact of retrieval depth, context size, chunk ordering, and a post-processing large language model (LLM) filter across multiple open and closed-source models.
The results show that retrieval is highly effective, with full coverage achieved using only a small number of retrieved chunks. However, increasing context size does not consistently improve performance, as it introduces a trade-off between precision and recall. Intermediate context sizes typically yield the best balance. Chunk ordering has a smaller but model-dependent effect, while different models show distinct behaviors ranging from conservative extraction to strong sensitivity to context composition.
Finally, a lightweight post-extraction filtering step significantly improves precision while preserving recall, substantially improving overall extraction performance. However, a performance gap remains compared to a state-of-the-art large model benchmark.
Overall, the study shows that smaller language models can achieve strong extraction performance when supported by carefully designed pipeline components, context design, and post-processing filters. ...
We develop a language-model-based extraction pipeline that combines retrieval-augmented generation, structured extraction using schema-driven prompting, and post-extraction filtering. The system is evaluated on a legal dataset (Kleister NDA) and a real-world Dutch medical dataset consisting of general practitioner referral letters. We study the impact of retrieval depth, context size, chunk ordering, and a post-processing large language model (LLM) filter across multiple open and closed-source models.
The results show that retrieval is highly effective, with full coverage achieved using only a small number of retrieved chunks. However, increasing context size does not consistently improve performance, as it introduces a trade-off between precision and recall. Intermediate context sizes typically yield the best balance. Chunk ordering has a smaller but model-dependent effect, while different models show distinct behaviors ranging from conservative extraction to strong sensitivity to context composition.
Finally, a lightweight post-extraction filtering step significantly improves precision while preserving recall, substantially improving overall extraction performance. However, a performance gap remains compared to a state-of-the-art large model benchmark.
Overall, the study shows that smaller language models can achieve strong extraction performance when supported by carefully designed pipeline components, context design, and post-processing filters. ...
Information extraction from long and complex documents is an important task in domains such as healthcare and law. This study investigates how performance-enhancing techniques can improve information extraction in smaller language models.
We develop a language-model-based extraction pipeline that combines retrieval-augmented generation, structured extraction using schema-driven prompting, and post-extraction filtering. The system is evaluated on a legal dataset (Kleister NDA) and a real-world Dutch medical dataset consisting of general practitioner referral letters. We study the impact of retrieval depth, context size, chunk ordering, and a post-processing large language model (LLM) filter across multiple open and closed-source models.
The results show that retrieval is highly effective, with full coverage achieved using only a small number of retrieved chunks. However, increasing context size does not consistently improve performance, as it introduces a trade-off between precision and recall. Intermediate context sizes typically yield the best balance. Chunk ordering has a smaller but model-dependent effect, while different models show distinct behaviors ranging from conservative extraction to strong sensitivity to context composition.
Finally, a lightweight post-extraction filtering step significantly improves precision while preserving recall, substantially improving overall extraction performance. However, a performance gap remains compared to a state-of-the-art large model benchmark.
Overall, the study shows that smaller language models can achieve strong extraction performance when supported by carefully designed pipeline components, context design, and post-processing filters.
We develop a language-model-based extraction pipeline that combines retrieval-augmented generation, structured extraction using schema-driven prompting, and post-extraction filtering. The system is evaluated on a legal dataset (Kleister NDA) and a real-world Dutch medical dataset consisting of general practitioner referral letters. We study the impact of retrieval depth, context size, chunk ordering, and a post-processing large language model (LLM) filter across multiple open and closed-source models.
The results show that retrieval is highly effective, with full coverage achieved using only a small number of retrieved chunks. However, increasing context size does not consistently improve performance, as it introduces a trade-off between precision and recall. Intermediate context sizes typically yield the best balance. Chunk ordering has a smaller but model-dependent effect, while different models show distinct behaviors ranging from conservative extraction to strong sensitivity to context composition.
Finally, a lightweight post-extraction filtering step significantly improves precision while preserving recall, substantially improving overall extraction performance. However, a performance gap remains compared to a state-of-the-art large model benchmark.
Overall, the study shows that smaller language models can achieve strong extraction performance when supported by carefully designed pipeline components, context design, and post-processing filters.
Red Teaming Large Language Models for Code
Exploring Dangerous and Unfair Software Applications
The rapid advancement of large language models has enabled numerous innovative, but also harmful applications. It is therefore essential to create these models to behave safely and responsibly. One way to improve these models is by red teaming them. In this study, we aim to identify prompts that lead large language models to exhibit unfair or dangerous behavior in software and cybersecurity contexts. We do this by manually creating prompts and manually assessing the harmfulness of the response. Our contributions include a taxonomy of dangerous and unfair use cases of large language models for Code, a dataset of 200 prompts tested on eight models, an investigation into how expanding the prompt, and how adding a code skeleton for the model to complete changes the level of harmfulness. Among the eight models evaluated, only CodeGemma and GPT-3.5-0125 were well-aligned against our taxonomy categories. The unaligned Dolphin-Mixtral and self-aligned Starcoder 2 were notably susceptible to harmful responses across all categories. We observed that the Model Attacks category was problematic for most models. Expanding prompts increased harmful responses in the Cyber Attacks, Model Attacks, and Phishing categories but decreased them in the Biased Code Generation category. Adding a code skeleton to prompts consistently raised harmfulness across all categories. Large language model alignment still needs further improvement, so we suggest employing red teaming techniques to enhance the safety features of large language models.
...
The rapid advancement of large language models has enabled numerous innovative, but also harmful applications. It is therefore essential to create these models to behave safely and responsibly. One way to improve these models is by red teaming them. In this study, we aim to identify prompts that lead large language models to exhibit unfair or dangerous behavior in software and cybersecurity contexts. We do this by manually creating prompts and manually assessing the harmfulness of the response. Our contributions include a taxonomy of dangerous and unfair use cases of large language models for Code, a dataset of 200 prompts tested on eight models, an investigation into how expanding the prompt, and how adding a code skeleton for the model to complete changes the level of harmfulness. Among the eight models evaluated, only CodeGemma and GPT-3.5-0125 were well-aligned against our taxonomy categories. The unaligned Dolphin-Mixtral and self-aligned Starcoder 2 were notably susceptible to harmful responses across all categories. We observed that the Model Attacks category was problematic for most models. Expanding prompts increased harmful responses in the Cyber Attacks, Model Attacks, and Phishing categories but decreased them in the Biased Code Generation category. Adding a code skeleton to prompts consistently raised harmfulness across all categories. Large language model alignment still needs further improvement, so we suggest employing red teaming techniques to enhance the safety features of large language models.