AA

A. Al-Kaswan

info

Please Note

12 records found

Conference paper (2026) - Jonathan Katzy, Ali Al-Kaswan, Razvan Mihai Popescu, Zhou Yang
Large language models are increasingly used to evaluate and support software engineering tasks, yet the validity of these evaluations is often undermined by uncertainty about whether benchmark instances were seen during pretraining. This can lead to data contamination, which may inflate performance and result in misleading conclusions about model capability. Despite this, the training corpora of many modern models are only partially disclosed, making direct decontamination infeasible. This creates a need for practical methods that can detect a large language models' prior exposure to training data without access to the full training corpus. To address this challenge, we organize the first Poisoned Chalice of LLM Evaluation Competition, co-located with the FSE-AIWare 2026 Competition Track. The competition frames contamination detection as a white-box membership inference task on source code and provides participants with curated datasets, target models, baseline attacks, and a final evaluation on a held-out model and dataset. This design encourages methods that generalize beyond superficial dataset artifacts and beyond a single training setting. This paper reports the setup and results of the competition. More broadly, the competition aims to catalyze the community around trustworthy LLM evaluation for software engineering. ...
Large Language Model (LLM) agents are increasingly proposed for autonomous cybersecurity tasks, but their capabilities in realistic offensive settings remain poorly understood. We present DeepRed, an open-source benchmark for evaluating LLM-based agents on realistic Capture The Flag (CTF) challenges in isolated virtualized environments. DeepRed places an agent in a Kali attacker environment with terminal tools and optional web search, connected over a private network to a target challenge, and records full execution traces for analysis. To move beyond binary solved/unsolved outcomes, we introduce a partial-credit scoring method based on challenge-specific checkpoints derived from public writeups, together with an automated summarise-then-judge labelling pipeline for assigning checkpoint completion from logs. Using DeepRed, we benchmark ten commercially accessible LLMs on ten VM-based CTF challenges spanning different challenge categories. The results indicate that current agents remain limited: the best model achieves only 35% average checkpoint completion, performing strongest on common challenge types and weakest on tasks requiring non-standard discovery and longer-horizon adaptation. ...
Doctoral thesis (2026) - A. Al-Kaswan, A. van Deursen, M. Izadi
Artificial Intelligence (AI) systems that can read and write computer code, so-called Large Language Models (LLMs), are becoming a routine part of how software is built. Because these systems are good at reasoning about both code and natural language, they are not just useful for everyday programming, they are also relevant to security work, where understanding how a piece of software behaves, or how it might be attacked, is central. At the same time, the very properties that make these AI systems useful also create risks of their own: a model trained on huge amounts of code can accidentally leak private information from that training data, produce code with security flaws, or even help someone carry out a harmful task.

In this dissertation we study this two-sided relationship between AI language models and software security. The first part looks at how these AI systems can be adapted and properly tested for realistic security tasks, going beyond simple, artificial test problems toward settings that better resemble how security work is actually done in practice. This includes teaching an AI model to explain compiled, “black box” software without access to the original code, testing an AI system’s ability to work through a simulated hacking challenge on its own, and checking whether AI coding assistants can be led into helping with harmful requests.

The second part turns the question around, examining the risks that come with using these AI systems themselves. This includes showing that AI models can memorise and later leak fragments of the code they were trained on, including sensitive details that should never have been exposed, and demonstrating that these models can also “remember” and repeat buggy code they encountered during training, even when they would otherwise know better.

Throughout this work, the emphasis lies on testing these AI systems in ways that reflect real-world use, and on openly and responsibly sharing the datasets, tools, and testing methods that were developed, so that other researchers can verify and build on these findings.

With these results, we offer a dual perspective: Large Language Models as tools that can genuinely support security work, and as systems that must themselves be carefully evaluated and secured. With these findings, this dissertation contributes to a more grounded, realistic understanding of both the opportunities and the risks that these AI systems bring to software security. ...

AST-guided Membership Inference for Code

Code Large Language Models are frequently trained on massive datasets containing restrictively licensed source code. This creates urgent data governance and copyright challenges. Membership Inference Attacks (MIAs) can serve as an auditing mechanism to detect unauthorized data usage in models. While attacks like the Loss Attack provide a baseline, more involved methods like Polarized Augment Calibration (PAC) remain underexplored in the code domain. This paper presents an exploratory study evaluating these methods on 3B-7B parameter code models. We find that while PAC generally outperforms the Loss baseline, its effectiveness relies on augmentation strategies that disregard the rigid syntax of code, leading to performance degradation on larger, complex files. To address this, we introduce AST-PAC, a domain-specific adaptation that utilizes Abstract Syntax Tree (AST) based perturbations to generate syntactically valid calibration samples. Preliminary results indicate that AST-PAC improves as syntactic size grows, where PAC degrades, but under-mutates small files and underperforms on alphanumeric-rich code. Overall, the findings motivate future work on syntax-aware and size-adaptive calibration as a prerequisite for reliable provenance auditing of code language models. ...

An empirical study of data leakage from code models under fine-tuning, quantization, and parameter-efficient adaptation

Journal article (2026) - F. Salerno, A. Al-Kaswan, M. Izadi
Although previous research has established that pre-trained code models can regurgitate their training data via extraction attacks, the privacy implications of the increasingly common practice of fine-tuning remain underexplored. Fine-tuning enables entities to adapt models to specialised tasks using smaller, curated datasets that may contain proprietary assets or sensitive information. This study provides a comparative investigation into data extractability, before and after this fine-tuning stage. We develop a benchmark to systematically assess the vulnerability of pre-training and fine-tuning samples to extraction attacks. Our experiments on the Starcoder2 models yield two key findings. First, fine-tuning significantly reduces the extractability of the model’s original pre-training data, decreasing the extraction rate from 54.9% to 23.5%. Second, this benefit is offset by the increased vulnerability in the fine-tuning data. Smaller models, in particular, show a strong propensity to leak this data. Extending this analysis, we find that moderate model compression generally preserves the same memorization patterns, whereas int4 quantization reduces absolute pre-training memorization but amplifies relative fine-tuning leakage. We further find that parameter-efficient fine-tuning mitigates fine-tuning data memorization compared to full fine-tuning, while exhibiting smoother forgetting of pre-training data. A qualitative analysis of 2,000 samples reveals that data structures and licensing information are the most frequently memorized categories. Our work highlights that fine-tuning introduces a privacy trade-off, shifting the risk from vast public pre-training data to smaller, higher-value proprietary fine-tuning data. Model deployers should evaluate these risks and explore strategies to mitigate leakage while preserving performance. ...
Conference paper (2026) - Ali Al-Kaswan, Claudio Spiess, Prem Devanbu, Arie Van Deursen, Maliheh Izadi
Large language models are increasingly used for code generation and debugging, but their outputs can still contain bugs that originate from training data. Distinguishing whether an LLM prefers correct code, or a familiar incorrect version might be influenced by what it's been exposed to during training. We introduce an exposure-aware evaluation framework that quantifies how prior exposure to buggy versus fixed code influences a model's preference. Using the ManySStuBs4J benchmark, we apply Data Portraits for membership testing on the Stack-V2 corpus to estimate whether each buggy and fixed variant was seen during training. We then stratify examples by exposure and compare model preference using code completion as well as multiple likelihood-based scoring metrics. We find that most examples (67%) have neither variant in the training data, and when only one is present, fixes are more frequently present than bugs. In model generations, models reproduce buggy lines far more often than fixes, with bug-exposed examples amplifying this tendency and fix-exposed examples showing only marginal improvement. In likelihood scoring, minimum and maximum token-probability metrics consistently prefer the fixed code across all conditions, indicating a stable bias toward correct fixes. In contrast, metrics like the Gini coefficient reverse preference when only the buggy variant was seen. Our results indicate that exposure can skew bug-fix evaluations and highlight the risk that LLMs may propagate memorised errors in practice. ...
Conference paper (2025) - Fabio Salerno, Ali Al-Kaswan, Maliheh Izadi
Code language models, while widely popular, are often trained on unsanitized source code gathered from across the Internet. Previous work revealed that pre-trained models can remember the content of their training data and regurgitate them through data extraction attacks. Due to the large size of current models, only a few entities have the resources for pre-training such models. However, fine-tuning requires fewer resources and is increasingly used by both small and large entities for its effectiveness on specialized data. Such small curated data for finetuning might contain sensitive information or proprietary assets. In this study, we attack both pre-trained and fine-tuned code language models to investigate the extent of data extractability. We first develop a custom benchmark to assess the vulnerability of both pre-training and fine-tuning samples to extraction attacks. Our findings reveal that 54.9% of extractable pre-training data could be retrieved from StarCoder2-15B, whereas this number decreased to 23.5% after fine-tuning. This indicates that finetuning reduces the extractability of pre-training data. However, compared to larger models, fine-tuning smaller models increases their vulnerability to data extraction attacks on fine-tuning data. Given the potential sensitivity of fine-tuning data, this can lead to more severe consequences. Lastly, we also manually analyzed 2000 extractable samples before and after fine-tuning. We also found that data carriers and licensing information are the most likely data categories to be memorized from pre-trained and finetuned models, while the latter is the most likely to be forgotten after fine-tuning. ...
Conference paper (2024) - Ali Al-Kaswan
Large Language Models (LLMs) are gaining popularity in the field of Natural Language Processing (NLP) due to their remarkable accuracy in various NLP tasks. LLMs designed for coding are trained on massive datasets, which enables them to learn the structure and syntax of programming languages. These datasets are scraped from the web and LLMs memorise information in these datasets. LLMs for code are also growing, making them more challenging to execute and making users increasingly reliant on external infrastructure.We aim to explore the challenges faced by LLMs for code and propose techniques to measure and prevent memorisation. Additionally, we suggest methods to compress models and run them locally on consumer hardware. ...
Previous work has shown that Large Language Models are susceptible to so-called data extraction attacks. This allows an attacker to extract a sample that was contained in the training data, which has massive privacy implications. The construction of data extraction attacks is challenging, current attacks are quite inefficient, and there exists a significant gap in the extraction capabilities of untargeted attacks and memorization. Thus, targeted attacks are proposed, which identify if a given sample from the training data, is extractable from a model. In this work, we apply a targeted data extraction attack to the SATML2023 Language Model Training Data Extraction Challenge. We apply a two-step approach. In the first step, we maximise the recall of the model and are able to extract the suffix for 69% of the samples. In the second step, we use a classifier-based Membership Inference Attack on the generations. Our AutoSklearn classifier achieves a precision of 0.841. The full approach reaches a score of 0.405 recall at a 10% false positive rate, which is an improvement of 34% over the baseline of 0.301. ...
Conference paper (2023) - A. Al-Kaswan, M. Izadi
In recent years, Large Language Models (LLMs) have gained significant popularity due to their ability to generate human-like text and their potential applications in various fields, such as Software Engineering. LLMs for Code are commonly trained on large unsanitized corpora of source code scraped from the Internet. The content of these datasets is memorized and emitted by the models, often in a verbatim manner. In this work, we will discuss the security, privacy, and licensing implications of memorization. We argue why the use of copyleft code to train LLMs is a legal and ethical dilemma. Finally, we provide four actionable recommendations to address this issue. ...
Conference paper (2023) - A. Al-Kaswan, M. Izadi, A. van Deursen
Code comments are a key resource for information about software artefacts. Depending on the use case, only some types of comments are useful. Thus, automatic approaches to clas-sify these comments have been proposed. In this work, we address this need by proposing, STACC, a set of SentenceTransformers- based binary classifiers. These lightweight classifiers are trained and tested on the NLBSE Code Comment Classification tool competition dataset, and surpass the baseline by a significant margin, achieving an average Fl score of 0.74 against the baseline of 0.31, which is an improvement of 139%. A replication package, as well as the models themselves, are publicly available. ...
Conference paper (2023) - Ali Al-Kaswan, Toufique Ahmed, Maliheh Izadi, Anand Ashok Sawant, Premkumar Devanbu, Arie van Deursen
Binary reverse engineering is used to understand and analyse programs for which the source code is unavailable. Decompilers can help, transforming opaque binaries into a more readable source code-like representation. Still, reverse engineering is difficult and costly, involving considering effort in labelling code with helpful summaries. While the automated summarisation of decompiled code can help reverse engineers understand and analyse binaries, current work mainly focuses on summarising source code, and no suitable dataset exists for this task. In this work, we extend large pre-trained language models of source code to summarise de-compiled binary functions. Further-more, we investigate the impact of input and data properties on the performance of such models. Our approach consists of two main components; the data and the model. We first build CAPYBARA, a dataset of 214K decompiled function-documentation pairs across various compiler optimisations. We extend CAPYBARA further by removing identifiers, and deduplicating the data. Next, we fine-tune the CodeT5 base model with CAPYBARA to create BinT5. BinT5 achieves the state-of-the-art BLEU-4 score of 60.83, 58.82 and, 44.21 for summarising source, decompiled, and obfuscated decompiled code, respectively. This indicates that these models can be extended to decompiled binaries successfully. Finally, we found that the performance of BinT5 is not heavily dependent on the dataset size and compiler optimisation level. We recommend future research to further investigate transferring knowledge when working with less expressive input formats such as stripped binaries. ...