A. Al-Kaswan
Please Note
12 records found
1
Large Language Model (LLM) agents are increasingly proposed for autonomous cybersecurity tasks, but their capabilities in realistic offensive settings remain poorly understood. We present DeepRed, an open-source benchmark for evaluating LLM-based agents on realistic Capture The Flag (CTF) challenges in isolated virtualized environments. DeepRed places an agent in a Kali attacker environment with terminal tools and optional web search, connected over a private network to a target challenge, and records full execution traces for analysis. To move beyond binary solved/unsolved outcomes, we introduce a partial-credit scoring method based on challenge-specific checkpoints derived from public writeups, together with an automated summarise-then-judge labelling pipeline for assigning checkpoint completion from logs. Using DeepRed, we benchmark ten commercially accessible LLMs on ten VM-based CTF challenges spanning different challenge categories. The results indicate that current agents remain limited: the best model achieves only 35% average checkpoint completion, performing strongest on common challenge types and weakest on tasks requiring non-standard discovery and longer-horizon adaptation.
In this dissertation we study this two-sided relationship between AI language models and software security. The first part looks at how these AI systems can be adapted and properly tested for realistic security tasks, going beyond simple, artificial test problems toward settings that better resemble how security work is actually done in practice. This includes teaching an AI model to explain compiled, “black box” software without access to the original code, testing an AI system’s ability to work through a simulated hacking challenge on its own, and checking whether AI coding assistants can be led into helping with harmful requests.
The second part turns the question around, examining the risks that come with using these AI systems themselves. This includes showing that AI models can memorise and later leak fragments of the code they were trained on, including sensitive details that should never have been exposed, and demonstrating that these models can also “remember” and repeat buggy code they encountered during training, even when they would otherwise know better.
Throughout this work, the emphasis lies on testing these AI systems in ways that reflect real-world use, and on openly and responsibly sharing the datasets, tools, and testing methods that were developed, so that other researchers can verify and build on these findings.
With these results, we offer a dual perspective: Large Language Models as tools that can genuinely support security work, and as systems that must themselves be carefully evaluated and secured. With these findings, this dissertation contributes to a more grounded, realistic understanding of both the opportunities and the risks that these AI systems bring to software security. ...
In this dissertation we study this two-sided relationship between AI language models and software security. The first part looks at how these AI systems can be adapted and properly tested for realistic security tasks, going beyond simple, artificial test problems toward settings that better resemble how security work is actually done in practice. This includes teaching an AI model to explain compiled, “black box” software without access to the original code, testing an AI system’s ability to work through a simulated hacking challenge on its own, and checking whether AI coding assistants can be led into helping with harmful requests.
The second part turns the question around, examining the risks that come with using these AI systems themselves. This includes showing that AI models can memorise and later leak fragments of the code they were trained on, including sensitive details that should never have been exposed, and demonstrating that these models can also “remember” and repeat buggy code they encountered during training, even when they would otherwise know better.
Throughout this work, the emphasis lies on testing these AI systems in ways that reflect real-world use, and on openly and responsibly sharing the datasets, tools, and testing methods that were developed, so that other researchers can verify and build on these findings.
With these results, we offer a dual perspective: Large Language Models as tools that can genuinely support security work, and as systems that must themselves be carefully evaluated and secured. With these findings, this dissertation contributes to a more grounded, realistic understanding of both the opportunities and the risks that these AI systems bring to software security.
AST-PAC
AST-guided Membership Inference for Code
The More You Tune, the More They Spill?
An empirical study of data leakage from code models under fine-tuning, quantization, and parameter-efficient adaptation
Although previous research has established that pre-trained code models can regurgitate their training data via extraction attacks, the privacy implications of the increasingly common practice of fine-tuning remain underexplored. Fine-tuning enables entities to adapt models to specialised tasks using smaller, curated datasets that may contain proprietary assets or sensitive information. This study provides a comparative investigation into data extractability, before and after this fine-tuning stage. We develop a benchmark to systematically assess the vulnerability of pre-training and fine-tuning samples to extraction attacks. Our experiments on the Starcoder2 models yield two key findings. First, fine-tuning significantly reduces the extractability of the model’s original pre-training data, decreasing the extraction rate from 54.9% to 23.5%. Second, this benefit is offset by the increased vulnerability in the fine-tuning data. Smaller models, in particular, show a strong propensity to leak this data. Extending this analysis, we find that moderate model compression generally preserves the same memorization patterns, whereas int4 quantization reduces absolute pre-training memorization but amplifies relative fine-tuning leakage. We further find that parameter-efficient fine-tuning mitigates fine-tuning data memorization compared to full fine-tuning, while exhibiting smoother forgetting of pre-training data. A qualitative analysis of 2,000 samples reveals that data structures and licensing information are the most frequently memorized categories. Our work highlights that fine-tuning introduces a privacy trade-off, shifting the risk from vast public pre-training data to smaller, higher-value proprietary fine-tuning data. Model deployers should evaluate these risks and explore strategies to mitigate leakage while preserving performance.
Large Language Models (LLMs) are gaining popularity in the field of Natural Language Processing (NLP) due to their remarkable accuracy in various NLP tasks. LLMs designed for coding are trained on massive datasets, which enables them to learn the structure and syntax of programming languages. These datasets are scraped from the web and LLMs memorise information in these datasets. LLMs for code are also growing, making them more challenging to execute and making users increasingly reliant on external infrastructure.We aim to explore the challenges faced by LLMs for code and propose techniques to measure and prevent memorisation. Additionally, we suggest methods to compress models and run them locally on consumer hardware.
Code comments are a key resource for information about software artefacts. Depending on the use case, only some types of comments are useful. Thus, automatic approaches to clas-sify these comments have been proposed. In this work, we address this need by proposing, STACC, a set of SentenceTransformers- based binary classifiers. These lightweight classifiers are trained and tested on the NLBSE Code Comment Classification tool competition dataset, and surpass the baseline by a significant margin, achieving an average Fl score of 0.74 against the baseline of 0.31, which is an improvement of 139%. A replication package, as well as the models themselves, are publicly available.
Binary reverse engineering is used to understand and analyse programs for which the source code is unavailable. Decompilers can help, transforming opaque binaries into a more readable source code-like representation. Still, reverse engineering is difficult and costly, involving considering effort in labelling code with helpful summaries. While the automated summarisation of decompiled code can help reverse engineers understand and analyse binaries, current work mainly focuses on summarising source code, and no suitable dataset exists for this task. In this work, we extend large pre-trained language models of source code to summarise de-compiled binary functions. Further-more, we investigate the impact of input and data properties on the performance of such models. Our approach consists of two main components; the data and the model. We first build CAPYBARA, a dataset of 214K decompiled function-documentation pairs across various compiler optimisations. We extend CAPYBARA further by removing identifiers, and deduplicating the data. Next, we fine-tune the CodeT5 base model with CAPYBARA to create BinT5. BinT5 achieves the state-of-the-art BLEU-4 score of 60.83, 58.82 and, 44.21 for summarising source, decompiled, and obfuscated decompiled code, respectively. This indicates that these models can be extended to decompiled binaries successfully. Finally, we found that the performance of BinT5 is not heavily dependent on the dataset size and compiler optimisation level. We recommend future research to further investigate transferring knowledge when working with less expressive input formats such as stripped binaries.