Methodological Considerations in Exploit Prediction Systems

Methodological Considerations in Exploit Prediction Systems

van Hooff, Sam (TU Delft Electrical Engineering, Mathematics and Computer Science)

Conti, M. (mentor)
Lal, C. (mentor)
Stefanov, Alexandru (graduation committee)
Liang, K. (graduation committee)

Delft University of Technology

Computer Science | Cyber Security

4TU Cyber Security Specialisation

2022-07-08

The growing number of software vulnerabilities being disclosed is posing a challenge to many organisations. With limited patching resources and only a fraction of the vulnerabilities posing a real threat, prioritization is key. Current prioritization methods, such as CVSS, are failing and are sometimes no better than random guessing. Exploit Prediction Systems (EPS) try to fill this gap leveraging a data-driven approach. Related works in the exploit prediction domain make EPS design decisions based on different methodological assumptions. Some of these assumptions are unrealistic or faulty, yielding models that fail to represent a real world situation.
The first contribution of this thesis is the identification of critical methodological assumptions in EPS design and the magnitude of their effects. Then, as second contribution, EPS performance is optimized under restricting yet realistic circumstances, by exploring different techniques to handle class-imbalance, creating richer textual features and/or leveraging different prediction algorithms. The third contribution of this thesis is the implementation of an open-source framework that enables easy experimentation with different machine learning techniques for exploit prediction.

Six critical methodological assumptions have been identified in the area of realistic data collection, correct processing of data, and proper model evaluation. Experiments show that when adhering to the most realistic assumptions, only a fraction of the predictive power of the evaluated EPS is sustained. Almost all prior works fall victim to at least one faulty or unrealistic assumption, and thereby report overoptimistic results.

Substantial improvements are achieved in the optimization step of this thesis. With an optimized EPS with a F1-score of 0.366, performance is insufficient to justify its deployment in a production environment. With the current level of maturity, exploit prediction could have value as a complementary measure to existing vulnerability prioritization systems. Further improvements and more transparent systems are essential for EPS to be suitable for practical usage.

Exploit prediction
Exploit Prediction Systems
EPS
patching management
vulnerability management

http://resolver.tudelft.nl/uuid:b45b527e-4ab0-49bd-9da6-4b59837949ea

Student theses

master thesis

© 2022 Sam van Hooff