Phishing attacks are a growing cause of cybersecurity incidents such as data breaches. With these attacks, malicious actors try to gain access to systems by exploiting the vulnerability of employees. Particularly, intruders use different tricks to create convincing phishing emails to force people to behave less securely. Unfortunately, technological applications are insufficient in detecting all possible phishing attempts. As the consequences of a successful breach can be disastrous for a company, especially in the banking sector, pressure is put on employees to recognise and report potential phishing emails. To aid them in this task, phishing training is provided by employers. Employees are taught what phishing emails look like and which actions they should take if an email appears malicious. However, the effectiveness of these training sessions is difficult to evaluate.
To understand the interaction of employees with phishing emails without influencing their behaviour, we study the emails employees report as suspicious to characterise the security culture at a bank. A better understanding of the behaviour provides grounds for recommendations to improve anti-phishing training and create a safer environment. The newfound metrics can provide an alternative to current methods.
This research uses Exploratory Data Analysis (EDA) to evaluate the email reporting behaviour of employees at a bank to answer the Research Question How can email reporting patterns in a large organisation measure the relationship between phishing training, reported emails and employee behaviour? With this case study, we apply EDA to a large dataset containing bank employees' reported emails over 16 months. We analysed the reported emails and related them to the provided phishing training events. Moreover, we did a text analysis of the emails' content using the Term Frequency - Inverse Document Frequency (TF-IDF) method. Additionally, we extract the dominant topics of the emails using topic modelling. Lastly, with the help of interviews, the results are tied to employees' experiences to understand their behaviour.
The major findings of the research are, firstly, the new metrics we identified to measure the security culture of a company. These metrics were found from both the analysis of employee behaviour over time, as well as the analysis of the email content. From the analysis of the reporting behaviour over time, new metrics include the unique reporters in relation to the total reported emails over time. Besides the unique reporters, unique reported emails can uncover the presence of campaigns. For example, a single email can explain the increase from 50 to 350 daily benign reports. The difference between the total reports and unique reports uncovered this. Secondly, topic analysis and content comparison show similarities between benign and malicious reported emails, indicating an increased vigilance of employees on these attributes.
A second finding originates from the analysis of the email components. One of the components was used in all simulation emails, while it was not present in all the benign and malicious reported emails. This shows that the simulation emails can be extended to include different scenarios. Therefore, we recommend the company to extend the phishing simulation emails to contain varied phishing tactics to expose employees to other types of attacks and incorporate all aspects taught during the E-learning.
Lastly, the analysis shows no concrete relation between the number of reported emails and the timing of the simulation wave. Although the reported p-value of benign emails after the simulation is 0.03, this significance can also be explained by external factors.
With these results, we can measure employees' security culture and awareness in real-world circumstances without influencing the employees' behaviour, providing a new approach to investigating phishing behaviour. Adding to the research of Steves et al. (2020),, the click rates can be explained by more than solely the employees' awareness levels, and new explanations come forward to handle phishing threats. Moreover, the absence of a required test environment for the analysis created a solution for existing gaps. For example, as seen in (Hillman et al., 2023).
A limitation of exploratory data analysis is that results are often ambiguous and mainly provide possible directions for future research. Furthermore, external factors influencing the behaviour could provide alternative reasons for the discussed interactions.
To conclude, by using the reported emails to measure security behaviour related to phishing, we found new metrics which do not influence the employees in their daily behaviour while still providing insights to improve the tactics of a company in combating phishing attacks. Reporting behaviour can be used to analyse the current anti-phishing tactics of a company and provide suggestions for improvements.
Future research should explore the differences in applying the method in other companies and across sectors. Overlap and differences can create an understanding of the diversity in security culture and the effect of external factors. Combining these results with a comprehensive understanding of a company's operations can expose directions for improvement in the security approach. Additionally, the effect of the recommendations can be analysed using the metrics we proposed. This can be done with a follow-up analysis of the behaviour to see whether the desired effect can be observed.