Application-level Network Behavior Analysis and Anomaly Detection using Density Based Clustering

Abstract

Nowadays, organization networks are facing an increased number of different attacks and existing intrusion and anomaly detection systems fail to keep up. By focusing on security policies, malicious signatures or generic network characteristics, existing systems are not able to cover the full landscape of attacks. In this thesis we try to tackle the problem of anomaly detection on a user network behavior level and an application level. In the proposed framework, network traffic is first separated into different flows based on the mobile application it originates from. Moving forward, the processed network flows are used as input for a flexible noise tolerant behavior modeling framework. The proposed framework is based on density based clustering and tries to identify temporal changes in the user behavior that qualify as anomalous. Moreover, we utilize the model to identify behavioral patterns shared by users and analyze the temporal consistency of user network behavior. To evaluate the framework performance, real network mobile traffic provided a private organization is used. The framework validation is performed by combining the captured network traffic with a conducted employee survey. Overall, the system is able to accurately follow changes in the user behavior based on each application, identify anomalies as well as provide insight on shared behaviors or reoccurring behavioral patterns.