Passive application fingerprinting is a technique to detect anomalous outgoing connections. By monitoring the network traffic, a security monitor passively learns the network characteristics of the applications installed on each machine, and uses them to detect the presence of new applications (e.g., malware infection). In this work, we propose HeadPrint, a novel passive fingerprinting approach that relies only on two orthogonal network header characteristics to distinguish applications, namely the order of the headers and their associated values. Our approach automatically identifies the set of characterizing headers, without relying on a predetermined set of header features. We implement HeadPrint, evaluate it in a real-world environment and we compare it with the state-of-the-art solution for passive application fingerprinting. We demonstrate our approach to be, on average, 20% more accurate and 30% more resilient to application updates than the state-of-the-art. Finally, we evaluate our approach in the setting of anomaly detection, and we show that HeadPrint is capable of detecting the presence of malicious communication, while generating significantly fewer false alarms than existing solutions.

We investigate the problem of detecting advanced covert channel techniques, namely victim-aware adaptive covert channels. An adaptive covert channel is considered victim-aware when the attacker mimics the content of its victim’s legitimate communication, such as application-layer metadata, in order to evade detection from a security monitor. In this paper, we show that victim-aware adaptive covert channels break the underlying assumptions of existing covert channel detection solutions, thereby exposing a lack of detection mechanisms against this threat. We first propose a toolchain, Chameleon, to create synthetic datasets containing victim-aware adaptive covert channel traffic. Armed with Chameleon, we evaluate state-of-the-art detection solutions and we show that they fail to effectively detect stealthy attacks. The design of detection techniques against these stealthy attacks is challenging because their network characteristics are similar to those of benign traffic. We explore a deception-based detection technique that we call HoneyTraffic, which generates network messages containing honey tokens, while mimicking the victim’s communication. Our approach detects victim-aware adaptive covert channels by observing inconsistencies in such tokens, which are induced by the attacker attempting to mimic the victim’s traffic. Although HoneyTraffic has limitations in detecting victim-aware adaptive covert channels, it complements existing detection methods and, in combination with them, it can to make evasion harder for an attacker.

We present DECANTeR, a system to detect anomalous outbound HTTP communication, which passively extracts fingerprints for each application running on a monitored host. The goal of our system is to detect unknown malware and backdoor communication indicated by unknown fingerprints extracted from a host's network traffic. We evaluate a prototype with realistic data from an international organization and datasets composed of malicious traffic. We show that our system achieves a false positive rate of 0.9% for 441 monitored host machines, an average detection rate of 97.7%, and that it cannot be evaded by malware using simple evasion techniques such as using known browser user agent values. We compare our solution with DUMONT [24], the current state-of-The-Art IDS which detects HTTP covert communication channels by focusing on benign HTTP traffic. The results show that DECANTeR outperforms DUMONT in terms of detection rate, false positive rate, and even evasion-resistance. Finally, DECANTeR detects 96.8% of information stealers in our dataset, which shows its potential to detect data exfiltration.