With rapidly evolving threat landscape surrounding malware, intelligent defenses based on machine learning are paramount. In this chapter, we review the literature proposed in the past decade and identify the state-of-the-art in various related research directions—malware detection, malware analysis, adversarial malware, and malware author attribution. We discuss challenges that emerge when machine learning is applied to malware. We also identify the key issues that need to be addressed by the research community in order to further deepen and systematize research in the malware domain.@en

This chapter contributes to the ongoing discussion of strengthening security by applying AI techniques in the scope of intrusion detection. The focus is set on open-world detection of attacks through data-driven network traffic analysis. This research topic is complementary to the earlier chapter on intelligent malware detection. In this chapter, we revisit the foundations of machine learning-based solutions for network security, which aim to make network defense tools more autonomous, adaptive, proactive and responsive. Specifically, we give a comprehensive introduction to the research on anomaly detection for network intrusion detection – that is, defensive schemes that do not assume complete prior knowledge of malicious patterns and instead learn the notion of normality from benign traffic. Along with outlining the recent advances in the field, we provide insights and reflect on the current limitations and research challenges. Therefore, this chapter presents compelling research opportunities to advance machine learning techniques in network security and push the boundaries of open-world network intrusion detection.@en