Internet-wide scanning services are widely used for attack surface discovery across organizations and the Internet. Enterprises, government agencies, and researchers rely on these tools to assess risks to Internet-facing infrastructure. However, their reliability and trustworthiness remain largely unexamined. This paper addresses this gap by comparing results from three commercial scanners – Shodan, ONYPHE, and LeakIX – with findings from our independent experiments using verified Nuclei templates, designed to identify specific vulnerabilities through crafted benign requests. We found that the payload based detections of Shodan are mostly confirmed. Yet, Nuclei finds many more vulnerable endpoints, so defenders might face massive underreporting. For Shodan’s banner-based detections, the opposite issue arises: a significant overreporting of false positives. This indicates that banner-based detections are unreliable. Moreover, three commercial services and Nuclei scans exhibit significant discrepancies. Our work has implications for industry users, policymakers, and the many academic researchers who rely on the results provided by these attack surface management services. By highlighting their shortcomings in vulnerability monitoring, this work serves as a call for action to advance and standardize such services to enhance their trustworthiness.

Malicious binaries have caused data and monetary loss to people, and these binaries keep evolving rapidly nowadays. With tons of new unknown attack binaries, one essential daily task for security analysts and researchers is to analyze and effectively identify malicious parts and report the critical behaviors within the binaries. While manual analysis is slow and ineffective, automated malware report generation is a long-term goal for malware analysts and researchers. This study moves one step toward the goal by identifying essential functions in malicious binaries to accelerate and even automate the analyzing process. We design and implement an expert system based on our proposed graph neural network called MalwareExpert. The system pinpoints the essential functions of an analyzed sample and visualizes the relationships between involved parts. We evaluate our proposed approach using executable binaries in the Windows operating system. The evaluation results show that our approach has a competitive detection performance (97.3% accuracy and 96.5% recall rate) compared to existing malware detection models. Moreover, it gives an intuitive and easy-to-understand explanation of the model predictions by visualizing and correlating essential functions. We compare the identified essential functions reported by our system against several expert-made malware analysis reports from multiple sources. Our qualitative and quantitative analyses show that the pinpointed functions indicate accurate directions. In the best case, the top 2% of functions reported from the system can cover all expert-annotated functions in three steps. We believe that the MalwareExpert system has shed light on automated program behavior analysis.