Active Exploration for VLM-Guided Anomaly Inspection using a UAV

Abstract

Autonomous exploration by drones in unknown environments has traditionally focused on maximizing spatial coverage without semantic understanding. This thesis presents a framework that integrates vision-language models (VLMs) with adaptive path planning to enable anomaly-aware exploration and inspection. The system employs a three-phase approach: frontier-based exploration, continuous VLM-based anomaly detection, and inspection of detected anomalies. Comparative experiments demonstrated that YOLO+CLIP with negative embeddings achieved the highest F1 score of 0.7218 on the SegmentifyMeIfYouCan benchmark. Experiments showed that dedicated inspection yielded improvements over solely exploration observations. However, system-level evaluation across nine experimental runs revealed that the inspection phase took up most of the mission time (85.9% average), with varying anomaly detection consistency across anomaly instances. False positive analysis identified VLM error as the primary limitation (52% of false positives), followed by simulation artifacts (37%) and semantic ambiguity (11%). The framework successfully demonstrated the feasibility of coupling VLM-based anomaly detection with adaptive planning, though precision limitations and large inspection inefficiencies show opportunities for future work.