Achieving accurate and low-latency spectrum sensing on resource-constrained devices is essential but very difficult. Traditional In-phase and Quadrature (I/Q)-based and the ShortTime Fourier Transform (STFT)-based methods fail to balance the computational overhead and classification accuracy. In this paper, we propose a novel framework -Spectrum Painting (SP)- which enables on-device signal classification with low latency and high accuracy. We design new signal processing methods to compress spectrograms while keeping global signal features and augmenting the salient features of small objects. SP achieves high-accuracy signal classification, assisted further by our proposed Dual-channel Convolutional Neural Network (DualCNN). We collect diverse datasets to evaluate the proposed SP, including synthesized data, and testbed data (from up to 18 commodity devices) obtained from real-world environments in the wild and office settings. Experimental results of SP running on Raspberry Pi 4B show a great reduction in latency up to 20 × while maintaining a 95% accuracy. Furthermore, SP demonstrates superior performance within both the centralized learning architecture and the Federated Learning (FL) architecture. For example, the challenging cross-environment evaluation of the SP in the iid-FL scenario yields a substantial accuracy improvement, on average from 24.6% to 83.8%.