The method of detecting ballast bed defects using ground penetrating radar (GPR) is an important method for guiding the maintenance of railway infrastructure. Currently, this technology primarily relies on time–frequency analysis to assess the condition of the ballast bed and manual interpretation of GPR images to identify defect areas and types, resulting in low automation levels. This paper proposes a bimodal deep learning classification model that enables intelligent classification of moisture and mud pumping defects in ballast beds. This model includes two channels, each processing a different data modality. One channel uses a Multilayer Perceptron (MLP) to extract features of A-scan data in the time domain. The other channel utilizes Short-Time Fourier Transform (STFT) to convert time domain signals into frequency domain signals, which are then processed by a ResNet18 to extract frequency domain features. By fusing the time and frequency features, the proposed Time-Frequency-Fusion ResNet model (TFF-ResNet) demonstrates superior performance. Experimental results show that TFF-ResNet outperforms the standalone MLP and ResNet18 models, with performance improvements of approximately 24% and 14% on the validation dataset, and 21% and 34% on the testing dataset, respectively.