Radar has long been considered an important technology for indoor monitoring and assisted living. As ageing has become a worldwide problem, it causes a huge burden on the government’s healthcare expenses and infrastructure. Radar-based human activity recognition (HAR) is foreseen to become a widespread sensing modality for health monitoring at home. Conventional radar-based HAR task usually adopts the amplitude of spectrograms as input to a convolutional neural network (CNN), which can limit the achieved performances. A hybrid fusion model is here proposed, which can integrate multiple radar data domains. The result shows that the proposed framework can achieve superior classification accuracy of 92.1% (+2.5% higher than conventional CNN) and a lighter computational load than the state-of-the-art techniques with 3D-CNN.