In the context of assisted living, human activity recognition (HAR) is of increased importance to maintain people at home living independently longer. Compared with directly using spectrograms (µD) or range profiles (RP) as inputs for classification discarding either range or explicit Doppler information, range-Doppler-surface (RDS) can more fully represent the information contained in the observed activities. However, because the data are collected from different activities, people, and locations, the RDS has a requires non-trivial adjustments for pre-processing as discussed in this paper to maintain the number of points in the RDS to fixed integer. Although it has improved performance (92%) compared to CNN (90%), this algorithm also discards phase information as most algorithms do in HAR with radar. In contrast, the phase information of the range-Doppler domain, although its performance was not the best (88%), it had no obvious weakness in the recognition of all the movements. Our proposed complex field-based fusion network (CFFN) combines the amplitude and phase which improves both the accuracy of classification (94%) as well as accelerating training time by 12.5%.