Radar-based solutions have attracted great attention in human activity recognition (HAR) for their advantages in accuracy, robustness, and privacy protection. The conventional approaches transform radar signals into feature maps and then directly process them as visual images. While effective, these image-based methods may not be the best solutions in terms of representation efficiency to encode the relevant information for classification. This article proposes a novel HAR method combining sparse theory and PointNet network, with both operations in the time-Doppler (TD) and range-Doppler (RD) domains. First, sparsity-based feature extraction is introduced to use a limited number of sparse solutions to characterize human activities in the form of TD sparse point clouds (TDSP) or dynamic RD sparse point clouds (DRDSP). This new representation is validated by comparing the reconstructed and original signals. Then, PointNet networks are adopted to summarize multidomain features and predict human activity labels by a sparse set of input point clouds. Comprehensive experiments were conducted to demonstrate that the proposed method can yield a higher representation efficiency, classification accuracy, and better generalization capability than existing ones.