Usage of asphalt mixture with poor gradation will most likely lead to pavement deficiency. There is a growing need for rapid and non-destructive methods to extract pavement aggregate gradation. In this study, a deep learning-based method that utilizes point clouds data for gradation extraction was proposed. Firstly, a data enhancement algorithm along with three data format conversion methods (aligned point cloud, voxel, and depth image) were proposed to preprocess the original collected point clouds. Subsequently, different neural network models were designed for each data format to extract gradation. Finally, a multi-feature fusion network was developed, which using extraction network as the backbone and additional auxiliary information. In the case study, the MAE loss of multi-feature fusion networks with PointNet, Vox-ResNet34 and GoogLeNet-v4 as the backbone respectively achieved 0.202, 0.142 and 0.046 on the test set, which means an estimation accuracy of more than 95% for the pavement aggregate gradation.