In recent years, monitoring the health condition of existing bridges has become a common requirement. By providing an information management system, Bridge Information Model (BrIM) can highly improve the efficiency of health inspection and the reliability of condition evaluation. However, the current modeling processes still largely rely on manual work, where the cost outweighs the benefits. The main barrier lies in the challenging step of semantic segmentation of point clouds. Efforts have been made to identify and segment the structural components of bridges in existing research. But these methods are either dependent on manual data preprocessing or need big training dataset, which, however, has rendered them unpractical in real-world applications. This paper presents a combined local descriptor and machine learning based method to automatically detect structural components of bridges from point clouds. Based on the geometrical features of bridges, we design a multi-scale local descriptor, which is then used to train a deep classification neural network. In the end, a result refinement algorithm is adopted to optimize the segmentation results. Experiments on real-world reinforced concrete (RC) slab and beam-slab bridges show an average precision of 97.26%, recall of 98.00%, and intersection over union (IoU) of 95.38%, which significantly outperforms PointNet. This method has provided a potential solution to semantic segmentation of infrastructures by small sample learning and will contribute to the fulfillment of the automatic BrIM generation of typical highway bridges from the point cloud in the future.