The maintenance of buildings, underpinned by the digital twin technique, becomes integral to heritage conservation efforts. To achieve efficient modelling with minimal manual intervention, automated component recognition based on semantic segmentation of point clouds is imperative. Confronted by the challenges of the paucity of requisite datasets and the inherent geometric diversity of historical buildings, a two-step strategy including feature extraction and classification is proposed. First, an improved SHOT descriptor is proposed to extract discriminative features by defining a specific local reference system and concatenating support fields at different scales. The extracted features are then classified with a learning-based network, avoiding a feature learning process that relies on sufficient data. Experiments on real-world heritage point clouds yield 93.7% accuracy and an 80.0% mean-intersection-over-union (mIoU) when descriptors with radii of 0.3 m and 0.9 m are combined, surpassing computationally expensive deep learning networks and data-intensive unsupervised learning. A slight decrease in segmentation performance with random removal of points indicates the high robustness of the proposed method against data missing and sampling density changes. Additionally, a geometric modelling process with an error of less than 10% is introduced to achieve a direct transition from point cloud to model, contributing to the establishment of digital twins for heritage structures.