Deep learning methods have been demonstrated to be promising in semantic segmentation of point clouds. Existing works focus on extracting informative local features based on individual points and their local neighborhood. They lack consideration of the general structures and latent contextual relations of underlying shapes among points. To this end, we design geometric priors to encode contextual relations of underlying shapes between corresponding point pairs. Geometric prior convolution operator is proposed to explicitly incorporate the contextual relations into the computation. Then, GP-net, which contains geometric prior convolution and a backbone network is constructed. Our experiments show that the performance of our backbone network can be improved by up to 6.9 percent in terms of mean Intersection over Union (mIoU) with the help of geometric prior convolution. We also analyze different design options of geometric prior convolution and GP-net. The GP-net has been tested on the Paris and Lille 3D benchmark, and it achieves the state-of-the-art performance of 74.7 % mIoU.