The point pair feature (PPF) is widely used in industrial applications for estimating 6D poses of known objects from unrecognized point clouds. The key to the success of PPF matching is to establish correct 3D correspondences between the object and the scene, i.e., finding as many valid similar point pairs as possible. Thus, a set of reference points in the scene should be sampled and paired with other points in the scene to create point pair features. However, efficient sampling of scene point pairs has been overlooked in existing frameworks. The novelty of our approach is a new sampling algorithm for selecting scene reference points based on the multi-subpopulation particle swarm optimization (MSPSO) guided by a probability map. We also introduce an effective pose clustering and hypotheses verification method to obtain the optimal pose. Moreover, we optimize the progressive sampling for multi-frame point clouds to improve processing efficiency. The experimental results show that our method outperforms previous methods by 6.6%, 3.9% in terms of accuracy on the public DTU and LineMOD datasets, respectively. We further validate our approach by applying it in a real robot grasping task.