Multitemporal interferometric synthetic aperture radar (InSAR) is increasingly being used for Earth observations. Inaccurate estimation of the covariance matrix is considered to be the most important source of error in such applications. Previous studies, namely, DeSpecKS and its variants, have demonstrated their advantages in improving the estimation accuracy for distributed targets by means of statistically homogeneous pixels (SHPs). However, these methods may be unreliable for small sample sizes and sensitive to data stacks showing large time spacing due to the variability of the temporal sample. Moreover, these methods are computationally intensive. In this paper, a new algorithm named fast SHP selection (FaSHPS) is proposed to solve both problems. FaSHPS explores the confidence interval for each pixel by invoking the central limit theorem and then selects SHPs using this interval. Based on identified SHPs, two estimators with respect to the despeckling and the bias mitigation of the sample coherence are proposed to refine the elements of the InSAR covariance matrix. A series of qualitative and quantitative evaluations are presented to demonstrate the effectiveness of our method.