In this paper we discuss the stability of heterodyne terahertz receivers based on small volume NbN phonon cooled hot electron bolometers (HEBs). The stability of these receivers can be broken down in two parts: the intrinsic stability of the HEB mixer and the stability of the local oscillator (LO) signal injection scheme. Measurements show that the HEB mixer stability is limited by gain fluctuations with a 1/f spectral distribution. In a 60?MHz noise bandwidth this results in an Allan variance stability time of ? 0.3?s. Measurement of the spectroscopic Allan variance between two intermediate frequency (IF) channels results in a much longer Allan variance stability time, i.e., 3?s between a 2.5 and a 4.7?GHz channel, and even longer for more closely spaced channels. This implies that the HEB mixer 1/f noise is strongly correlated across the IF band and that the correlation gets stronger the closer the IF channels are spaced. In the second part of the paper we discuss atmospheric and mechanical system stability requirements on the LO-mixer cavity path length. We calculate the mixer output noise fluctuations as a result of small perturbations of the LO-mixer standing wave, and find very stringent mechanical and atmospheric tolerance requirements for receivers operating at terahertz frequencies.