Extreme ultraviolet pulses as generated by high harmonic generation (HHG) are a powerful tool for both time-resolved spectroscopy and coherent diffractive imaging. However, the combination of these techniques to achieve spatio-spectro-temporal data remains hardly explored due to the challenging and time-consuming data acquisition. Here, we present Fourier-transform spectroscopic holography (FTSH), an interferometric approach to spectroscopic imaging that combines Fourier-transform spectroscopy with holography. By encoding spectral information in the measured diffraction pattern, FTSH dramatically reduces the sampling requirements by one order of magnitude. This enables us to record full spectroscopic imaging data in less than 2 minutes, and makes FTSH especially promising for femtosecond time-resolved nano-spectroscopy using table-top HHG sources.