Using near-field, terahertz time-domain spectroscopy (THz-TDS), we investigate how the addition of a dielectric material into a subwavelength-diameter, cylindrical waveguide affects its transmission properties. The THz electric near-field is imaged with deep subwavelength resolution as it emerges from filled and unfilled waveguides. Spectroscopic data measured for waveguides filled with polycrystalline D-tartaric acid, and with polyethylene and silicon powders, illustrate the feasibility of this approach for obtaining spectroscopic information from a tiny sample volume.