On the Retrieval of the Directional Scattering Matrix from Directional Noise