Aperture based scanning near field optical microscopes are important instruments to study light at the nanoscale and to understand the optical functionality of photonic nanostructures. In general, a detected image is affected by both the transverse electric and magnetic field components of light. The discrimination of the individual field components is challenging as these four field components are contained within two signals in the case of a polarization resolved measurement. Here, we develop a methodology to solve the inverse imaging problem and to retrieve the vectorial field components from polarization and phase resolved measurements. Our methodology relies on the discussion of the image formation process in aperture based scanning near field optical microscopes. On this basis, we are also able to explain how the relative contributions of the electric and magnetic field components within detected images depend on the chosen probe. We can therefore also describe the influence of geometrical and material parameters of individual probes within the image formation process. This allows probes to be designed that are primarily sensitive either to the electric or magnetic field components of light.