Light is a union of electric and magnetic fields, and nowhere is the complex relationship between these fields more evident than in the near fields of nanophotonic structures. There, complicated electric and magnetic fields varying over subwavelength scales are generally present, which results in photonic phenomena such as extraordinary optical momentum, superchiral fields, and a complex spatial evolution of optical singularities. An understanding of such phenomena requires nanoscale measurements of the complete optical field vector. Although the sensitivity of near-field scanning optical microscopy to the complete electromagnetic field was recently demonstrated, a separation of different components required a priori knowledge of the sample. Here, we introduce a robust algorithm that can disentangle all six electric and magnetic field components from a single near-field measurement without any numerical modeling of the structure. As examples, we unravel the fields of two prototypical nanophotonic structures: a photonic crystal waveguide and a plasmonic nanowire. These results pave the way for new studies of complex photonic phenomena at the nanoscale and for the design of structures that optimize their optical behavior.