We report the first comprehensive experimental and theoretical study of the optical properties of single crystal phase quantum dots in InP nanowires. Crystal phase quantum dots are defined by a transition in the crystallographic lattice between zinc blende and wurtzite segments and therefore offer unprecedented potential to be controlled with atomic layer accuracy without random alloying. We show for the first time that crystal phase quantum dots are a source of pure single-photons and cascaded photon-pairs from type II transitions with excellent optical properties in terms of intensity and line width. We notice that the emission spectra consist often of two peaks close in energy, which we explain with a comprehensive theory showing that the symmetry of the system plays a crucial role for the hole levels forming hybridized orbitals. Our results state that crystal phase quantum dots have promising quantum optical properties for single photon application and quantum optics.