Semiconductor devices have been scaled to the point that transport can be dominated by only a single dopant atom. As a result, in a Si fin-type field effect transistor Kondo physics can govern transport when one electron is bound to the single dopant. Orbital (valley) degrees of freedom, apart from the standard spin, strongly modify the Kondo effect in such systems. Owing to the small size and the s-like orbital symmetry of the ground state of the dopant, these orbital degrees of freedom do not couple to external magnetic fields which allows us to tune the symmetry of the Kondo effect. Here we study this tunable Kondo effect and demonstrate experimentally a symmetry crossover from an SU(4) ground state to a pure orbital SU(2) ground state as a function of magnetic field. Our claim is supported by theoretical calculations that unambiguously show that the SU(2) symmetric case corresponds to a pure valley Kondo effect of fully polarized electrons.