We report the effect of hydrogen on the crystallization process of silicon nanocrystals embedded in a silicon oxide matrix. We show that hydrogen gas during annealing leads to a lower sub-band gap absorption, indicating passivation of defects created during annealing. Samples annealed in pure nitrogen show expected trends according to crystallization theory. Samples annealed in forming gas, however, deviate from this trend. Their crystallinity decreases for increased annealing time. Furthermore, we observe a decrease in the mean nanocrystal size and the size distribution broadens, indicating that hydrogen causes a size reduction of the silicon nanocrystals.