In semiconductor quantum dots (QDs), charge carrier cooling is in direct competition with processes such as carrier multiplication or hot charge extraction that may improve the light conversion efficiency of photovoltaic devices. Understanding charge carrier cooling is therefore of great interest. We investigate high-energy optical transitions in PbSe QDs using hyperspectral transient absorption spectroscopy. We observe bleaching of optical transitions involving higher valence and conduction bands upon band edge excitation. The kinetics of rise of the bleach of these transitions after a pump laser pulse allow us to monitor, for the first time, cooling of hot electrons and hot holes separately. Our results show that holes cool significantly faster than electrons in PbSe QDs. This is in contrast to the common assumption that electrons and holes behave similarly in Pb chalcogenide QDs and has important implications for the utilization of hot charge carriers in photovoltaic devices.