This study compares the fracture toughness of high-speed steel produced by powder metallurgy and subjected to different heat treatments to obtain either martensitic or bainitic/martensitic microstructures. The heat-treatment process involved austenitization at 1150 °C, followed by either martempering or austempering at 235 °C, and final tempering. Microstructural analysis was performed using electron backscatter diffraction (EBSD), field-emission scanning electron microscopy (FESEM), and X-ray diffraction (XRD). Fracture toughness was evaluated using circumferential notched tensile (CNT) specimens. The results showed that austempered CNT samples exhibited significantly higher fracture toughness compared to martempered ones, indicating improved resistance to crack propagation. Microstructural characterization revealed distinct differences: the austempered samples featured bainitic laths, retained austenite blocks, and martensite plates, whereas the martempered sample contained martensite plates and austenite islands. However, small differences in prior austenite grain size, lath thickness, and dislocation density were insufficient to fully account for the enhanced toughness in the austempered sample. Further analysis indicated that the increased fraction of high-angle grain boundaries and higher kernel average misorientation (KAM) in the austempered sample acted as effective barriers to crack propagation. Additionally, a greater volume fraction of nano-sized carbides contributed to a more pronounced strengthening effect, further enhancing fracture toughness.