We present a theory for quantum impurity relaxometry of magnons in thin films, exhibiting quantitative agreement with recent experiments without needing arbitrary scale factors used in theoretical models thus far. Our theory reveals that chiral coupling between prototypical spin>1/2 quantum impurities and magnons plays a central role in determining impurity relaxation, which is further corroborated by our experiments on nickel films interfaced with nitrogen-vacancy centers. Along with advancing magnonics and understanding decoherence in hybrid quantum platforms with magnets, the ability of a quantum impurity spin to sense chiral magnetic noise presents an opportunity to probe chiral phenomena in condensed matter.