Implementation of Delineation Error Detection Systems in Clinical Practice: Do AI-Supported Optimization and Human Preferences Meet?

Abstract

Artificial Intelligence (AI)-based auto-delineation technologies rapidly delineate multiple structures of interest like organs-at-risk and tumors in 3D medical images, reducing personnel load and facilitating time-critical therapies. Despite its accuracy, the AI may produce flawed delineations, requiring clinician attention. Quality assessment (QA) of these delineations is laborious and demanding. Delineation error detection systems aim to aid QA, yet questions linger about clinician adoption, challenges, and time-saving potential. In this study, we address these queries in two stages. First, we investigate the error detection workflow of a radiotherapy technologist and a radiation oncologist from Holland Proton Therapy Center, a Dutch cancer treatment center. The user study revealed which information sources clinicians prefer to use for the error prioritization task and elucidated clinicians' slice-based navigation workflows with and without system assistance. Based on the findings from the user study, we developed a simulation model of the QA process, which we used to assess different error detection workflows on a retrospective cohort of 42 head and neck cancer patients. The simulation study results indicate potential time savings through error and dose information, contingent on per-slice analysis time remaining near the current baseline. Our findings encourage the development of user-centric delineation error detection systems and provide a new way to model and evaluate these systems' potential clinical value.