Explainable Artificial Intelligence for Multi-Modal Margin Assessments in Oncological Surgery

Abstract

Introduction. Ultrasound (US) and diffuse reflectance spectroscopy (DRS) hold significant potential to enhance intra-operative decision-making by providing real-time feedback on surgical resection margins. This is particularly valuable for cancers with complex presentations, such as soft tissue sarcoma (STS), where reducing incomplete (R1) resections can improve patient outcomes.
Aim. This thesis aims to develop resection margin assessment models that leverage US and DRS for automated tumor margin (TM) predictions in ex vivo STS specimens.
Method. A high-quality multi-modal dataset of STS was acquired with a hybrid US-DRS probe. Three artificial intelligence (AI) models were developed: a segmentation model to delineate STS in US images, a classification model to differentiate tissue types in STS using DRS spectra, and a multi-modal regression model combining US and DRS data for TM predictions. The potential added value of Explainable AI (XAI) was explored through a qualitative survey with oncological surgeons.
Results. The dataset included 302 measurement locations from 49 patients, encompassing 17 STS subtypes. The US segmentation model achieved a mean TM prediction accuracy of 0.72 mm and a mean Dice score of 0.97 on an unseen test set. The DRS classifier, leveraging data from six different sampling depths, achieved a tumor detection sensitivity of 0.93 and specificity of 0.86. The multi-modal model, integrating outputs from the segmentation and classification models, demonstrated a mean TM prediction accuracy of 0.57 mm. The survey revealed key requirements for a clinically valuable user interface and showed that XAI integration enhanced model transparency and user trust.
Conclusion. This thesis developed segmentation and classification models for intra-operative resection margin assessments in STS specimens, achieving strong performances compared to existing research. Additionally, a novel multi-modal approach combining US and DRS significantly enhanced the accuracy of TM predictions, highlighting its potential to reduce R1 resections and improve surgical outcomes. Collaboration with clinical end-users ensured the clinical relevance and usability of the proposed model interface, paving the way for integration into surgical practice and advancing oncological care.