Purpose: 18F-fluorodeoxyglucose (18F-FDG) positron emission tomography (PET) is used in the diagnostic process and management of patients with metastatic colorectal cancer (mCRC). Also, 18F-FDG PET radiomic features have been found to hold prognostic value for clinical outcome in mCRC. However, no prognostic model has yet been developed to predict clinical outcome in mCRC using 18F-FDG PET images. Computer-aided pattern recognition can be helpful in this process but needs to be validated. The aim of this work was to develop and evaluate a medical-based explainable artificial intelligence (XAI) framework for discriminating between dichotomous progression free survival (PFS) in patients with mCRC undergoing anti-epidermal growth factor receptor (anti-EGFR) monoclonal antibody (mAb) treatment using pre-treatment 18F-FDG PET images. Methods: We conducted an analysis of 18F-FDG PET images, expressed in standardized uptake values (SUV), obtained from 80 patients with mCRC who were eligible for third-line treatment with an anti-EGFR mAb as part of the IMPACT study. A coronal 2.5D Convolutional Neural Network (CNN) was built to capture features of the 18F-FDG PET images specific for the two patient groups and a medical-based XAI framework was developed to extract the 18F-FDG PET features used by the CNN. The images were randomly divided into a training and a validation set (10-fold cross-validation). Performance of the CNN was evaluated based on the average area under the curve (AUC), accuracy, sensitivity and specificity from the cross-validation. A statistical analysis was performed to assess the predictive value of the 18F-FDG PET features extracted by the XAI framework. Results: The coronal 2.5D-CNN was able to discriminate between dichotomous PFS (median PFS: 152 days) in patients with mCRC undergoing anti-EGFR mAb treatment using pre-treatment 18F-FDG PET images, with an average AUC of 0.95  ± 0.11 (SD), accuracy of 94% ± 12, sensitivity of 91% ± 21 and specificity of 94% ± 21 %. The XAI framework showed that especially low 18F-FDG PET uptake volume features hold significant differences between the two patient groups. Conclusion: The coronal 2.5D-CNN showed good performance to predict dichotomous PFS from pre-treatment 18F-FDG PET images in patients with mCRC undergoing anti-EGFR mAb treatment. Low 18F-FDG PET uptake volume features seem to have potential as IB in this patient cohort, but further validation is required.