Colorectal surgery is the standard treatment for patients with colorectal cancer. To overcome two of the main challenges, the circumferential resection margin and postoperative complications, real-time tissue assessment could be of great benefit during surgery. In this ex vivo study, diffuse reflectance spectroscopy (DRS) was used to differentiate tumor tissue from healthy surrounding tissues in patients with colorectal neoplasia. DRS spectra were obtained from tumor tissue, healthy colon, or rectal wall and fat tissue, for every patient. Data were randomly divided into training (80%) and test (20%) sets. After spectral band selection, the spectra were classified using a quadratic classifier and a linear support vector machine. Of the 38 included patients, 36 had colorectal cancer and 2 had an adenoma. When the classifiers were applied to the test set, colorectal cancer could be discriminated from healthy tissue with an overall accuracy of 0.95 (±0.03). This study demonstrates the possibility to separate colorectal cancer from healthy surrounding tissue by applying DRS. High classification accuracies were obtained both in homogeneous and inhomogeneous tissues. This is a fundamental step toward the development of a tool for real-time in vivo tissue assessment during colorectal surgery.