Establishing adequate resection margins during colorectal cancer surgery is challenging. Currently, in up to 30% of the cases the tumor is not completely removed, which emphasizes the lack of a real-time tissue discrimination tool that can assess resection margins up to multiple millimeters in depth. Therefore, we propose to combine spectral data from diffuse reflectance spectroscopy (DRS) with spatial information from ultrasound (US) imaging to evaluate multi-layered tissue structures. First, measurements with animal tissue were performed to evaluate the feasibility of the concept. The phantoms consisted of muscle and fat layers, with a varying top layer thickness of 0-10 mm. DRS spectra of 250 locations were obtained and corresponding US images were acquired. DRS features were extracted using the wavelet transform. US features were extracted based on the graph theory and first-order gradient. Using a regression analysis and combined DRS and US features, the top layer thickness was estimated with an error of up to 0.48 mm. The tissue types of the first and second layers were classified with accuracies of 0.95 and 0.99 respectively, using a support vector machine model.