In the past two decades, dental implants have been an essential component of the public health. However, lack of a reliable and safe dental surgery imaging modality remains a major problem. Optical coherence tomography (OCT) is an interferometic optical imaging modality that is somewhat similar to B-mode ultrasound but utilizes infrared light instead of sound waves. It can provide optical biopsy at a very high resolution and in real time. The compact fiber-optic components of OCT make it easy to integrate into dental drill bit to provide the dentists with real-time image feedback about the anatomical structure in the vicinity of the drill bit. This research focuses on the image processing for such a concept designing OCT integrated dental drilling system. Various image processing methods are applied to the OCT images with the intention of finding out the suitable solutions for speckle noise reduction, image reconstruction and dental tissues segmentation. OCT imaging system obtains the cross sectional structure of samples by measuring the back reflected light. The coherent detection makes the OCT images are always subject to speckle noise. The existence of speckle noise limits the interpretation of OCT images. In this study, various speckle reduction digital filters are applied to the images obtained from OCT during high speed rotation. Our results indicate that the adaptive Wiener filter is particularly suitable for our specific application. It is a time–saving method that not only significantly suppresses speckle noise but also preserves image edges. The study also concerns the image restoration due to the non-uniform rotational scanning. One modified method based on the cross correlation between successive A-scans is posted and studied. The assembled images indicate our algorithm is capable of reconstructing the manually-scanned images with an acceptable assemble accuracy. In tissue segmentation, the porous structure of the trabecular bones enables us to discriminate different them from cortical bones. However, due to the intensity decay along the depth direction, the way for image automatic segmentation is still a major problem need to be solved in future research. To test and verify our algorithms, the experiments are conducted to both real animal jawbones and silicone elastomer-based optical tissue phantoms. This proof-of-concept study shows that OCT is able to probe dental tissues in rotational scanning. The results from both samples are highly in accord with each other. This conclusion suggests that our optical phantom is a good jawbone substitute. The durable, easy-to-make phantoms can significantly reduce the cost of future researches.