Objective:
In mitral valve surgery, it is important to be aware of adjacent intraoperatively invisible anatomy, to avoid complications and enhance safety. In this feasibility study, we aimed to develop semi-automated intraoperative 3-dimensional (3D) augmented reality (3D-AR) overlays for robotic mitral valve repair.

Methods:
In 5 patients undergoing robot-assisted mitral valve repair, a 3D point cloud was generated, using intraoperatively recorded images from both eyes of the stereoscopic da Vinci camera (Intuitive Surgical, Sunnyvale, CA, USA). An intraoperative 3D-AR overlay was created using a scale-adaptive iterative closest point algorithm and landmarks placed on the mitral valve annulus. Finally, important anatomical structures such as the circumflex artery, Koch’s triangle, and aortic valve leaflets could be visualized as a 3D-AR overlay on top of the surgical vision. To evaluate the accuracy, these 3D point clouds were validated by calculating the 3D point cloud accuracy and landmark registration error (LRE).

Results:
The 3D point clouds and 3D-AR overlays were successfully created for all 5 patients. The 3D point clouds were accurate, with a median error of −0.92 mm, and the LRE was 5.12 mm. The time for creating the 3D-AR overlay was approximately 5 min. Besides creating the 3D-AR overlays, we could visualize the models directly within the robotic console during the surgical procedure.

Conclusions:
We present an algorithm for generating accurate semiautomatic 3D-AR overlays, visualizing essential anatomical structures during robot-assisted mitral valve repair. This may lead to automated intraoperative 3D-AR vision during robotic cardiac surgery, with the potential of increasing safety, accuracy, and efficiency.

Purpose: In this feasibility study, we aimed to create a dedicated pulmonary augmented reality (AR) workflow to enable a semi-automated intraoperative overlay of the pulmonary anatomy during video-assisted thoracoscopic surgery (VATS) or robot-assisted thoracoscopic surgery (RATS). Methods: Initially, the stereoscopic cameras were calibrated to obtain the intrinsic camera parameters. Intraoperatively, stereoscopic images were recorded and a 3D point cloud was generated from these images. By manually selecting the bifurcation key points, the 3D segmentation (from the diagnostic CT scan) was registered onto the intraoperative 3D point cloud. Results: Image reprojection errors were 0.34 and 0.22 pixels for the VATS and RATS cameras, respectively. We created disparity maps and point clouds for all eight patients. Time for creation of the 3D AR overlay was 5 min. Validation of the point clouds was performed, resulting in a median absolute error of 0.20 mm [IQR 0.10–0.54]. We were able to visualize the AR overlay and identify the arterial bifurcations adequately for five patients. In addition to creating AR overlays of the visible or invisible structures intraoperatively, we successfully visualized branch labels and altered the transparency of the overlays. Conclusion: An algorithm was developed transforming the operative field into a 3D point cloud surface. This allowed for an accurate registration and visualization of preoperative 3D models. Using this system, surgeons can navigate through the patient's anatomy intraoperatively, especially during crucial moments, by visualizing otherwise invisible structures. This proposed registration method lays the groundwork for automated intraoperative AR navigation during minimally invasive pulmonary resections.