Minimally Invasive Procedures (MIPs) emerged as an alternative to more invasive surgical approaches, offering patient benefits such as smaller incisions, less pain, and shorter hospital stay. In one class of MIPs, where natural body lumens or small incisions are used to access deeper anatomical locations, Flexible Surgical and Interventional Robots (FSIRs) such as catheters and endoscopes are widely used. Due to their flexible and compliant nature, FSIRs can be inserted via natural orifices or small incisions, then moved towards hard-to-reach targets to perform interventional tasks. However, existing FSIRs are confronted with challenges in sensing, control, and navigation. These issues stem from the robot's non-linear behavior and the intricate nature of the lumens, where accurately modeling the complex interactions and disturbances proves to be exceptionally difficult. The rapid advances in Machine Learning (ML) have facilitated the widespread adoption of ML techniques in FSIRs. This article provides an overview of these efforts by first introducing a classification of existing ML algorithms, including traditional ML methods and modern Deep Learning (DL) approaches, commonly used in FSIRs. Next, the use of ML algorithms is surveyed per sub-domain, namely for perception, modeling, control, and navigation. Trends, popularity, strengths, and/or limitations of different ML algorithms are analyzed. The different roles that ML plays among tasks are investigated and described. Finally, discussions are conducted on the limitations and the prospects of ML in MIPs.@en

Continuum robots such as robotic catheters are increasingly being used in minimally invasive surgery. Compliance contributes to enhanced safety during e.g. catheter insertion, however, estimation of contact force and location may help clinicians avoiding exerting excessive force. Ultimately this could lead to faster and safer interventions. Researchers proposed force sensors integrated in the catheter tip in the past. However, such sensors add extra complexity to the catheter design. Also, tip force sensors do not provide insights on forces that act along the catheter length. This paper proposes a data-driven approach for localizing contact forces that appear over the length of the catheter. The proposed approach consists of a collision detection method and a contact localization method. The framework only requires the measurement of the catheter's shape which can be done by an embedded multi-core Fiber Bragg Grating fiber. The method was validated experimentally with a 3D-printed continuum robot with an integrated multi-core fiber. A second contact localization method which is based on identifying the discontinuity in the measured curvature, is also implemented and compared with the proposed method. The static and dynamic experiments show a mean average localization error of 2.3 mm and 4.3 mm which correspond to respectively 3.3% and 6.1% of a 70 mm long flexible robot. These findings demonstrate that the proposed framework outperforms the previous methods and yield promising results. The contact state estimation algorithm can detect collisions in at most approximately 1.08s.@en

Optical fiber-based shape sensing is gaining popularity in cardiac catheterization lately. Typically, these procedures are taking place under the guidance of fluoroscopy. However, fluoroscopy has several disadvantages. Thanks to fiber optic shape sensing and Electromagnetic Tracking (EMT), the 3D catheter shape can now be tracked in real-time without the need for fluoroscopy. Traditional optical fiber and EMT-based shape tracking methods have the drawback of the highest shape sensing error at the tip. The information offered by the EMT sensors is used mainly to localize the estimated shape in a fixed coordinate frame. In this letter, a novel approach for tracking the catheter is introduced to address the aforementioned problem. The catheter shape is directly reconstructed in the EMT coordinate frame by approximating the catheter shape by a number of Bézier curves while taking into account the curvatures measured by the optical fiber. Both 2D and 3D shape sensing experiments are conducted. The results of the 3D experiment show that the proposed method reduces the mean shape tracking error by approximately 38% (from 12.1 mm to 5.4 mm for a sensed length of 540 mm long) compared to the traditional method where the same number of sensors are used.@en

In cardiovascular interventions, when steering catheters and especially robotic catheters, great care should be paid to prevent applying too large forces on the vessel walls as this could dislodge calcifications, induce scars or even cause perforation. To address this challenge, this paper presents a novel compliant motion control algorithm that relies solely on position sensing of the catheter tip and knowledge of the catheter's behavior. The proposed algorithm features a data-driven tip position controller. The controller is trained based on a so-called control Long Short-Term Memory Network (control-LSTM). Trajectory following experiments are conducted to validate the quality of the proposed control-LSTM. Results demonstrated superior positioning capability with sub-degree precision of the new approach in the presence of severe rate-dependent hysteresis. Experiments both in a simplified setup as well as in an aortic phantom further show that the proposed approach allows reducing the interaction forces with the environment by around 70%. This work shows how deep learning can be exploited advantageously to avoid tedious modeling that would be needed to precisely steer continuum robots in constrained environments such as the patient's vasculature.@en

In this article, a deep learning method for the shape sensing of continuum robots based on multicore fiber bragg grating (FBG) fiber is introduced. The proposed method, based on an artificial neural network (ANN), differs from traditional approaches, where accurate shape reconstruction requires a tedious characterization of many characteristic parameters. A further limitation of traditional approaches is that they require either multiple fibers, whose location relative to the centerline must be precisely known (calibrated), or a single multicore fiber whose position typically coincides with the neutral line. The proposed method addresses this limitation and, thus, allows shape sensing based on a single multicore fiber placed off-center. This helps in miniaturizing and leaves the central channel available for other purposes. The proposed approach was compared to a recent state-of-the-art model-based shape sensing approach. A two-degree-of-freedom benchtop fluidics-driven catheter system was built to validate the proposed ANN. The proposed ANN-based shape sensing approach was evaluated on a 40-mm-long steerable continuum robot in both 3-D free-space and 2-D constrained environments, yielding an average shape sensing error of 0.24 and 0.49 mm, respectively. With these results, the superiority of the proposed approach compared to the recent model-based shape sensing method was demonstrated.@en

Minimally invasive catheter-based interventions normally take place under the guidance of fluoroscopy. However, fluoroscopy is harmful to both patients and clinicians. Moreover, it only offers 2-D shape visualization of flexible devices. To solve the problem of harmful radiation and offer 3-D pose and shape information, recent studies propose a combination of electromagnetic tracking (EMT) sensors and multicore fiber Bragg grating (FBG) fiber sensing. However, for robust localization, at least two EMT sensors are required to be attached to each multicore fiber. This may make the catheter overly complex and fragile. Furthermore, the inability of multicore FBG fibers to distinguish between twist-induced strain and bend-induced strain impacts shape sensing accuracy. This article proposes a new approach offering a precise shape sensing method that is robust against torsional twists and exploits symmetry and geometry to compensate for limited sensing information. The proposed approach originates from the observation that many interventional procedures employ a plurality of concentric instruments. By distributing sensors over these instruments, the complexity per instrument can be kept acceptable. The proposed sensor fusion approach ensures robust and superior shape reconstruction. Experiments in 3-D with ground truth generated by a stereo vision system have been done and yielded promising results. Compared to the state-of-the-art methods, the presented framework uses only half of the required EMT sensors per instrument resulting in significant spatial conservation while improving the catheter shape tracking accuracy by 57%.@en

Endovascular intervention is a minimally invasive method for treating cardiovascular diseases. Although fluoroscopy, known for real-time catheter visualization, is commonly used, it exposes patients and physicians to ionizing radiation and lacks depth perception due to its 2D nature. To address these limitations, a study was conducted using teleoperation and 3D visualization techniques. This <italic>in-vitro</italic> study involved the use of a robotic catheter system and aimed to evaluate user performance through both subjective and objective measures. The focus was on determining the most effective modes of interaction. Three interactive modes for guiding robotic catheters were compared in the study: 1) Mode GM, using a gamepad for control and a standard 2D monitor for visual feedback; 2) Mode GH, with a gamepad for control and HoloLens providing 3D visualization; and 3) Mode HH, where HoloLens serves as both control input and visualization device. Mode GH outperformed other modalities in subjective metrics, except for mental demand. It exhibited a median tracking error of 4.72 mm, a median targeting error of 1.01 mm, a median duration of 82.34 s, and a median natural logarithm of dimensionless squared jerk of 40.38 in the <italic>in-vitro</italic> study. Mode GH showed 8.5%, 4.7%, 6.5%, and 3.9% improvements over Mode GM and 1.5%, 33.6%, 34.9%, and 8.1% over Mode HH for tracking error, targeting error, duration, and dimensionless squared jerk, respectively. To sum up, the user study emphasizes the potential benefits of employing HoloLens for enhanced 3D visualization in catheterization. The user study also illustrates the advantages of using a gamepad for catheter teleoperation, including user-friendliness and passive haptic feedback, compared to HoloLens. To further gauge the potential of using a more traditional joystick as a control input device, an additional study utilizing the Haption VirtuoseTM robot was conducted. It reveals the potential for achieving smoother trajectories, with a 38.9% reduction in total path length compared to a gamepad, potentially due to its larger range of motion and single-handed control.@en

Endovascular intervention is a minimally invasive method for treating cardiovascular diseases. Although fluoroscopy, known for real-time catheter visualization, is commonly used, it exposes patients and physicians to ionizing radiation and lacks depth perception due to its 2D nature. To address these limitations, a study was conducted using teleoperation and 3D visualization techniques. This <italic>in-vitro</italic> study involved the use of a robotic catheter system and aimed to evaluate user performance through both subjective and objective measures. The focus was on determining the most effective modes of interaction. Three interactive modes for guiding robotic catheters were compared in the study: 1) Mode GM, using a gamepad for control and a standard 2D monitor for visual feedback; 2) Mode GH, with a gamepad for control and HoloLens providing 3D visualization; and 3) Mode HH, where HoloLens serves as both control input and visualization device. Mode GH outperformed other modalities in subjective metrics, except for mental demand. It exhibited a median tracking error of 4.72 mm, a median targeting error of 1.01 mm, a median duration of 82.34 s, and a median natural logarithm of dimensionless squared jerk of 40.38 in the <italic>in-vitro</italic> study. Mode GH showed 8.5%, 4.7%, 6.5%, and 3.9% improvements over Mode GM and 1.5%, 33.6%, 34.9%, and 8.1% over Mode HH for tracking error, targeting error, duration, and dimensionless squared jerk, respectively. To sum up, the user study emphasizes the potential benefits of employing HoloLens for enhanced 3D visualization in catheterization. The user study also illustrates the advantages of using a gamepad for catheter teleoperation, including user-friendliness and passive haptic feedback, compared to HoloLens. To further gauge the potential of using a more traditional joystick as a control input device, an additional study utilizing the Haption VirtuoseTM robot was conducted. It reveals the potential for achieving smoother trajectories, with a 38.9% reduction in total path length compared to a gamepad, potentially due to its larger range of motion and single-handed control.@en

Precise control of robotic catheters remains challenging in interventions. Inherent non-linearities such as hysteresis and external disturbances such as blood flow or contact with the vessel walls have a large impact on the reachable positioning precision. As inaccurate positioning of the catheter tip could lead to tissue damage, controllers that would perform adequately in the presence of hysteresis and environmental contacts would be highly desirable. This paper proposes a method based on multiple Long Short-Term Memory Networks (LSTMs). To this end, a so-called free-space-LSTM (f-LSTM) is trained in order to steer the catheter when it moves in free. Constrained-space-LSTMs (c-LSTMs) are trained to drive the catheter when it is in contact with an obstacle. Based on contact estimation methods, LSTMs are switched. The f-LSTM and c-LSTMs are first tested in free space motion and under constraint situations. The results reveal that LSTMs perform well (RMSE < 0.5 mm) for a steerable robot section with a total length of 77 mm when tested in the same situation where trained. However, when f-LSTM and c-LSTM were tested in an environment different from the one in which they were trained, errors tended to increase. The results highlight the need to exhaustively estimate the contact location and switch between different LSTMs accordingly. The effective working range of a c-LSTM was investigated as well. Experiments showed that a well-Trained single c-LSTM could be used effectively in a range of 8.8 mm among the entire length of a steerable catheter section, while maintaining the average tip positioning error below 2 mm in this range. @en