Autonomous vision-based navigation is a crucial element for space applications involving a potentially uncooperative target, such as proximity operations for on-orbit servicing or active debris removal. Due to low mass and power characteristics, monocular vision sensors are an attractive choice for onboard vision-based navigation systems. This work focuses on the problem of utilizing images from a monocular vision sensor for estimation of the target's state relative to the servicer spacecraft. Of special interest is the underlying problem of estimating position and attitude (pose) from a single monocular image, given the knowledge of its 3D model. Motivated by the recent advancements in computer vision and machine learning, this work investigates a learning-based approach that has the potential to enable a new paradigm of robust and accurate onboard navigation systems.

A novel framework is proposed for pose initialization and tracking of an uncooperative spacecraft in close-proximity using monocular images and deep learning. An approach based on the use of Convolutional Neural Networks (CNN) is investigated for its scope in enabling reliable on-orbit operations. With a monocular camera as the sole navigation sensor, the underlying problem of relative pose estimation is tackled with deep learning in CNNs to provide robustness to illumination conditions, as opposed to conventional image processing approaches. The CNNs are trained on synthetic images generated from photorealistic renderings of the target spacecraft and integrated into a navigation loop. The emphasis is put on the robustness of such a CNN-based navigation loop, as CNN models are susceptible to learning implicit data distributions that generalize poorly to reality when trained on synthetic data. The central analysis in this work focuses on the European Space Agency’s decommissioned Envisat spacecraft as the target, due to its potential debris generation risk. To that extent, a navigation framework is designed that uses two CNNs- a single-shot object detection network and a high-resolution keypoint detection network, to detect predefined surface keypoints on the target spacecraft. A heatmap representation is used for keypoint detection that provides contextual information per detection and allows indirect quantification of the observation uncertainty. The detected keypoint coordinates and the associated covariances are then used to solve the Perspective-$n$-Points (P$n$P) problem using a Maximum Likelihood P$n$P (MLP$n$P) solver. The MLP$n$P solver provides a pose estimate and the associated uncertainty, which is used by a loosely-coupled Multiplicative Extended Kalman Filter to track the state of the target spacecraft. The pose estimation pipeline in the first two stages is benchmarked and validated on the Spacecraft Pose Estimation Dataset (\textit{SPEED}) from the Stanford Rendezvous Laboratory, containing images of the Tango spacecraft from the PRISMA mission. Subsequently, the framework is evaluated for the Envisat target case, using existing Envisat synthetic image datasets. The complete navigation loop is evaluated on a simulated perturbation-free trajectory of the Envisat spacecraft tumbling along the V-bar.


The proposed framework takes a step towards enabling real on-orbit operations by addressing critical challenges in learning-based methods for the navigation problem. The \textit{SPEED} benchmark results for the pose estimation pipeline show comparable performance with the current state-of-the-art approaches, with a desirable balance in speed and accuracy of the CNNs. The CNN models trained on synthetic images and the resulting pose estimation pipeline also demonstrates robustness to previously unseen real images. Subsequent evaluations on the existing Envisat datasets reveal their inadequacy for training and evaluation of CNNs towards the real on-orbit operation. To tackle this, a new augmented image dataset of the Envisat spacecraft is introduced, which improves over the existing datasets by modeling Earth background and common corruptions in the images. The proposed dataset provides objective improvements for training deep neural networks towards robust and reliable on-orbit operations. Finally, a preliminary navigation analysis on a simplified V-bar scenario for Envisat, reveals that the proposed loosely-coupled estimation in the navigation loop provides an accurate navigation solution.