Deformable Image Registration (DIR) is a medical imaging process involving the spatial alignment of two or more images using a transformation model that can account for non-rigid deformations. B-spline-based transformation models have emerged as a common approach to express such spatial alignments. However, without additional measures, their flexibility can lead to physically implausible deformations. This flexibility has motivated the inclusion of penalty terms to improve the smoothness and regularity of the transformation. Determining an appropriate weight for this penalty term is difficult, as each registration problem requires a different trade-off between this penalty term and the quality of the transformed image.

Gradient-based methods are commonly used as optimization methods in medical image registration toolboxes due to their computational efficiency and fast convergence rates. However, due to their gradient-based approach, they may converge prematurely in local minima. In this thesis, we investigate the efficacy of a gradient-less alternative: the Real-Valued Gene-pool Optimal Mixing Evolutionary Algorithm (RV-GOMEA), a population-based method that can exploit the problem structure of optimization problems through explicit mappings of dependencies between problem variables. To improve the computational efficiency of RV-GOMEA when applied to DIR, we show how to apply partial evaluations for common image similarity metrics and penalty terms when using B-spline-based transformation models.

We test RV-GOMEA on a synthetic registration problem to better understand its performance in the context of DIR. Based on our findings, we propose several methods that hybridize RV-GOMEA with a gradient-based method and impose specific constraints on the B-spline-based transformation model. We validate the performance of these methods on clinical registration problems and find that RV-GOMEA with a gradient-based local search operator can provide significant benefits over purely gradient-based methods for DIR problems. Additionally, placing specific constraints on the transformation model can increase the regularity of transformations without requiring a penalty term.