Imaging by inversion of acoustic or electromagnetic wave fields have applications in a wide variety of areas, such as non-destructive testing, biomedical applications, and geophysical explorations. Unfortunately, each modality suffers from its own application specific limitations, typically being difficulties in distinguishing different materials or tissues from each other in the case of acoustic wave fields and a low spatial resolution in the case of electromagnetic wave fields. To exploit the advantages of both imaging modalities, methods to combine them include image fusion, usage of spatial priors and application of joint or multi-physics inversion methods. The latter can be based on empirical relations between acoustic and electromagnetic medium properties or on structural similarity. In this work, two joint inversion algorithms based on structural similarity are presented. To account for the structural similarity the error-functional of standard Born inversion is extended with an additional penalty term. This additional term is either based on the L2-norm of the cross-gradient (CG), i.e. the cross product of the gradients of the acoustic and electromagnetic contrasts or on the L2-norm of the gradient difference (GD), i.e. the difference between the normalized gradients of both contrasts. To test the proposed methods, two synthetic models are considered; one with the gradients of the contrasts pointing in the same direction and one where the gradients point in opposite directions. Results show that the GD constraint significantly improves the resolution for the electromagnetic reconstruction compared to separate BI. The mean square errors (MSE) of the reconstructed profiles for the separate BI are 0.12 for the acoustic and 0.51 for the electromagnetic case, and for the joint GD inversion, 0.09 for the acoustic and 0.46 for the electromagnetic case. The joint GD inversion fails when using the model with the gradients of the contrasts pointing in opposite directions. The joint CG inversion does not enhance the reconstructed images, but shows similar performances for the different models. In conclusion, joint inversion based on structural constraints is shown to improve the electromagnetic resolution, especially using the GD constraint. Further research needs to be conducted to extend the functionality of the GD constraint to acoustic and electromagnetic contrasts with opposite contrast gradient directions.