Impaired cardiac filling in response to increased passive myocardial stiffness contributes to the pathophysiology of heart failure. By leveraging cardiac MRI data and ventricular pressure measurements, we can estimate in vivo passive myocardial stiffness using personalized inverse finite element models. While it is well-known that this approach is subject to uncertainties, only few studies quantify the accuracy of these stiffness estimates. This lack of validation is, at least in part, due to the absence of ground truth in vivo passive myocardial stiffness values. Here, using 3D printing, we created soft, homogenous, isotropic, hyperelastic heart phantoms of varying geometry and stiffness and simulate diastolic filling by incorporating the phantoms into an MRI-compatible left ventricular inflation system. We estimate phantom stiffness from MRI and pressure data using inverse finite element analyses based on a Neo-Hookean model. We demonstrate that our identified softest and stiffest values of 215.7 and 512.3 kPa agree well with the ground truth of 226.2 and 526.4 kPa. Overall, our estimated stiffnesses revealed a good agreement with the ground truth (< 5.8 % error) across all models. Our results suggest that MRI-driven computational constitutive modeling can accurately estimate synthetic heart material stiffnesses in the range of 200–500 kPa.