Cardiac magnetic resonance (CMR) is used extensively in the diagnosis and management of cardiovascular disease. Deep learning methods have proven to deliver segmentation results comparable to human experts in CMR imaging, however, no successful attempts have been made at fully automated diagnosis. This has been contributed to a lack of sufficiently large datasets required for end-to-end learning of diagnoses. Here we propose to exploit the excellent results obtained in segmentation by jointly training with diagnosis in a multitask learning setting. We hypothesize that segmentation has a regularizing effect on learning and promotes learning of features relevant for diagnosis. Results show a three-fold reduction of the classification error to 0.12 compared to a baseline without segmentation, both results are obtained by training on just 75 cases in a dataset (ACDC) that is equally distributed over 5 classes. ... Read more