Machine Learning models are increasingly used to assist or replace humans in a variety of decision-making domains. However, a lot of concerns have been raised about the impact of these decisions on people’s lives. In this work we focus on two main problems. The first one is that there might be discrimination between different groups of people with respect to their protected attributes in the aforementioned Machine Learning decision making applications. The second one is that there is lack of methods that actually interpret and explain the predictions of these Machine Learning systems which are then used to help decision making. Particularly, we focus on a specific aspect of these decision making applications, namely in Machine Learning training data for the problem of classification. Most research tackles solely individual aspects like trying to detect and mitigate bias and does not pay any attention to explain their reasoning in a human interpretable way. Also, the main method that they use is to balance the distribution of the training data with respect to the protected attribute. However, as we show, this is not always the solution to the problem. On the contrary, we study concurrently three steps (detection, semantic interpretation and mitigation of bias) to overcome these shortcomings and limitations and show that there is the presence of specific visual clues that leads to that bias. Finally, we perform an extensive evaluation of our method in order to verify its efficiency and effectiveness on the particular use-case Gender Bias in Profession Prediction from Images.