On the fairness of crowdsourced training data and Machine Learning models for the prediction of subjective properties. The case of sentence toxicity: To be or not to be #$@&%*! toxic? To be or not to be fair?

Training machine learning (ML) models for natural language processing usually requires lots of data that is often acquired through crowdsourcing. In crowdsourcing, crowd workers annotate data samples according to one or more properties, such as the sentiment of a sentence, the violence of a video segment, the aesthetics of an image, ... To...