Evaluating catastrophic forgetting of state-of-the-art NLP models for predicting moral values

Abstract

Personal moral values represent the motivation behind individuals' actions and opinions. Understanding these values is helpful both in predicting individuals' actions, such as violent protests, and building AI that can better collaborate with humans. Predicting moral values is a challenging problem due to the abstract and subjective essence of moral values. With the help of seven Twitter datasets corresponding to different domains, we train state-of-the-art Natural Language Processing models in predicting moral values. An interesting limitation of the models is that they all suffer from catastrophic forgetting. Catastrophic forgetting is the degree to which models worsen their performance on older data after being trained on new data. We conclude that catastrophic forgetting occurs irrespective of the models being trained and can be mitigated by not only training on new data but by training on a combination of old and new data. This is all possible under one assumption: old data is available. We deliver an evaluation of catastrophic forgetting for each model, explain the differences between the models, and suggest possible future work that can be built upon this research.