Evaluating catastrophic forgetting of state-of-the-art NLP models for predicting moral values
F.I. Arsene (TU Delft - Electrical Engineering, Mathematics and Computer Science)
Pradeep Murukannaiah – Mentor (TU Delft - Interactive Intelligence)
Enrico Liscio – Mentor (TU Delft - Interactive Intelligence)
R. Marroquim – Graduation committee member (TU Delft - Computer Graphics and Visualisation)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Personal moral values represent the motivation behind individuals' actions and opinions. Understanding these values is helpful both in predicting individuals' actions, such as violent protests, and building AI that can better collaborate with humans. Predicting moral values is a challenging problem due to the abstract and subjective essence of moral values. With the help of seven Twitter datasets corresponding to different domains, we train state-of-the-art Natural Language Processing models in predicting moral values. An interesting limitation of the models is that they all suffer from catastrophic forgetting. Catastrophic forgetting is the degree to which models worsen their performance on older data after being trained on new data. We conclude that catastrophic forgetting occurs irrespective of the models being trained and can be mitigated by not only training on new data but by training on a combination of old and new data. This is all possible under one assumption: old data is available. We deliver an evaluation of catastrophic forgetting for each model, explain the differences between the models, and suggest possible future work that can be built upon this research.