Impact of Dissimilarity Loss on Out of Distribution Generalization

Abstract

Deep Learning has made neural networks ubiquitous in all kinds of applications. During training, models extract features that are predictive of labels, achieving high accuracy values when tested on in-distribution data. However, issues arise when these extracted features, while indicative in training, do not capture the actual underlying causal features of the data. This reliance on spurious correlations is known as "shortcut learning" and leads to failure to generalize on unseen data. In this paper, we introduce a novel regularizer, dissimilarity loss, which aims to penalize the excessive similarity between representations of samples that share the same spurious predictors. This encourages the model to move beyond shortcut features and learn more robust, task-relevant representations. We show that this additional regularization provides significant benefits to out-of-distribution accuracy compared to a baseline and discuss its drawbacks. Furthermore, we apply it without the spurious feature labels, a regime in which dissimilarity loss still remains effective under distribution shift, and explore other possible directions in which improvements can be made by future work.