Understanding Risk Extrapolation (REx) and when it finds Invariant Relationships

Understanding Risk Extrapolation (REx) and when it finds Invariant Relationships

Hofland, Jeroen (TU Delft Electrical Engineering, Mathematics and Computer Science)

Krijthe, J.H. (mentor)
Karlsson, R.K.A. (mentor)
Bongers, S.R. (mentor)
Höllt, T. (graduation committee)

Delft University of Technology

Computer Science and Engineering

CSE3000 Research Project

2022-06-24

Generalizing models for new unknown datasets is a common problem in machine learning. Algorithms that perform well for test instances with the same distribution as their training dataset often perform severely on new datasets with a different distribution. This problem is caused by distributional shifts between the training of the model and applying that model to a test domain. This paper addresses whether and in what situations Risk Extrapolation (REx) can tackle this problem of Out-Of-Distribution generalization by exploiting invariant relationships. These relationships are based on features that are invariant across all domains. By learning these relationships, REx aims to learn the concept of the problem we are trying to solve. We show in what situations REx can learn these invariant relationships and when it does not. We translate the definition of an invariant relationship into a homoscedastic synthetic dataset with either covariate, confounded, anti-causal, or hybrid shift. We expose REx to experiments in sample complexity, the number of training domains, and the training domain distance. We show that REx performs better for invariant prediction in situations with larger sample sizes and training domain distance and that if these criteria are met, REx performs equivalently in all four distributional shifts. We also compare REx to Invariant- and Empirical Risk Minimization and show that; REx is less sensitive and thus robust to the shifting of the average distributional variance in the training domains; REx asymptotically out-performs the methods in the more complex distributional shifts.

Risk Extrapolation
Machine Learning
Invariant Relationships
Domain shift
Distributial shift

http://resolver.tudelft.nl/uuid:0a3234aa-9c7a-48ad-8d14-d75b050e6fc5

https://gitlab.com/hofland.jeroen/rex-distributional-shift Code

Student theses

bachelor thesis

© 2022 Jeroen Hofland