AxFuncta - Fit instead of Predict for Accidental Scene Conditions

Abstract

In recent years, strong progress has been made in creating learnable affine-equivariant models for downstream tasks such as classification. However, these models encounter increased data requirements to represent all possible transformations due to greater task complexity, while having been shown to generalize poorly to out-of-distribution data. In this work, we introduce a test-time approach for generalizing to out-of-distribution data. Namely, by utilizing a network trained to reconstruct any image that is part of a standardized training distribution, our model can infer an affine transform that moves new samples in-distribution by minimizing their reconstruction loss. With this, this approach closely matches the work of Spatial Transformer Networks, which instead learn to transform data, and inverted neural renderers for pose estimation. Through experiments, we show that this method contains a strong level of out-of-distribution translation and scale invariance, as well as a small level of rotation invariance. Namely, we show that it can handle significant transformations beyond those produced by commonly used benchmarks such as AffNIST. Using this strength we show that this method excels especially in low data regimes, outperforming existing competitors.