How Does the Downstream Accuracy of Barlow Twins Scale with Pre-training Set Size?

Abstract

Modern computer vision often reuses a single model, trained once on many images, as a start- ing point for new tasks. Because labels are ex- pensive, a common way to train such a model is self-supervised learning (SSL), which learns from unlabeled images. SSL normally uses millions of images, and it is unclear how well it works when far fewer are available. We study one SSL method, Barlow Twins, in that case. We pre-train a small vision transformer (5.4M parameters) on parts of Tiny-ImageNet, from 1k to 100k unlabeled images, and train every run for the same 1000 epochs, so the only thing that changes is the amount of data. We then freeze each model and measure how well its features transfer to the 19 VTAB-1k tasks. Pre- training helps at every dataset size: the VTAB-1k average rises from 33.7% with 1k images to 39.2% with 100k, well above a 24.4% untrained baseline. But this average hides large differences between tasks: accuracy on natural-image tasks keeps rising with data, while accuracy on more specialized and structured tasks (medical, satellite, and geometric images) changes little. On the smallest dataset, training too long even lowers accuracy. And as the dataset grows, the checkpoint that scores best on the pre-training data moves further from the one that transfers best. At this small scale, then, the amount of data is not the only thing that matters: the kind of downstream task and the checkpoint we keep matter just as much.