Reducing Data for Vision Foundation Models
Data-Efficiency of Self-Supervised Learning with DINO Multi-Crop
L. Margulis (TU Delft - Electrical Engineering, Mathematics and Computer Science)
J.C. van Gemert – Graduation committee member (TU Delft - Electrical Engineering, Mathematics and Computer Science)
M.J.G. Olsthoorn – Graduation committee member (TU Delft - Electrical Engineering, Mathematics and Computer Science)
A.D. Manolache – Mentor (TU Delft - Electrical Engineering, Mathematics and Computer Science)
P.J.W. Reijalt – Mentor (TU Delft - Electrical Engineering, Mathematics and Computer Science)
More Info
expand_more
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Abstract
Self-supervised learning (SSL) lets computer vision models learn from unlabelled image datasets. Most DINO benchmarks pretrain on ImageNet — a million-image dataset that takes days of multi-GPU training per run, out of reach for the rapid iteration cycles smaller research groups rely on. This leaves practitioners with smaller datasets unsure whether DINO is worth running, or which of its design choices still hold at this scale.
We pretrain a small Vision Transformer (ViT-Tiny/8) using DINO on Tiny-ImageNet subsets from 1K to 100K images at 64x64 resolution, evaluated on downstream classification tasks. Downstream accuracy grows steadily with pretraining-set size and approaches the accuracy of a fully supervised baseline at the largest scale.
Our main contribution is a multi-crop ablation across data scale, training duration, and downstream task category. We find that multi-crop's benefit at sub-ImageNet scale is delayed rather than absent, and that the optimal multi-crop count depends on the downstream task category — no single setting wins across all tasks.
These findings show that the canonical DINO recipe does not transfer cleanly to sub-ImageNet scale. We recommend choosing the multi-crop count based on training budget and downstream task type, rather than copying the ImageNet default.