The CNN Era
For most of the deep learning era, convolutional neural networks (CNNs) were the unchallenged architecture for vision tasks. AlexNet (2012) launched the deep learning era. VGG, ResNet, EfficientNet, and ConvNeXt progressively improved performance and efficiency. CNNs have inductive biases well-suited to vision: translation equivariance (a cat is a cat regardless of where it appears in the image) and local connectivity (nearby pixels are more related than distant ones).
By 2020, CNNs had achieved remarkable results across object classification, detection, and segmentation. The question wasn't whether to use CNNs for vision — it was which CNN.
ViT: Applying Transformer to Images
Vision Transformer (ViT), from Google Brain in 2020, applied the transformer architecture to images with minimal changes. The approach: divide the image into 16×16 pixel patches, embed each patch as a single token, add position embeddings, and run a standard transformer encoder. The resulting "sequence of image patches" is processed identically to a sequence of words in NLP.
The paper's key finding: ViT matches or exceeds CNNs at scale, but requires much more data and compute to train. On ImageNet (1.2M images), a ViT trained from scratch underperforms ResNet. But pre-trained on JFT-300M (300M images), ViT significantly outperforms ResNet.
This data-dependence was initially seen as a limitation. It turned out to be a preview of a more general principle: transformers improve more rapidly with scale than CNNs, eventually overtaking them as scale increases.
Why ViT Wins at Scale
CNNs have strong inductive biases (local connectivity, translation equivariance) that help with limited data. Transformers have weaker inductive biases but can learn any relationship between arbitrary positions — including non-local ones that CNNs find difficult. With enough data, transformers learn the inductive biases that CNNs have built in, plus additional ones CNNs can't express.
Self-attention also scales favorably: larger ViT models show consistent improvement on LSVRC ImageNet-1K without the performance saturation that affects large CNNs. The quadratic attention cost is less of a problem for images than text (images are shorter sequences when patched) and can be addressed with efficient attention variants.
DINO and Self-Supervised ViT
DINO (Caron et al., 2021) from Meta demonstrated that self-supervised training — without labels — could produce ViT representations with striking properties: clear semantic segmentation emerging without any segmentation supervision, nearest-neighbor retrieval matching supervised methods, and probing for object boundaries. DINO's success showed that ViT's inductive biases, when combined with sufficient self-supervised training data, produce feature spaces with qualitatively different properties from CNNs.
DINOv2 (2023) scaled this up with curated training data, producing backbone features competitive with supervised training on nearly all downstream tasks. DINOv2 features are now a standard starting point for many computer vision applications.
Hybrid Architectures
The CNN vs. ViT debate has largely been resolved in practice: hybrid architectures combining convolutional and attention layers often outperform pure versions of either. ConvNeXt (2022) showed that carefully modernized CNNs match ViTs; MaxViT, EfficientViT, and Swin Transformer use hierarchical designs that combine local convolutional processing with global attention. The practical choice depends on the task, data scale, and deployment constraints rather than a blanket architecture preference.