Model Details

Domain:

Vision

Task:

Image classification

Model Access:

Open weights (unrestricted)

Citations:

7535

Introduction

In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) that stand out compared to convolutional networks (convnets). Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets. Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. Our study also underlines the importance of momentum encoder, multi-crop training, and the use of small patches with ViTs. We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels. We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base.

Benchmarking

FLOPs

210000000000000000000

Notes: "Overall, training DINO with Vision Transformers achieves 76.1 top-1 accuracy using two 8-GPU servers for 3 days" GPU is V100 16 * 125 teraflops * 3 days * 0.4 utilization = 2.1e20 However, this isn't the best result in the paper (which is 80.1% with ViT-B/8). 76.1% is the result from ViT-B/16 per Table 2, which may be 5x cheaper than ViT-B/8 based on Table 1? upd: "Table 8: Time and memory requirements. We show total running time and peak memory per GPU (“mem.”) when running ViT-S/16 DINO models on two 8-GPU machines." 2*8*125 teraflops*72.6h*3600*0.4=2.09088e+20