arxiv:2401.08541

Scalable Pre-training of Large Autoregressive Image Models

Published on Jan 16

· Submitted by

akhaliq on Jan 17

#2 Paper of the day

Upvote

Authors:

Alaaeldin El-Nouby ,

Michal Klein ,

Miguel Angel Bautista ,

Alexander Toshev ,

Vaishaal Shankar ,

Joshua M Susskind ,

Abstract

This paper introduces AIM, a collection of vision models pre-trained with an autoregressive objective. These models are inspired by their textual counterparts, i.e., Large Language Models (LLMs), and exhibit similar scaling properties. Specifically, we highlight two key findings: (1) the performance of the visual features scale with both the model capacity and the quantity of data, (2) the value of the objective function correlates with the performance of the model on downstream tasks. We illustrate the practical implication of these findings by pre-training a 7 billion parameter AIM on 2 billion images, that achieves 84.0% on ImageNet-1k with a frozen trunk. Interestingly, even at this scale, we observe no sign of saturation in performance, suggesting that AIM potentially represents a new frontier for training large-scale vision models. The pre-training of AIM is similar to the pre-training of LLMs, and does not require any image-specific strategy to stabilize the training at scale.

View arXiv page View PDF Add to collection

Community

MichaelBarryUK

Jan 17

Is this an image classification model?

bone

Jan 17

Never liked the super specific self supervised training for images with so many hacks and complexities only working under very specific conditions. Contrastive learning is the worst.

dondosss

Jan 18

•

edited Jan 18

Is this an image classification model?

Like GPT, it is a next token prediction model that operates in the pixel domain. It is trained to predict the next patch based on a sequence of prefix patches. Similar to the self-supervised ViT models like DINO, you can use intermediate representations of the trained model for linear probing or kNN classification, as shown in Figure 10, or you can come up with an original few-shot prompting/finetuning technique, as in the language modeling GPT.

MichaelBarryUK

Jan 19

Is this an image classification model?

Like GPT, it is a next token prediction model that operates in the pixel domain. It is trained to predict the next patch based on a sequence of prefix patches. Similar to the self-supervised ViT models like DINO, you can use intermediate representations of the trained model for linear probing or kNN classification, as shown in Figure 10, or you can come up with an original few-shot prompting/finetuning technique, as in the language modeling GPT.

Interesting, thank you