Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget

Authors

  • Johannes Lehner ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning Johannes Kepler University, Linz, Austria
  • Benedikt Alkin ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning Johannes Kepler University, Linz, Austria
  • Andreas Fürst ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning Johannes Kepler University, Linz, Austria
  • Elisabeth Rumetshofer ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning Johannes Kepler University, Linz, Austria
  • Lukas Miklautz Faculty of Computer Science, University of Vienna, Vienna, Austria UniVie Doctoral School Computer Science, University of Vienna
  • Sepp Hochreiter ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning Johannes Kepler University, Linz, Austria Institute of Advanced Research in Artificial Intelligence (IARAI)

DOI:

https://doi.org/10.1609/aaai.v38i4.28078

Keywords:

CV: Representation Learning for Vision, CV: Large Vision Models, General

Abstract

Masked Image Modeling (MIM) methods, like Masked Autoencoders (MAE), efficiently learn a rich representation of the input. However, for adapting to downstream tasks, they require a sufficient amount of labeled data since their rich features code not only objects but also less relevant image background. In contrast, Instance Discrimination (ID) methods focus on objects. In this work, we study how to combine the efficiency and scalability of MIM with the ability of ID to perform downstream classification in the absence of large amounts of labeled data. To this end, we introduce Masked Autoencoder Contrastive Tuning (MAE-CT), a sequential approach that utilizes the implicit clustering of the Nearest Neighbor Contrastive Learning (NNCLR) objective to induce abstraction in the topmost layers of a pre-trained MAE. MAE-CT tunes the rich features such that they form semantic clusters of objects without using any labels. Notably, MAE-CT does not rely on hand-crafted augmentations and frequently achieves its best performances while using only minimal augmentations (crop & flip). Further, MAE-CT is compute efficient as it requires at most 10% overhead compared to MAE re-training. Applied to large and huge Vision Transformer (ViT) models, MAE-CT excels over previous self-supervised methods trained on ImageNet in linear probing, k-NN and low-shot classification accuracy as well as in unsupervised clustering accuracy. With ViT-H/16 MAE-CT achieves a new state-of-the-art in linear probing of 82.2%. Project page: github.com/ml-jku/MAE-CT.

Published

2024-03-24

How to Cite

Lehner, J., Alkin, B., Fürst, A., Rumetshofer, E., Miklautz, L., & Hochreiter, S. (2024). Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget. Proceedings of the AAAI Conference on Artificial Intelligence, 38(4), 2965-2973. https://doi.org/10.1609/aaai.v38i4.28078

Issue

Section

AAAI Technical Track on Computer Vision III