Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Understanding Dimensional Collapse in Contrastive Self-supervised
  Learning   Self-supervised visual representation learning aims to learn useful
representations without relying on human annotations. Joint embedding approach
bases on maximizing the agreement between embedding vectors from different
views of the same image. Various methods have been proposed to solve the
collapsing problem where all embedding vectors collapse to a trivial constant
solution. Among these methods, contrastive learning prevents collapse via
negative sample pairs. It has been shown that non-contrastive methods suffer
from a lesser collapse problem of a different nature: dimensional collapse,
whereby the embedding vectors end up spanning a lower-dimensional subspace
instead of the entire available embedding space. Here, we show that dimensional
collapse also happens in contrastive learning. In this paper, we shed light on
the dynamics at play in contrastive learning that leads to dimensional
collapse. Inspired by our theory, we propose a novel contrastive learning
method, called DirectCLR, which directly optimizes the representation space
without relying on an explicit trainable projector. Experiments show that
DirectCLR outperforms SimCLR with a trainable linear projector on ImageNet.

17
The categories of this paper are: q-fin
3
The categories of this paper are: physics
19
The categories of this paper are: eess
10
The categories of this paper are: q-bio
4
The categories of this paper are: cond-mat
26
The categories of this paper are: bayes-an
34
The categories of this paper are: chem-ph
28
The categories of this paper are: alg-geom
18
The categories of this paper are: econ
15
The categories of this paper are: nucl-ex
31
The categories of this paper are: ao-sci
16
The categories of this paper are: stat
2
The categories of this paper are: math
21
The categories of this paper are: adap-org
32
The categories of this paper are: atom-ph
5
The categories of this paper are: gr-qc