Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?   Despite the recent success of video self-supervised learning, there is much
still to be understood about their generalization capability. In this paper, we
investigate how sensitive video self-supervised learning is to the currently
used benchmark convention and whether methods generalize beyond the canonical
evaluation setting. We do this across four different factors of sensitivity:
domain, samples, actions and task. Our comprehensive set of over 500
experiments, which encompasses 7 video datasets, 9 self-supervised methods and
6 video understanding tasks, reveals that current benchmarks in video
self-supervised learning are not a good indicator of generalization along these
sensitivity factors. Further, we find that self-supervised methods considerably
lag behind vanilla supervised pre-training, especially when domain shift is
large and the amount of available downstream samples are low. From our analysis
we distill the SEVERE-benchmark, a subset of our experiments, and discuss its
implication for evaluating the generalizability of representations obtained by
existing and future self-supervised video learning methods.

28
The categories of this paper are: alg-geom
27
The categories of this paper are: comp-gas
10
The categories of this paper are: q-bio
20
The categories of this paper are: acc-phys
18
The categories of this paper are: econ
33
The categories of this paper are: plasm-ph
23
The categories of this paper are: solv-int
36
The categories of this paper are: cmp-lg
0
The categories of this paper are: hep-ph
11
The categories of this paper are: hep-lat
7
The categories of this paper are: hep-th
13
The categories of this paper are: quant-ph
14
The categories of this paper are: math-ph
16
The categories of this paper are: stat
25
The categories of this paper are: patt-sol
19
The categories of this paper are: eess