Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Structure Inducing Pre-Training   We present a theoretical analysis from first principles that establishes a
novel connection between relational inductive bias of pre-training and
fine-tuning performance while providing an extended view on general
pre-training models. We further explore how existing pre-training methods
impose relational inductive biases, finding that the vast majority of existing
approaches focus almost exclusively on modelling relationships in an
intra-sample manner, rather than a per-sample manner. We build upon these
findings with simulations and empirical studies on standard benchmarks spanning
3 data modalities and 10 downstream tasks. These investigations validate our
theoretical analyses, and provides a recipe to produce new pre-training methods
which incorporate provably richer inductive biases than do existing methods in
line with user specified relational graphs.

26
The categories of this paper are: bayes-an
12
The categories of this paper are: nucl-th
29
The categories of this paper are: funct-an
30
The categories of this paper are: q-alg
14
The categories of this paper are: math-ph
0
The categories of this paper are: hep-ph
15
The categories of this paper are: nucl-ex
16
The categories of this paper are: stat
36
The categories of this paper are: cmp-lg
3
The categories of this paper are: physics
8
The categories of this paper are: hep-ex
22
The categories of this paper are: chao-dyn
35
The categories of this paper are: mtrl-th
20
The categories of this paper are: acc-phys
19
The categories of this paper are: eess
37
The categories of this paper are: supr-con