Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: The Inductive Bias of In-Context Learning: Rethinking Pretraining
  Example Design   Pretraining Neural Language Models (NLMs) over a large corpus involves
chunking the text into training examples, which are contiguous text segments of
sizes processable by the neural architecture. We highlight a bias introduced by
this common practice: we prove that the pretrained NLM can model much stronger
dependencies between text segments that appeared in the same training example,
than it can between text segments that appeared in different training examples.
This intuitive result has a twofold role. First, it formalizes the motivation
behind a broad line of recent successful NLM training heuristics, proposed for
the pretraining and fine-tuning stages, which do not necessarily appear related
at first glance. Second, our result clearly indicates further improvements to
be made in NLM pretraining for the benefit of Natural Language Understanding
tasks. As an example, we propose "kNN-Pretraining": we show that including
semantically related non-neighboring sentences in the same pretraining example
yields improved sentence representations and open domain question answering
abilities. This theoretically motivated degree of freedom for pretraining
example design indicates new training schemes for self-improving
representations.

28
The categories of this paper are: alg-geom
12
The categories of this paper are: nucl-th
10
The categories of this paper are: q-bio
36
The categories of this paper are: cmp-lg
7
The categories of this paper are: hep-th
13
The categories of this paper are: quant-ph
16
The categories of this paper are: stat
2
The categories of this paper are: math
25
The categories of this paper are: patt-sol
9
The categories of this paper are: nlin
11
The categories of this paper are: hep-lat
34
The categories of this paper are: chem-ph
21
The categories of this paper are: adap-org
22
The categories of this paper are: chao-dyn
14
The categories of this paper are: math-ph
35
The categories of this paper are: mtrl-th