Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Pretrained Transformers Improve Out-of-Distribution Robustness   Although pretrained Transformers such as BERT achieve high accuracy on
in-distribution examples, do they generalize to new distributions? We
systematically measure out-of-distribution (OOD) generalization for seven NLP
datasets by constructing a new robustness benchmark with realistic distribution
shifts. We measure the generalization of previous models including bag-of-words
models, ConvNets, and LSTMs, and we show that pretrained Transformers'
performance declines are substantially smaller. Pretrained transformers are
also more effective at detecting anomalous or OOD examples, while many previous
models are frequently worse than chance. We examine which factors affect
robustness, finding that larger models are not necessarily more robust,
distillation can be harmful, and more diverse pretraining data can enhance
robustness. Finally, we show where future work can improve OOD robustness.

26
The categories of this paper are: bayes-an
36
The categories of this paper are: cmp-lg
34
The categories of this paper are: chem-ph
10
The categories of this paper are: q-bio
7
The categories of this paper are: hep-th
3
The categories of this paper are: physics
24
The categories of this paper are: dg-ga
29
The categories of this paper are: funct-an
20
The categories of this paper are: acc-phys
13
The categories of this paper are: quant-ph
33
The categories of this paper are: plasm-ph
14
The categories of this paper are: math-ph
18
The categories of this paper are: econ
32
The categories of this paper are: atom-ph
15
The categories of this paper are: nucl-ex
25
The categories of this paper are: patt-sol