Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: The Effect of Natural Distribution Shift on Question Answering Models   We build four new test sets for the Stanford Question Answering Dataset
(SQuAD) and evaluate the ability of question-answering systems to generalize to
new data. Our first test set is from the original Wikipedia domain and measures
the extent to which existing systems overfit the original test set. Despite
several years of heavy test set re-use, we find no evidence of adaptive
overfitting. The remaining three test sets are constructed from New York Times
articles, Reddit posts, and Amazon product reviews and measure robustness to
natural distribution shifts. Across a broad range of models, we observe average
performance drops of 3.8, 14.0, and 17.4 F1 points, respectively. In contrast,
a strong human baseline matches or exceeds the performance of SQuAD models on
the original domain and exhibits little to no drop in new domains. Taken
together, our results confirm the surprising resilience of the holdout method
and emphasize the need to move towards evaluation metrics that incorporate
robustness to natural distribution shifts.

27
The categories of this paper are: comp-gas
36
The categories of this paper are: cmp-lg
20
The categories of this paper are: acc-phys
23
The categories of this paper are: solv-int
32
The categories of this paper are: atom-ph
31
The categories of this paper are: ao-sci
14
The categories of this paper are: math-ph
30
The categories of this paper are: q-alg
18
The categories of this paper are: econ
5
The categories of this paper are: gr-qc
25
The categories of this paper are: patt-sol
12
The categories of this paper are: nucl-th
9
The categories of this paper are: nlin
22
The categories of this paper are: chao-dyn
13
The categories of this paper are: quant-ph
10
The categories of this paper are: q-bio