Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Selective Question Answering under Domain Shift   To avoid giving wrong answers, question answering (QA) models need to know
when to abstain from answering. Moreover, users often ask questions that
diverge from the model's training data, making errors more likely and thus
abstention more critical. In this work, we propose the setting of selective
question answering under domain shift, in which a QA model is tested on a
mixture of in-domain and out-of-domain data, and must answer (i.e., not abstain
on) as many questions as possible while maintaining high accuracy. Abstention
policies based solely on the model's softmax probabilities fare poorly, since
models are overconfident on out-of-domain inputs. Instead, we train a
calibrator to identify inputs on which the QA model errs, and abstain when it
predicts an error is likely. Crucially, the calibrator benefits from observing
the model's behavior on out-of-domain data, even if from a different domain
than the test data. We combine this method with a SQuAD-trained QA model and
evaluate on mixtures of SQuAD and five other QA datasets. Our method answers
56% of questions while maintaining 80% accuracy; in contrast, directly using
the model's probabilities only answers 48% at 80% accuracy.

31
The categories of this paper are: ao-sci
2
The categories of this paper are: math
4
The categories of this paper are: cond-mat
29
The categories of this paper are: funct-an
33
The categories of this paper are: plasm-ph
0
The categories of this paper are: hep-ph
25
The categories of this paper are: patt-sol
30
The categories of this paper are: q-alg
27
The categories of this paper are: comp-gas
35
The categories of this paper are: mtrl-th
15
The categories of this paper are: nucl-ex
9
The categories of this paper are: nlin
8
The categories of this paper are: hep-ex
23
The categories of this paper are: solv-int
32
The categories of this paper are: atom-ph
24
The categories of this paper are: dg-ga