Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Assessing the Benchmarking Capacity of Machine Reading Comprehension
  Datasets   Existing analysis work in machine reading comprehension (MRC) is largely
concerned with evaluating the capabilities of systems. However, the
capabilities of datasets are not assessed for benchmarking language
understanding precisely. We propose a semi-automated, ablation-based
methodology for this challenge; By checking whether questions can be solved
even after removing features associated with a skill requisite for language
understanding, we evaluate to what degree the questions do not require the
skill. Experiments on 10 datasets (e.g., CoQA, SQuAD v2.0, and RACE) with a
strong baseline model show that, for example, the relative scores of a baseline
model provided with content words only and with shuffled sentence words in the
context are on average 89.2% and 78.5% of the original score, respectively.
These results suggest that most of the questions already answered correctly by
the model do not necessarily require grammatical and complex reasoning. For
precise benchmarking, MRC datasets will need to take extra care in their design
to ensure that questions can correctly evaluate the intended skills.

19
The categories of this paper are: eess
25
The categories of this paper are: patt-sol
22
The categories of this paper are: chao-dyn
3
The categories of this paper are: physics
24
The categories of this paper are: dg-ga
34
The categories of this paper are: chem-ph
0
The categories of this paper are: hep-ph
5
The categories of this paper are: gr-qc
35
The categories of this paper are: mtrl-th
10
The categories of this paper are: q-bio
13
The categories of this paper are: quant-ph
31
The categories of this paper are: ao-sci
21
The categories of this paper are: adap-org
14
The categories of this paper are: math-ph
30
The categories of this paper are: q-alg
9
The categories of this paper are: nlin