Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in
  Natural Language Inference   Quantitative reasoning is a higher-order reasoning skill that any intelligent
natural language understanding system can reasonably be expected to handle. We
present EQUATE (Evaluating Quantitative Understanding Aptitude in Textual
Entailment), a new framework for quantitative reasoning in textual entailment.
We benchmark the performance of 9 published NLI models on EQUATE, and find that
on average, state-of-the-art methods do not achieve an absolute improvement
over a majority-class baseline, suggesting that they do not implicitly learn to
reason with quantities. We establish a new baseline Q-REAS that manipulates
quantities symbolically. In comparison to the best performing NLI model, it
achieves success on numerical reasoning tests (+24.2%), but has limited verbal
reasoning capabilities (-8.1%). We hope our evaluation framework will support
the development of models of quantitative reasoning in language understanding.

19
The categories of this paper are: eess
33
The categories of this paper are: plasm-ph
16
The categories of this paper are: stat
10
The categories of this paper are: q-bio
20
The categories of this paper are: acc-phys
24
The categories of this paper are: dg-ga
32
The categories of this paper are: atom-ph
12
The categories of this paper are: nucl-th
2
The categories of this paper are: math
8
The categories of this paper are: hep-ex
35
The categories of this paper are: mtrl-th
30
The categories of this paper are: q-alg
6
The categories of this paper are: astro-ph
37
The categories of this paper are: supr-con
11
The categories of this paper are: hep-lat
28
The categories of this paper are: alg-geom