Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Significant Improvements over the State of the Art? A Case Study of the
  MS MARCO Document Ranking Leaderboard   Leaderboards are a ubiquitous part of modern research in applied machine
learning. By design, they sort entries into some linear order, where the
top-scoring entry is recognized as the "state of the art" (SOTA). Due to the
rapid progress being made in information retrieval today, particularly with
neural models, the top entry in a leaderboard is replaced with some regularity.
These are touted as improvements in the state of the art. Such pronouncements,
however, are almost never qualified with significance testing. In the context
of the MS MARCO document ranking leaderboard, we pose a specific question: How
do we know if a run is significantly better than the current SOTA? We ask this
question against the backdrop of recent IR debates on scale types: in
particular, whether commonly used significance tests are even mathematically
permissible. Recognizing these potential pitfalls in evaluation methodology,
our study proposes an evaluation framework that explicitly treats certain
outcomes as distinct and avoids aggregating them into a single-point metric.
Empirical analysis of SOTA runs from the MS MARCO document ranking leaderboard
reveals insights about how one run can be "significantly better" than another
that are obscured by the current official evaluation metric (MRR@100).

18
The categories of this paper are: econ
26
The categories of this paper are: bayes-an
10
The categories of this paper are: q-bio
23
The categories of this paper are: solv-int
32
The categories of this paper are: atom-ph
8
The categories of this paper are: hep-ex
4
The categories of this paper are: cond-mat
34
The categories of this paper are: chem-ph
21
The categories of this paper are: adap-org
36
The categories of this paper are: cmp-lg
17
The categories of this paper are: q-fin
15
The categories of this paper are: nucl-ex
20
The categories of this paper are: acc-phys
0
The categories of this paper are: hep-ph
5
The categories of this paper are: gr-qc
29
The categories of this paper are: funct-an