Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Cross-replication Reliability -- An Empirical Approach to Interpreting
  Inter-rater Reliability   We present a new approach to interpreting IRR that is empirical and
contextualized. It is based upon benchmarking IRR against baseline measures in
a replication, one of which is a novel cross-replication reliability (xRR)
measure based on Cohen's kappa. We call this approach the xRR framework. We
opensource a replication dataset of 4 million human judgements of facial
expressions and analyze it with the proposed framework. We argue this framework
can be used to measure the quality of crowdsourced datasets.

31
The categories of this paper are: ao-sci
34
The categories of this paper are: chem-ph
26
The categories of this paper are: bayes-an
37
The categories of this paper are: supr-con
35
The categories of this paper are: mtrl-th
15
The categories of this paper are: nucl-ex
14
The categories of this paper are: math-ph
27
The categories of this paper are: comp-gas
29
The categories of this paper are: funct-an
4
The categories of this paper are: cond-mat
21
The categories of this paper are: adap-org
11
The categories of this paper are: hep-lat
12
The categories of this paper are: nucl-th
30
The categories of this paper are: q-alg
36
The categories of this paper are: cmp-lg
7
The categories of this paper are: hep-th