Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Toward a Robust Crowd-labeling Framework using Expert Evaluation and
  Pairwise Comparison   Crowd-labeling emerged from the need to label large-scale and complex data, a
tedious, expensive, and time-consuming task. One of the main challenges in the
crowd-labeling task is to control for or determine in advance the proportion of
low-quality/malicious labelers. If that proportion grows too high, there is
often a phase transition leading to a steep, non-linear drop in labeling
accuracy as noted by Karger et al. [2014]. To address these challenges, we
propose a new framework called Expert Label Injected Crowd Estimation (ELICE)
and extend it to different versions and variants that delay phase transition
leading to a better labeling accuracy. ELICE automatically combines and boosts
bulk crowd labels supported by labels from experts for limited number of
instances from the dataset. The expert-labels help to estimate the individual
ability of crowd labelers and difficulty of each instance, both of which are
used to aggregate the labels. Empirical evaluation shows the superiority of
ELICE as compared to other state-of-the-art methods. We also derive a lower
bound on the number of expert-labeled instances needed to estimate the crowd
ability and dataset difficulty as well as to get better quality labels.

4
The categories of this paper are: cond-mat
5
The categories of this paper are: gr-qc
10
The categories of this paper are: q-bio
16
The categories of this paper are: stat
33
The categories of this paper are: plasm-ph
15
The categories of this paper are: nucl-ex
29
The categories of this paper are: funct-an
31
The categories of this paper are: ao-sci
18
The categories of this paper are: econ
19
The categories of this paper are: eess
7
The categories of this paper are: hep-th
23
The categories of this paper are: solv-int
20
The categories of this paper are: acc-phys
25
The categories of this paper are: patt-sol
17
The categories of this paper are: q-fin
12
The categories of this paper are: nucl-th