Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Dealing with Disagreements: Looking Beyond the Majority Vote in
  Subjective Annotations   Majority voting and averaging are common approaches employed to resolve
annotator disagreements and derive single ground truth labels from multiple
annotations. However, annotators may systematically disagree with one another,
often reflecting their individual biases and values, especially in the case of
subjective tasks such as detecting affect, aggression, and hate speech.
Annotator disagreements may capture important nuances in such tasks that are
often ignored while aggregating annotations to a single ground truth. In order
to address this, we investigate the efficacy of multi-annotator models. In
particular, our multi-task based approach treats predicting each annotators'
judgements as separate subtasks, while sharing a common learned representation
of the task. We show that this approach yields same or better performance than
aggregating labels in the data prior to training across seven different binary
classification tasks. Our approach also provides a way to estimate uncertainty
in predictions, which we demonstrate better correlate with annotation
disagreements than traditional methods. Being able to model uncertainty is
especially useful in deployment scenarios where knowing when not to make a
prediction is important.

21
The categories of this paper are: adap-org
8
The categories of this paper are: hep-ex
10
The categories of this paper are: q-bio
11
The categories of this paper are: hep-lat
34
The categories of this paper are: chem-ph
7
The categories of this paper are: hep-th
22
The categories of this paper are: chao-dyn
0
The categories of this paper are: hep-ph
4
The categories of this paper are: cond-mat
14
The categories of this paper are: math-ph
9
The categories of this paper are: nlin
26
The categories of this paper are: bayes-an
3
The categories of this paper are: physics
12
The categories of this paper are: nucl-th
33
The categories of this paper are: plasm-ph
35
The categories of this paper are: mtrl-th