  Building a benchmark dataset for hate speech detection presents various
challenges. Firstly, because hate speech is relatively rare, random sampling of
tweets to annotate is very inefficient in finding hate speech. To address this,
prior datasets often include only tweets matching known "hate words". However,
restricting data to a pre-defined vocabulary may exclude portions of the
real-world phenomenon we seek to model. A second challenge is that definitions
of hate speech tend to be highly varying and subjective. Annotators having
diverse prior notions of hate speech may not only disagree with one another but
also struggle to conform to specified labeling guidelines. Our key insight is
that the rarity and subjectivity of hate speech are akin to that of relevance
in information retrieval (IR). This connection suggests that well-established
methodologies for creating IR test collections can be usefully applied to
create better benchmark datasets for hate speech. To intelligently and
efficiently select which tweets to annotate, we apply standard IR techniques of
{\em pooling} and {\em active learning}. To improve both consistency and value
of annotations, we apply {\em task decomposition} and {\em annotator rationale}
techniques. We share a new benchmark dataset for hate speech detection on
Twitter that provides broader coverage of hate than prior datasets. We also
show a dramatic drop in accuracy of existing detection models when tested on
these broader forms of hate. Annotator rationales we collect not only justify
labeling decisions but also enable future work opportunities for
dual-supervision and/or explanation generation in modeling. Further details of
our approach can be found in the supplementary materials.
