  Human annotations are vital to supervised learning, yet annotators often
disagree on the correct label, especially as annotation tasks increase in
complexity. A strategy to improve label quality is to ask multiple annotators
to label the same item and aggregate their labels. Many aggregation models have
been proposed for categorical or numerical annotation tasks, but far less work
has considered more complex annotation tasks involving open-ended,
multivariate, or structured responses. While a variety of bespoke models have
been proposed for specific tasks, our work is the first to introduce
aggregation methods that generalize across many diverse complex tasks,
including sequence labeling, translation, syntactic parsing, ranking, bounding
boxes, and keypoints. This generality is achieved by devising a task-agnostic
method to model distances between labels rather than the labels themselves.
  This article extends our prior work with investigation of three new research
questions. First, how do complex annotation properties impact aggregation
accuracy? Second, how should a task owner navigate the many modeling choices to
maximize aggregation accuracy? Finally, what diagnoses can verify that
aggregation models are specified correctly for the given data? To understand
how various factors impact accuracy and to inform model selection, we conduct
simulation studies and experiments on real, complex datasets. Regarding
testing, we introduce unit tests for aggregation models and present a suite of
such tests to ensure that a given model is not mis-specified and exhibits
expected behavior.
  Beyond investigating these research questions above, we discuss the
foundational concept of annotation complexity, present a new aggregation model
as a bridge between traditional models and our own, and contribute a new
semi-supervised learning method for complex label aggregation that outperforms
prior work.
