  The global surge in AI applications is transforming industries, leading to
displacement and complementation of existing jobs, while also giving rise to
new employment opportunities. Human oversight of AI is an emerging task in
which human workers interact with an AI model to improve its performance,
safety, and compliance with normative principles. Data annotation, encompassing
the labelling of images or annotating of texts, serves as a critical human
oversight process, as the quality of a dataset directly influences the quality
of AI models trained on it. Therefore, the efficiency of human oversight work
stands as an important competitive advantage for AI developers. This paper
delves into the foundational economics of human oversight, with a specific
focus on the impact of norm design and monetary incentives on data quality and
costs. An experimental study involving 307 data annotators examines six groups
with varying task instructions (norms) and monetary incentives. Results reveal
that annotators provided with clear rules exhibit higher accuracy rates,
outperforming those with vague standards by 14%. Similarly, annotators
receiving an additional monetary incentive perform significantly better, with
the highest accuracy rate recorded in the group working with both clear rules
and incentives (87.5% accuracy). However, both groups require more time to
complete tasks, with a 31% increase in average task completion time compared to
those working with standards and no incentives. These empirical findings
underscore the trade-off between data quality and efficiency in data curation,
shedding light on the nuanced impact of norm design and incentives on the
economics of AI development. The paper contributes experimental insights to
discussions on the economical, ethical, and legal considerations of AI
technologies.
