  The notion of experiment precision quantifies the variance of user ratings in
a subjective experiment. Although there exist measures that assess subjective
experiment precision, there are no systematic analyses of these measures
available in the literature. To the best of our knowledge, there is also no
systematic framework in the Multimedia Quality Assessment field for comparing
subjective experiments in terms of their precision. Therefore, the main idea of
this paper is to propose a framework for comparing subjective experiments in
the field of MQA based on appropriate experiment precision measures. We present
three experiment precision measures and three related experiment precision
comparison methods. We systematically analyse the performance of the measures
and methods proposed. We do so both through a simulation study (varying user
rating variance and bias) and by using data from four real-world Quality of
Experience (QoE) subjective experiments. In the simulation study we focus on
crowdsourcing QoE experiments, since they are known to generate ratings with
higher variance and bias, when compared to traditional subjective experiment
methodologies. We conclude that our proposed measures and related comparison
methods properly capture experiment precision (both when tested on simulated
and real-world data). One of the measures also proves capable of dealing with
even significantly biased responses. We believe our experiment precision
assessment framework will help compare different subjective experiment
methodologies. For example, it may help decide which methodology results in
more precise user ratings. This may potentially inform future standardisation
activities.
