  Hooge et al. asked the question: "Is human classification by experienced
untrained observers a gold standard in fixation detection?" They conclude the
answer is no. If they had entitled their paper: "Is human classification by
experienced untrained observers a gold standard in fixation detection when data
quality is very poor, data are error-filled, data presentation was not optimal,
and the analysis was seriously flawed?", I would have no case to make. In the
present report, I will present evidence to support my view that this latter
title is justified. The low quality data assessment is based on using a
relatively imprecise eye-tracker, the absence of head restraint for any
subjects, and the use of infants as the majority of subjects (60 of 70
subjects). Allowing subjects with more than 50% missing data (as much as 95%)
is also evidence of low quality data. The error-filled assessment is based on
evidence that a number of the "fixations" classified by "experts" have obvious
saccades within them, and that, apparently, a number of fixations were
classified on the basis of no signal at all. The evidence for non-optimal data
presentation stems from the fact that, in a number of cases, perfectly good
data was not presented to the coders. The flaws in the analysis are evidenced
by the fact that entire stretches of missing data were considered classified,
and that the measurement of saccade amplitude was based on many cases in which
there was no saccade at all. Without general evidence to the contrary, it is
correct to assume that some human classifiers under some conditions may meet
the criteria for a gold standard, and classifiers under other conditions may
not. This conditionality is not recognized by Hooge et al. A fair assessment
would conclude that whether or not humans can be considered a gold standard is
still very much an open question.
