  Objective: To investigate whether performance (number of correct decisions)
of humans supported by a computer alerting tool can be improved by tailoring
the tool's alerting threshold (sensitivity/specificity combination) according
to user ability and task difficulty. Background: Many researchers implicitly
assume that for each tool there exists a single ideal threshold. But research
shows the effects of alerting tools on decision errors to vary depending on
variables such as user ability and task difficulty. These findings motivated
our investigation. Method: Forty-seven participants edited text passages, aided
by a simulated spell-checker tool. We experimentally manipulated passage
difficulty and tool alerting threshold, measured participants' editing and
dictation ability, and counted participants' decision errors (false positives +
false negatives). We investigated whether alerting threshold, user ability,
task difficulty and their interactions affected error count. Results: Which
alerting threshold better helped a user depended on an interaction between user
ability and passage difficulty. Some effects were large: for higher ability
users, a more sensitive threshold reduced errors by 30%, on the easier
passages. Participants were not significantly more likely to prefer the
alerting threshold with which they performed better. Conclusion: Adjusting
alerting thresholds for individual users' ability and task difficulty could
substantially improve effects of automated alerts on user decisions. This
potential deserves further investigation. Improvement size and rules for
adjustment will be application-specific. Application: Computer alerting tools
have critical roles in many domains. Designers should assess potential benefits
of adjustable alerting thresholds for their specific CAT application. Guidance
for choosing thresholds will be essential for realizing these benefits in
practice.
