  We present and analyze results from a pilot study that explores how
crowdsourcing can be used in the process of generating distractors (incorrect
answer choices) in multiple-choice concept inventories (conceptual tests of
understanding). To our knowledge, we are the first to propose and study this
approach. Using Amazon Mechanical Turk, we collected approximately 180
open-ended responses to several question stems from the Cybersecurity Concept
Inventory of the Cybersecurity Assessment Tools Project and from the Digital
Logic Concept Inventory. We generated preliminary distractors by filtering
responses, grouping similar responses, selecting the four most frequent groups,
and refining a representative distractor for each of these groups. We analyzed
our data in two ways. First, we compared the responses and resulting
distractors with those from the aforementioned inventories. Second, we obtained
feedback from Amazon Mechanical Turk on the resulting new draft test items
(including distractors) from additional subjects. Challenges in using
crowdsourcing include controlling the selection of subjects and filtering out
responses that do not reflect genuine effort. Despite these challenges, our
results suggest that crowdsourcing can be a very useful tool in generating
effective distractors (attractive to subjects who do not understand the
targeted concept). Our results also suggest that this method is faster, easier,
and cheaper than is the traditional method of having one or more experts draft
distractors, and building on talk-aloud interviews with subjects to uncover
their misconceptions. Our results are significant because generating effective
distractors is one of the most difficult steps in creating multiple-choice
assessments.
