  Large archival collections, such as email or government documents, must be
manually reviewed to identify any sensitive information before the collection
can be released publicly. Sensitivity classification has received a lot of
attention in the literature. However, more recently, there has been increasing
interest in developing sensitivity-aware search engines that can provide users
with relevant search results, while ensuring that no sensitive documents are
returned to the user. Sensitivity-aware search would mitigate the need for a
manual sensitivity review prior to collections being made available publicly.
To develop such systems, there is a need for test collections that contain
relevance assessments for a set of information needs as well as ground-truth
labels for a variety of sensitivity categories. The well-known Enron email
collection contains a classification ground-truth that can be used to represent
sensitive information, e.g., the Purely Personal and Personal but in
Professional Context categories can be used to represent sensitive personal
information. However, the existing Enron collection does not contain a set of
information needs and relevance assessments. In this work, we present a
collection of fifty information needs (topics) with crowdsourced query
formulations (3 per topic) and relevance assessments (11,471 in total) for the
Enron collection (mean number of relevant documents per topic = 11, variance =
34.7). The developed information needs, queries and relevance judgements are
available on GitHub and will be available along with the existing Enron
collection through the popular ir_datasets library. Our proposed collection
results in the first freely available test collection for developing
sensitivity-aware search systems.
