A comparison of primary and secondary relevance judgements for real-life topics

Simon Wakeling, Martin Halvey, Robert Villa, Laura Hasler

Research output: Book chapter/Published conference paperConference paperpeer-review

7 Citations (Scopus)

Abstract

The notion of relevance is fundamental to the field of Information Retrieval. Within the field a generally accepted conception of relevance as inherently subjective has emerged, with an individual's assessment of relevance influenced by numerous contextual factors. In this paper we present a user study that examines in detail the differences between primary and secondary assessors on a set of "real-world" topics which were gathered specifically for the work. By gathering topics which are representative of the staff and students at a major university, at a particular point in time, we aim to explore differences between primary and secondary relevance judgements for real-life search tasks. Findings suggest that while secondary assessors may find the assessment task challenging in various ways (they generally possess less interest and knowledge in secondary topics and take longer to assess documents), agreement between primary and secondary assessors is high.
Original languageEnglish
Title of host publicationCHIIR 2016
Subtitle of host publicationProceedings of the 2016 ACM on conference on human information interaction and retrieval
PublisherAssociation for Computing Machinery, Inc (ACM)
Pages173-182
Number of pages10
ISBN (Print)9781450337519
DOIs
Publication statusPublished - 13 Mar 2016
Event2016 ACM SIGIR Conference on Human Information Interaction and Retrieval - University of North Carolina, Chapel Hill, United States
Duration: 13 Mar 201617 Mar 2016
http://sigir.org/chiir2016/

Conference

Conference2016 ACM SIGIR Conference on Human Information Interaction and Retrieval
Country/TerritoryUnited States
CityChapel Hill
Period13/03/1617/03/16
Internet address

Fingerprint

Dive into the research topics of 'A comparison of primary and secondary relevance judgements for real-life topics'. Together they form a unique fingerprint.

Cite this
https://researchoutput.csu.edu.au/en/publications/a-comparison-of-primary-and-secondary-relevance-judgements-for-re
researchoutput.csu.edu.au
CC-MAIN-2024-30
59792
application/http; msgtype=response
sha1:HMTIOAIKK7YK3OOUCFZE5MHXVWGLWZFU
<urn:uuid:802ef2b6-be27-4af2-9e40-8b09401478c3>
2024-07-12T13:54:16Z
172.64.148.16
text/html
sha1:UJ6CXZON4OKZOV6QXZXBDBQUZ7VBEYYJ
<urn:uuid:57d0f2d5-85e7-4db7-943e-ed9090a8bea7>
https://researchoutput.csu.edu.au/en/publications/a-comparison-of-primary-and-secondary-relevance-judgements-for-re
response
<urn:uuid:0557bc3f-0dfc-477a-957a-c7c3c525ef61>
isPartOf: CC-MAIN-2024-30
publisher: Common Crawl
description: Wide crawl of the web for July 2024
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-110
software: Apache Nutch 1.20 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.5-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/