Carnegie Mellon University
Browse
file.pdf (82.17 kB)
Download file

Revisiting the Effect of Topic Set Size on Retrieval Error

Download (82.17 kB)
journal contribution
posted on 1977-01-01, 00:00 authored by Wei-Hao Lin, Alexander Hauptmann
Evaluating retrieval systems in a controlled environment with a large set of topics has been the core paradigm in the information retrieval community. Voorhees and Buckley proposed to estimate the reliability of retrieval experiments by calculating the probability of making wrong effectiveness judgments between two retrieval systems over two retrieval experiments[2], which is called Retrieval Experiment Error Rate (REER) in this paper. They have successfully shown how the topic set sizes affect the retrieval experiment reliability. However, the REER model in the previous work was empirically justified without providing a derivation based on statistical principles. We fill this gap and show that REER can indeed be derived from statistical principles. Based on the derived model we can explain why a successful experiment design depends on factors including a sufficient number of topics, large enough measurement score difference between systems, and a homogeneous distribution of retrieval scores for topics and systems, which reduces the variance of the score differences.

History

Publisher Statement

All Rights Reserved

Date

1977-01-01

Usage metrics

    Exports
https://kilthub.cmu.edu/articles/journal_contribution/Revisiting_the_Effect_of_Topic_Set_Size_on_Retrieval_Error/6609104/1
kilthub.cmu.edu
CC-MAIN-2023-23
142332
application/http; msgtype=response
sha1:UNK5TAAU7A3J4JZMCVBP2POZNKBEL2QV
<urn:uuid:cda75274-8e49-4c25-b4ef-22126edcbb01>
2023-05-31T11:26:04Z
52.18.151.32
text/html
sha1:VVF2SXLMI7WJPJF5J4G3TX5JHHGORR3S
<urn:uuid:2e3ada89-bc58-46fd-9f8f-e59d9ab7cb1e>
https://kilthub.cmu.edu/articles/journal_contribution/Revisiting_the_Effect_of_Topic_Set_Size_on_Retrieval_Error/6609104/1
response
<urn:uuid:f7af8a26-591a-40f3-b9fd-4308dc26d647>
isPartOf: CC-MAIN-2023-23
publisher: Common Crawl
description: Wide crawl of the web for May/June 2023
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-7
software: Apache Nutch 1.19 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.4-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/