# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="p4kLrN3axVrb"
# # ir-measures
#
# This is an interactive demonstration of the [`ir-measures`](https://ir-measur.es/) tool.
#
# Let's start by installing the package via `pip`:

# + colab={"base_uri": "https://localhost:8080/"} id="WmDiVg8CxTxH" outputId="ef6198c7-18fd-4b8f-8b2b-5d44849facb7"
# !pip install -q git+https://github.com/terrierteam/ir_measures

# + [markdown] id="7wSVmX1VzSjb"
# We'll now grab all the data we need. Let's use data from Round 1 of the TREC COVID task.

# + colab={"base_uri": "https://localhost:8080/"} id="vefPwMwMzQyY" outputId="6b1a14d0-48ca-4da7-cb90-0bcd20a8f62c"
# !wget https://ir.nist.gov/covidSubmit/data/qrels-rnd1.txt
# !wget https://ir.nist.gov/covidSubmit/archive/round1/sab20.1.meta.docs
# !wget https://ir.nist.gov/covidSubmit/archive/round1/run2 -O GUIR_s2_run2

# + [markdown] id="zij5_yAd0Mwq"
# ## Command Line Interface
#
# The official evaluation measures for the task were P@5, nDCG@10, AP, and Ppref, but let's also check out the performance of the measures that use binary judgments using a threshold of 2 to see how the systems do on highly relevant documents. We'll also check the judgment rate of the top documents to ensure that they are sufficiently labeled.
#
# We can express these measures naturally to the ir_measures command line tool:

# + colab={"base_uri": "https://localhost:8080/"} id="gn9Ti4E3xlLt" outputId="7f423b43-0841-47ef-d415-e0bb87001ad3"
# !ir_measures qrels-rnd1.txt sab20.1.meta.docs 'P@5 P(rel=2)@5 nDCG@10 AP AP(rel=2) Bpref Bpref(rel=2) Judged@10'

# + colab={"base_uri": "https://localhost:8080/"} id="g667kjjt0i3U" outputId="37eadb39-a369-4879-e23c-642cf0adb3ec"
# !ir_measures qrels-rnd1.txt GUIR_s2_run2 'P@5 P(rel=2)@5 nDCG@10 AP AP(rel=2) Bpref Bpref(rel=2) Judged@10'

# + [markdown] id="hxVZbAry37Eq"
# In the above example, ir-measures automatically runs [trec_eval](https://github.com/usnistgov/trec_eval) twice: once for the measures that do not use a custom relevance thredhold and once for those that have `rel=2`. It also runs the judgment rate script from [OpenNIR](https://opennir.net).
#
# If you have [ir_datasets](https://ir-datasets.com/) installed, you can specify a dataset identifier in place of the qrels file. This takes care of automatically downloading the necessary qrels for you.

# + colab={"base_uri": "https://localhost:8080/"} id="2imc_acv36YI" outputId="a257839d-33cb-4722-f7e7-cb8edc7e4719"
# !pip install -q ir_datasets

# + colab={"base_uri": "https://localhost:8080/"} id="OTukF06O13y1" outputId="196d81db-d1a1-41f1-e533-643da7296b5a"
# !ir_measures cord19/trec-covid/round1 sab20.1.meta.docs 'P@5 P(rel=2)@5 nDCG@10 AP AP(rel=2) Bpref Bpref(rel=2) Judged@10'

# + colab={"base_uri": "https://localhost:8080/"} id="AglPyXR-1-A7" outputId="b481abc9-1c7d-4c5f-b197-73bb8d36b9f7"
# !ir_measures cord19/trec-covid/round1 GUIR_s2_run2 'P@5 P(rel=2)@5 nDCG@10 AP AP(rel=2) Bpref Bpref(rel=2) Judged@10'

# + [markdown] id="2uTXErGo5sEq"
# You can specify other options to the command line tool as well. For instance, if you want per-query results as jsonl format, you can specify the `-q -o jsonl` flags. (This output is pretty long, so we'll just show the top 10 lines using the `head` command.)

# + colab={"base_uri": "https://localhost:8080/"} id="jm57Cs8j5e8S" outputId="d54dd4e0-ac6d-4a31-f334-f5e3ff92a241"
# !ir_measures cord19/trec-covid/round1 sab20.1.meta.docs 'P@5 P(rel=2)@5 nDCG@10 AP AP(rel=2) Bpref Bpref(rel=2) Judged@10' -q -o jsonl | head

# + [markdown] id="jMCXZ0_x6d-O"
# ## Python Interface
#
# We can run the same commands directly in Python as well.

# + id="PxGrWgUJ59zm"
import ir_measures
from ir_measures import * # import natural measure names

# + id="_teXikr26tgE"
# read qrels and run files
qrels = list(ir_measures.read_trec_qrels('qrels-rnd1.txt'))
sab = list(ir_measures.read_trec_run('sab20.1.meta.docs'))
guir = list(ir_measures.read_trec_run('GUIR_s2_run2'))

# + colab={"base_uri": "https://localhost:8080/"} id="8D2Yiw-z6wCd" outputId="7bcbf6cf-0c87-4aa9-fade-554888ad736c"
ir_measures.calc_aggregate([P@5, P(rel=2)@5, nDCG@10, AP, AP(rel=2), Bpref, Bpref(rel=2), Judged@10], qrels, sab)

# + colab={"base_uri": "https://localhost:8080/"} id="bLeRf4EL7A-I" outputId="08763002-2270-4037-bfda-caf818f265a5"
ir_measures.calc_aggregate([P@5, P(rel=2)@5, nDCG@10, AP, AP(rel=2), Bpref, Bpref(rel=2), Judged@10], qrels, guir)

# + [markdown] id="fzp_kMNA7btm"
# The above code is inefficient becuase it needs to process the qrels twice -- once for each run. You can use an evaluator to eliminate this extra work.

# + id="5-iqV_8a7Z0I"
evaluator = ir_measures.evaluator([P@5, P(rel=2)@5, nDCG@10, AP, AP(rel=2), Bpref, Bpref(rel=2), Judged@10], qrels)

# + colab={"base_uri": "https://localhost:8080/"} id="CEBcgYxq7v6I" outputId="4b6dc938-3a2a-4ccb-cbf1-6acf4db459ab"
evaluator.calc_aggregate(guir)

# + colab={"base_uri": "https://localhost:8080/"} id="qmtt5Nyz7xgA" outputId="861b8685-8550-4b51-8740-f7474e77c325"
from timeit import timeit
time = timeit(lambda: ir_measures.calc_aggregate([P@5, P(rel=2)@5, nDCG@10, AP, AP(rel=2), Bpref, Bpref(rel=2), Judged@10], qrels, guir), number=10)
print(f'ir_measures.calc_aggregate: {time/10*1000:0.2f}ms/invocation')
time = timeit(lambda: evaluator.calc_aggregate(guir), number=10)
print(f'evaluator.calc_aggregate:   {time/10*1000:0.2f}ms/invocation')

# + [markdown] id="wTdMxd_x9L2z"
# You can also get per-query results using `iter_calc`. This allows us to analyse per-query performance and conduct statistical tests.

# + colab={"base_uri": "https://localhost:8080/"} id="TmWvXOhb75gN" outputId="76614515-debc-4a76-efc8-889903c0cb61"
count = 0
for metric in ir_measures.iter_calc([P@5, P(rel=2)@5, nDCG@10, AP, AP(rel=2), Bpref, Bpref(rel=2), Judged@10], qrels, guir):
  print(metric)
  count += 1
  if count >= 10: break # only show top 10 items

sab_p_rel2_5 = {m.query_id: m.value for m in ir_measures.iter_calc([Bpref(rel=2)], qrels, sab)}
guir_p_rel2_5 = {m.query_id: m.value for m in ir_measures.iter_calc([Bpref(rel=2)], qrels, guir)}

from scipy.stats import ttest_rel
qids = list(sab_p_rel2_5.keys())
ttest_rel([sab_p_rel2_5[v] for v in qids], [guir_p_rel2_5[v] for v in qids])

# + [markdown] id="10kg_Jm4-9U5"
# ## PyTerrer Integration
#
# ir-datasets is easy to use in other tools. Here, we see how [PyTerrier](https://pyterrier.readthedocs.io/) uses ir-measures for specifying evaluation criteria in experiments:

# + colab={"base_uri": "https://localhost:8080/"} id="RJqcv6aD9VVV" outputId="924142ca-6667-42c8-88aa-df651504b471"
# !pip install -q git+https://github.com/terrier-org/pyterrier.git

# + colab={"base_uri": "https://localhost:8080/"} id="1WAYUdsM_Poh" outputId="a71461f0-1cd9-44d6-a993-1f82f5c788ad"
import pyterrier as pt
if not pt.started():
  pt.init()

# + colab={"base_uri": "https://localhost:8080/", "height": 556, "referenced_widgets": ["07432eee05c14c38916241410f692c6a", "cdf406e1749c4479bd268a22db11526c", "a82c0181c97e48249b775698204b63a7", "86a5d201e25944dbb4e63c7e9471cdba", "e80b23c4f32643b988237e376b489e50", "<KEY>", "8b49253640a44551a24901913a73648b", "20f4ae3ead36436488d46929248f6641", "b31231ff52aa4d6cbd999c8017526ec2", "12561d8aff784a08a54c3e8265f1f7ba", "8b12754b22a0416ca75895e2e710e913"]} id="a-5mqmTL_UVK" outputId="21e41ab3-64b6-4065-f6e1-7e20b00f7773"
# NOTE: this example uses TREC COVID complete, rather than round1
dataset = pt.get_dataset('irds:cord19/trec-covid')
pt.Experiment(
    [pt.BatchRetrieve.from_dataset('trec-covid', 'terrier_stemmed', wmodel='DPH'),
     pt.BatchRetrieve.from_dataset('trec-covid', 'terrier_stemmed', wmodel='BM25')],
    dataset.get_topics('description'),
    dataset.get_qrels(),
    eval_metrics=[P@5, P(rel=2)@5, nDCG@10, AP, AP(rel=2), Bpref, Bpref(rel=2), Judged@10],
#                 ^ using ir_measures
)

# + id="xvq2uicI_yGd"


StarcoderdataJupyterScriptsDedupFiltered