<jupyter_start><jupyter_text>AlignmentsThis notebook analyzes page alignments and prepares metrics for final use.  SetupWe begin by loading necessary libraries:<jupyter_code>from pathlib import Path
import pandas as pd
import xarray as xr
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import gzip
import pickle
import binpickle
from natural.size import binarysize
codec = binpickle.codecs.Blosc('zstd')<jupyter_output><empty_output><jupyter_text>Set up progress bar and logging support:<jupyter_code>from tqdm.auto import tqdm
tqdm.pandas(leave=False)
import sys, logging
logging.basicConfig(level=logging.INFO, stream=sys.stderr)
log = logging.getLogger('alignment')<jupyter_output><empty_output><jupyter_text>Import metric code:<jupyter_code>%load_ext autoreload
%autoreload 1
%aimport metrics
from trecdata import scan_runs<jupyter_output><empty_output><jupyter_text>Loading DataWe first load the page metadata:<jupyter_code>pages = pd.read_json('data/trec_metadata_eval.json.gz', lines=True)
pages = pages.drop_duplicates('page_id')
pages.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 6023415 entries, 0 to 6023435
Data columns (total 5 columns):
 #   Column                Dtype  
---  ------                -----  
 0   page_id               int64  
 1   quality_score         float64
 2   quality_score_disc    object 
 3   geographic_locations  object 
 4   gender                object 
dtypes: float64(1), int64(1), object(3)
memory usage: 275.7+ MB<jupyter_text>Now we will load the evaluation topics:<jupyter_code>eval_topics = pd.read_json('data/eval-topics-with-qrels.json.gz', lines=True)
eval_topics.info()
train_topics = pd.read_json('data/trec_topics.json.gz', lines=True)
train_topics.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 57 entries, 0 to 56
Data columns (total 6 columns):
 #   Column    Non-Null Count  Dtype 
---  ------    --------------  ----- 
 0   id        57 non-null     int64 
 1   title     57 non-null     object
 2   keywords  57 non-null     object
 3   scope     57 non-null     object
 4   homepage  57 non-null     object
 5   rel_docs  57 non-null     object
dtypes: int64(1), object(5)
memory usage: 2.8+ KB<jupyter_text>Train and eval topics use a disjoint set of IDs:<jupyter_code>train_topics['id'].describe()
eval_topics['id'].describe()<jupyter_output><empty_output><jupyter_text>This allows us to create a single, integrated topics list for convenience:<jupyter_code>topics = pd.concat([train_topics, eval_topics], ignore_index=True)
topics['eval'] = False
topics.loc[topics['id'] >= 100, 'eval'] = True
topics.head()<jupyter_output><empty_output><jupyter_text>Finally, a bit of hard-coded data - the world population:<jupyter_code>world_pop = pd.Series({
    'Africa': 0.155070563,
    'Antarctica': 1.54424E-07,
    'Asia': 0.600202585,
    'Europe': 0.103663858,
    'Latin America and the Caribbean': 0.08609797,
    'Northern America': 0.049616733,
    'Oceania': 0.005348137,
})
world_pop.name = 'geography'<jupyter_output><empty_output><jupyter_text>And a gender global target:<jupyter_code>gender_tgt = pd.Series({
    'female': 0.495,
    'male': 0.495,
    'third': 0.01
})
gender_tgt.name = 'gender'
gender_tgt.sum()<jupyter_output><empty_output><jupyter_text>Xarray intesectional global target:<jupyter_code>geo_tgt_xa = xr.DataArray(world_pop, dims=['geography'])
gender_tgt_xa = xr.DataArray(gender_tgt, dims=['gender'])
int_tgt = geo_tgt_xa * gender_tgt_xa
int_tgt<jupyter_output><empty_output><jupyter_text>And the order of work-needed codes:<jupyter_code>work_order = [
    'Stub',
    'Start',
    'C',
    'B',
    'GA',
    'FA',
]<jupyter_output><empty_output><jupyter_text>Query RelevanceWe now need to get the qrels for the topics.  This is done by creating frames with entries for every relevant document; missing documents are assumed irrelevant (0).First the training topics:<jupyter_code>train_qrels = train_topics[['id', 'rel_docs']].explode('rel_docs', ignore_index=True)
train_qrels.rename(columns={'rel_docs': 'page_id'}, inplace=True)
train_qrels['page_id'] = train_qrels['page_id'].astype('i4')
train_qrels = train_qrels.drop_duplicates()
train_qrels.head()
eval_qrels = eval_topics[['id', 'rel_docs']].explode('rel_docs', ignore_index=True)
eval_qrels.rename(columns={'rel_docs': 'page_id'}, inplace=True)
eval_qrels['page_id'] = eval_qrels['page_id'].astype('i4')
eval_qrels = eval_qrels.drop_duplicates()
eval_qrels.head()<jupyter_output><empty_output><jupyter_text>And concatenate:<jupyter_code>qrels = pd.concat([train_qrels, eval_qrels], ignore_index=True)<jupyter_output><empty_output><jupyter_text>Page AlignmentsAll of our metrics require page "alignments": the protected-group membership of each page.  GeographyLet's start with the straight page geography alignment for the public evaluation of the training queries.  The page metadata has that; let's get the geography column.<jupyter_code>page_geo = pages[['page_id', 'geographic_locations']].explode('geographic_locations', ignore_index=True)
page_geo.head()<jupyter_output><empty_output><jupyter_text>And we will now pivot this into a matrix so we get page alignment vectors:<jupyter_code>page_geo_align = page_geo.assign(x=1).pivot(index='page_id', columns='geographic_locations', values='x')
page_geo_align.rename(columns={np.nan: 'Unknown'}, inplace=True)
page_geo_align.fillna(0, inplace=True)
page_geo_align.head()<jupyter_output><empty_output><jupyter_text>And convert this to an xarray for multidimensional usage:<jupyter_code>page_geo_xr = xr.DataArray(page_geo_align, dims=['page', 'geography'])
page_geo_xr
binarysize(page_geo_xr.nbytes)<jupyter_output><empty_output><jupyter_text>GenderThe "undisclosed personal attribute" is gender.  Not all articles have gender as a relevant variable - articles not about a living being generally will not.We're going to follow the same approach for gender:<jupyter_code>page_gender = pages[['page_id', 'gender']].explode('gender', ignore_index=True)
page_gender.fillna('unknown', inplace=True)
page_gender.head()<jupyter_output><empty_output><jupyter_text>We need to do a little targeted repair - there is an erroneous record of a gender of "Taira no Kiyomori" is actually male. Replace that:<jupyter_code>page_gender = page_gender.loc[page_gender['gender'] != 'Taira no Kiyomori']<jupyter_output><empty_output><jupyter_text>Now, we're going to do a little more work to reduce the dimensionality of the space.  Points:1. Trans men are men2. Trans women are women3. Cisgender is an adjective that can be dropped for the present purposesThe result is that we will collapse "transgender female" and "cisgender female" into "female".The **downside** to this is that trans men are probabily significantly under-represented, but are now being collapsed into the dominant group.<jupyter_code>pgcol = page_gender['gender']
pgcol = pgcol.str.replace(r'(?:tran|ci)sgender\s+((?:fe)?male)', r'\1', regex=True)<jupyter_output><empty_output><jupyter_text>Now, we're going to group the remaining gender identities together under the label 'third'.  As noted above, this is a debatable exercise that collapses a lot of identity.<jupyter_code>genders = ['unknown', 'male', 'female', 'third']
pgcol[~pgcol.isin(genders)] = 'third'<jupyter_output><empty_output><jupyter_text>Now put this column back in the frame and deduplicate.<jupyter_code>page_gender['gender'] = pgcol
page_gender = page_gender.drop_duplicates()<jupyter_output><empty_output><jupyter_text>And make an alignment matrix (reordering so 'unknown' is first for consistency):<jupyter_code>page_gend_align = page_gender.assign(x=1).pivot(index='page_id', columns='gender', values='x')
page_gend_align.fillna(0, inplace=True)
page_gend_align = page_gend_align.reindex(columns=['unknown', 'female', 'male', 'third'])
page_gend_align.head()<jupyter_output><empty_output><jupyter_text>Let's see how frequent each of the genders is:<jupyter_code>page_gend_align.sum(axis=0).sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>And convert to an xarray:<jupyter_code>page_gend_xr = xr.DataArray(page_gend_align, dims=['page', 'gender'])
page_gend_xr
binarysize(page_gend_xr.nbytes)<jupyter_output><empty_output><jupyter_text>Intersectional AlignmentWe'll now convert this data array to an **intersectional** alignment array:<jupyter_code>page_xalign = page_geo_xr * page_gend_xr
page_xalign
binarysize(page_xalign.nbytes)<jupyter_output><empty_output><jupyter_text>Make sure that did the right thing and we have intersectional numbers:<jupyter_code>page_xalign.sum(axis=0)<jupyter_output><empty_output><jupyter_text>And make sure combination with targets work as expected:<jupyter_code>(page_xalign.sum(axis=0) + int_tgt) * 0.5<jupyter_output><empty_output><jupyter_text>Task 1 Metric PreparationNow that we have our alignments and qrels, we are ready to prepare the Task 1 metrics.Task 1 ignores the "unknown" alignment category, so we're going to create a `kga` frame (for **K**nown **G**eographic **A**lignment), and corresponding frames for intersectional alignment.<jupyter_code>page_kga = page_geo_align.iloc[:, 1:]
page_kga.head()<jupyter_output><empty_output><jupyter_text>Intersectional is a little harder to do, because things can be **intersectionally unknown**: we may know gender but not geography, or vice versa.  To deal with these missing values for Task 1, we're going to ignore *totally unknown* values, but keep partially-known as a category.We also need to ravel our tensors into a matrix for compatibility with the metric code. Since 'unknown' is the first value on each axis, we can ravel, and then drop the first column.<jupyter_code>xshp = page_xalign.shape
xshp = (xshp[0], xshp[1] * xshp[2])
page_xa_df = pd.DataFrame(page_xalign.values.reshape(xshp), index=page_xalign.indexes['page'])
page_xa_df.head()<jupyter_output><empty_output><jupyter_text>And drop unknown, to get our page alignment vectors:<jupyter_code>page_kia = page_xa_df.iloc[:, 1:]<jupyter_output><empty_output><jupyter_text>Geographic AlignmentWe'll start with the metric configuration for public training data, considering only geographic alignment.  We configure the metric to do this for both the training and the eval queries. Training Queries<jupyter_code>train_qalign = train_qrels.join(page_kga, on='page_id').drop(columns=['page_id']).groupby('id').sum()
tqa_sums = train_qalign.sum(axis=1)
train_qalign = train_qalign.divide(tqa_sums, axis=0)
train_qalign.head()
train_qtarget = (train_qalign + world_pop) * 0.5
train_qtarget.head()<jupyter_output><empty_output><jupyter_text>And we can prepare a metric and save it:<jupyter_code>t1_train_metric = metrics.Task1Metric(train_qrels.set_index('id'), page_kga, train_qtarget)
binpickle.dump(t1_train_metric, 'task1-train-geo-metric.bpk', codec=codec)<jupyter_output>INFO:binpickle.write:pickled 337312647 bytes with 5 buffers<jupyter_text>Eval QueriesDo the same thing for the eval data for a geo-only eval metric:<jupyter_code>eval_qalign = eval_qrels.join(page_kga, on='page_id').drop(columns=['page_id']).groupby('id').sum()
eqa_sums = eval_qalign.sum(axis=1)
eval_qalign = eval_qalign.divide(eqa_sums, axis=0)
eval_qtarget = (eval_qalign + world_pop) * 0.5
t1_eval_metric = metrics.Task1Metric(eval_qrels.set_index('id'), page_kga, eval_qtarget)
binpickle.dump(t1_eval_metric, 'task1-eval-geo-metric.bpk', codec=codec)<jupyter_output>INFO:binpickle.write:pickled 337312643 bytes with 5 buffers<jupyter_text>Intersectional AlignmentNow we need to apply similar logic, but for the intersectional (geography * gender) alignment.As noted as above, we need to carefully handle the unknown cases.  DemoTo demonstrate how the logic works, let's first work it out in cells for one query (1).What are its documents?<jupyter_code>qdf = qrels[qrels['id'] == 1]
qdf.name = 1
qdf<jupyter_output><empty_output><jupyter_text>We can use these page IDs to get its alignments:<jupyter_code>q_xa = page_xalign.loc[qdf['page_id'].values, :, :]
q_xa<jupyter_output><empty_output><jupyter_text>Summing over the first axis ('page') will produce an alignment matrix:<jupyter_code>q_am = q_xa.sum(axis=0)
q_am<jupyter_output><empty_output><jupyter_text>Now we need to do reset the (0,0) coordinate (full unknown), and normalize to a proportion.<jupyter_code>q_am[0, 0] = 0
q_am = q_am / q_am.sum()
q_am<jupyter_output><empty_output><jupyter_text>Ok, now we have to - very carefully - average with our target modifier.  There are three groups:- known (use intersectional target)- known-geo (use geo target)- known-gender (use gender target)For each of these, we need to respect the fraction of the total it represents.  Let's compute those fractions:<jupyter_code>q_fk_all = q_am[1:, 1:].sum()
q_fk_geo = q_am[1:, :1].sum()
q_fk_gen = q_am[:1, 1:].sum()
q_fk_all, q_fk_geo, q_fk_gen<jupyter_output><empty_output><jupyter_text>And now do some surgery.  Weighted-average to incorporate the target for fully-known:<jupyter_code>q_tm = q_am.copy()
q_tm[1:, 1:] *= 0.5
q_tm[1:, 1:] += int_tgt * 0.5 * q_fk_all
q_tm<jupyter_output><empty_output><jupyter_text>And for known-geo:<jupyter_code>q_tm[1:, :1] *= 0.5
q_tm[1:, :1] += geo_tgt_xa * 0.5 * q_fk_geo<jupyter_output><empty_output><jupyter_text>And known-gender:<jupyter_code>q_tm[:1, 1:] *= 0.5
q_tm[:1, 1:] += gender_tgt_xa * 0.5 * q_fk_gen
q_tm<jupyter_output><empty_output><jupyter_text>Now we can unravel this and drop the first entry:<jupyter_code>q_tm.values.ravel()[1:]<jupyter_output><empty_output><jupyter_text>ImplementationNow, to do this for every query, we'll use a function that takes a data frame for a query's relevant docs and performs all of the above operations:<jupyter_code>def query_xalign(qdf):
    pages = qdf['page_id']
    pages = pages[pages.isin(page_xalign.indexes['page'])]
    q_xa = page_xalign.loc[pages.values, :, :]
    q_am = q_xa.sum(axis=0)

    # clear and normalize
    q_am[0, 0] = 0
    q_am = q_am / q_am.sum()
    
    # compute fractions in each section
    q_fk_all = q_am[1:, 1:].sum()
    q_fk_geo = q_am[1:, :1].sum()
    q_fk_gen = q_am[:1, 1:].sum()
    
    # known average
    q_am[1:, 1:] *= 0.5
    q_am[1:, 1:] += int_tgt * 0.5 * q_fk_all
    
    # known-geo average
    q_am[1:, :1] *= 0.5
    q_am[1:, :1] += geo_tgt_xa * 0.5 * q_fk_geo
    
    # known-gender average
    q_am[:1, 1:] *= 0.5
    q_am[:1, 1:] += gender_tgt_xa * 0.5 * q_fk_gen
    
    # and return the result
    return pd.Series(q_am.values.ravel()[1:])
query_xalign(qdf)<jupyter_output><empty_output><jupyter_text>Now with that function, we can compute the alignment vector for each query.<jupyter_code>train_qtarget = train_qrels.groupby('id').apply(query_xalign)
train_qtarget<jupyter_output><empty_output><jupyter_text>And save:<jupyter_code>t1_train_metric = metrics.Task1Metric(train_qrels.set_index('id'), page_kia, train_qtarget)
binpickle.dump(t1_train_metric, 'task1-train-metric.bpk', codec=codec)<jupyter_output>INFO:binpickle.write:pickled 1493808204 bytes with 5 buffers<jupyter_text>Do the same for eval:<jupyter_code>eval_qtarget = eval_qrels.groupby('id').apply(query_xalign)
t1_eval_metric = metrics.Task1Metric(eval_qrels.set_index('id'), page_kia, eval_qtarget)
binpickle.dump(t1_eval_metric, 'task1-eval-metric.bpk', codec=codec)<jupyter_output>INFO:binpickle.write:pickled 1493808200 bytes with 5 buffers<jupyter_text>Task 2 Metric PreparationTask 2 requires some different preparation.We're going to start by computing work-needed information:<jupyter_code>page_work = pages.set_index('page_id').quality_score_disc.astype(pd.CategoricalDtype(ordered=True))
page_work = page_work.cat.reorder_categories(work_order)
page_work.name = 'quality'<jupyter_output><empty_output><jupyter_text>Work and Target ExposureThe first thing we need to do to prepare the metric is to compute the work-needed for each topic's pages, and use that to compute the target exposure for each (relevant) page in the topic.This is because an ideal ranking orders relevant documents in decreasing order of work needed, followed by irrelevant documents.  All relevant documents at a given work level should receive the same expected exposure.First, look up the work for each query page ('query page work', or qpw):<jupyter_code>qpw = qrels.join(page_work, on='page_id')
qpw<jupyter_output><empty_output><jupyter_text>And now  use that to compute the number of documents at each work level:<jupyter_code>qwork = qpw.groupby(['id', 'quality'])['page_id'].count()
qwork<jupyter_output><empty_output><jupyter_text>Now we need to convert this into target exposure levels.  This function will, given a series of counts for each work level, compute the expected exposure a page at that work level should receive.<jupyter_code>def qw_tgt_exposure(qw_counts: pd.Series) -> pd.Series:
    if 'id' == qw_counts.index.names[0]:
        qw_counts = qw_counts.reset_index(level='id', drop=True)
    qwc = qw_counts.reindex(work_order, fill_value=0).astype('i4')
    tot = int(qwc.sum())
    da = metrics.discount(tot)
    qwp = qwc.shift(1, fill_value=0)
    qwc_s = qwc.cumsum()
    qwp_s = qwp.cumsum()
    res = pd.Series(
        [np.mean(da[s:e]) for (s, e) in zip(qwp_s, qwc_s)],
        index=qwc.index
    )
    return res<jupyter_output><empty_output><jupyter_text>We'll then apply this to each topic, to determine the per-topic target exposures:<jupyter_code>qw_pp_target = qwork.groupby('id').apply(qw_tgt_exposure)
qw_pp_target.name = 'tgt_exposure'
qw_pp_target<jupyter_output>C:\Users\michaelekstrand\Miniconda3\envs\wptrec\lib\site-packages\numpy\core\fromnumeric.py:3440: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
C:\Users\michaelekstrand\Miniconda3\envs\wptrec\lib\site-packages\numpy\core\_methods.py:189: RuntimeWarning: invalid value encountered in true_divide
  ret = ret.dtype.type(ret / rcount)<jupyter_text>We can now merge the relevant document work categories with this exposure, to compute the target exposure for each relevant document:<jupyter_code>qp_exp = qpw.join(qw_pp_target, on=['id', 'quality'])
qp_exp = qp_exp.set_index(['id', 'page_id'])['tgt_exposure']
qp_exp.index.names = ['q_id', 'page_id']
qp_exp<jupyter_output><empty_output><jupyter_text>Geographic AlignmentNow that we've computed per-page target exposure, we're ready to set up the geographic alignment vectors for computing the per-*group* expected exposure with geographic data.We're going to start by getting the alignments for relevant documents for each topic:<jupyter_code>qp_geo_align = qrels.join(page_geo_align, on='page_id').set_index(['id', 'page_id'])
qp_geo_align.index.names = ['q_id', 'page_id']
qp_geo_align<jupyter_output><empty_output><jupyter_text>Now we need to compute the per-query target exposures.  This starst with aligning our vectors:<jupyter_code>qp_geo_exp, qp_geo_align = qp_exp.align(qp_geo_align, fill_value=0)<jupyter_output><empty_output><jupyter_text>And now we can multiply the exposure vector by the alignment vector, and summing by topic - this is equivalent to the matrix-vector multiplication on a topic-by-topic basis.<jupyter_code>qp_aexp = qp_geo_align.multiply(qp_geo_exp, axis=0)
q_geo_align = qp_aexp.groupby('q_id').sum()<jupyter_output><empty_output><jupyter_text>Now things get a *little* weird.  We want to average the empirical distribution with the world population to compute our fairness target.  However, we don't have empirical data on the distribution of articles that do or do not have geographic alignments.Therefore, we are going to average only the *known-geography* vector with the world population.  This proceeds in N steps:1. Normalize the known-geography matrix so its rows sum to 1.2. Average each row with the world population.3. De-normalize the known-geography matrix so it is in the original scale, but adjusted w/ world population4. Normalize the *entire* matrix so its rows sum to 1Let's go.<jupyter_code>qg_known = q_geo_align.drop(columns=['Unknown'])<jupyter_output><empty_output><jupyter_text>Normalize (adding a small value to avoid division by zero - affected entries will have a zero numerator anyway):<jupyter_code>qg_ksums = qg_known.sum(axis=1)
qg_kd = qg_known.divide(np.maximum(qg_ksums, 1.0e-6), axis=0)<jupyter_output><empty_output><jupyter_text>Average:<jupyter_code>qg_kd = (qg_kd + world_pop) * 0.5<jupyter_output><empty_output><jupyter_text>De-normalize:<jupyter_code>qg_known = qg_kd.multiply(qg_ksums, axis=0)<jupyter_output><empty_output><jupyter_text>Recombine with the Unknown column:<jupyter_code>q_geo_tgt = q_geo_align[['Unknown']].join(qg_known)<jupyter_output><empty_output><jupyter_text>Normalize targets:<jupyter_code>q_geo_tgt = q_geo_tgt.divide(q_geo_tgt.sum(axis=1), axis=0)
q_geo_tgt<jupyter_output><empty_output><jupyter_text>This is our group exposure target distributions for each query, for the geographic data.  We're now ready to set up the matrix.<jupyter_code>train_geo_qtgt = q_geo_tgt.loc[train_topics['id']]
eval_geo_qtgt = q_geo_tgt.loc[eval_topics['id']]
t2_train_geo_metric = metrics.Task2Metric(train_qrels.set_index('id'), 
                                          page_geo_align, page_work, 
                                          train_geo_qtgt)
binpickle.dump(t2_train_geo_metric, 'task2-train-geo-metric.bpk', codec=codec)
t2_eval_geo_metric = metrics.Task2Metric(eval_qrels.set_index('id'), 
                                         page_geo_align, page_work, 
                                         eval_geo_qtgt)
binpickle.dump(t2_eval_geo_metric, 'task2-eval-geo-metric.bpk', codec=codec)<jupyter_output>INFO:binpickle.write:pickled 2014 bytes with 9 buffers<jupyter_text>Intersectional AlignmentNow we need to compute the intersectional targets for Task 2.  We're going to take a slightly different approach here, based on the intersectional logic for Task 1, because we've come up with better ways to write the code, but the effect is the same: only known aspects are averaged.We'll write a function very similar to the one for Task 1:<jupyter_code>def query_xideal(qdf, ravel=True):
    pages = qdf['page_id']
    pages = pages[pages.isin(page_xalign.indexes['page'])]
    q_xa = page_xalign.loc[pages.values, :, :]
    
    # now we need to get the exposure for the pages, and multiply
    p_exp = qp_exp.loc[qdf.name]
    assert p_exp.index.is_unique
    p_exp = xr.DataArray(p_exp, dims=['page'])
    
    # and we multiply!
    q_xa = q_xa * p_exp

    # normalize into a matrix (this time we don't clear)
    q_am = q_xa.sum(axis=0)
    q_am = q_am / q_am.sum()
    
    # compute fractions in each section - combined with q_am[0,0], this should be about 1
    q_fk_all = q_am[1:, 1:].sum()
    q_fk_geo = q_am[1:, :1].sum()
    q_fk_gen = q_am[:1, 1:].sum()
    
    # known average
    q_am[1:, 1:] *= 0.5
    q_am[1:, 1:] += int_tgt * 0.5 * q_fk_all
    
    # known-geo average
    q_am[1:, :1] *= 0.5
    q_am[1:, :1] += geo_tgt_xa * 0.5 * q_fk_geo
    
    # known-gender average
    q_am[:1, 1:] *= 0.5
    q_am[:1, 1:] += gender_tgt_xa * 0.5 * q_fk_gen
    
    # and return the result
    if ravel:
        return pd.Series(q_am.values.ravel())
    else:
        return q_am<jupyter_output><empty_output><jupyter_text>Test this function out:<jupyter_code>query_xideal(qdf, ravel=False)<jupyter_output><empty_output><jupyter_text>And let's go!<jupyter_code>q_xtgt = qrels.groupby('id').progress_apply(query_xideal)
q_xtgt
train_qtgt = q_xtgt.loc[train_topics['id']]
eval_qtgt = q_xtgt.loc[eval_topics['id']]
t2_train_metric = metrics.Task2Metric(train_qrels.set_index('id'), 
                                      page_xa_df, page_work, 
                                      train_qtgt)
binpickle.dump(t2_train_metric, 'task2-train-metric.bpk', codec=codec)
t2_eval_metric = metrics.Task2Metric(eval_qrels.set_index('id'), 
                                     page_xa_df, page_work, 
                                     eval_qtgt)
binpickle.dump(t2_eval_metric, 'task2-eval-metric.bpk', codec=codec)<jupyter_output>INFO:binpickle.write:pickled 1875 bytes with 9 buffers
StarcoderdataJupyterStructuredCleanDedup