  Since a user usually has to issue a sequence of queries and examine multiple
documents to resolve a complex information need in a search session,
researchers have paid much attention to evaluating search systems at the
session level rather than the single-query level. Most existing session-level
metrics evaluate each query separately and then aggregate the query-level
scores using a session-level weighting function. The assumptions behind these
metrics are that all queries in the session should be involved, and their
orders are fixed. However, if a search system could make the user satisfied
with her first few queries, she may not need any subsequent queries. Besides,
in most real-world search scenarios, due to a lack of explicit feedback from
real users, we can only leverage some implicit feedback, such as users' clicks,
as relevance labels for offline evaluation. Such implicit feedback might be
different from the real relevance in a search session as some documents may be
omitted in the previous query but identified in the later reformulations. To
address the above issues, we make two assumptions about session-based
evaluation, which explicitly describe an ideal session-search system and how to
enhance click-through data in computing session-level evaluation metrics. Based
on our assumptions, we design a session-level metric called Normalized
U-Measure (NUM). NUM evaluates a session as a whole and utilizes an ideal
session to normalize the result of the actual session. Besides, it infers
session-level relevance labels based on implicit feedback. Experiments on two
public datasets demonstrate the effectiveness of NUM by comparing it with
existing session-based metrics in terms of correlation with user satisfaction
and intuitiveness. We also conduct ablation studies to explore whether these
assumptions hold.
