  Moffat recently commented on our previous work. Our work focused on how
laying the foundations of our evaluation methodology into the theory of
measurement can improve our knowledge and understanding of the evaluation
measures we use in IR and how it can shed light on the different types of
scales adopted by our evaluation measures; we also provided evidence, through
extensive experimentation, on the impact of the different types of scales on
the statistical analyses, as well as on the impact of departing from their
assumptions. Moreover, we investigated, for the first time in IR, the concept
of meaningfulness, i.e. the invariance of the experimental statements and
inferences you draw, and proposed it as a way to ensure more valid and
generalizabile results. Moffat's comments build on: (i) misconceptions about
the representational theory of measurement, such as what an interval scale
actually is and what axioms it has to comply with; (ii) they totally miss the
central concept of meaningfulness. Therefore, we reply to Moffat's comments by
properly framing them in the representational theory of measurement and in the
concept of meaningfulness. All in all, we can only reiterate what we said
several times: the goal of this research line is to theoretically ground our
evaluation methodology - and IR is a field where it is extremely challenging to
perform any theoretical advances - in order to aim for more robust and
generalizable inferences - something we currently lack in the field. Possibly
there are other and better ways to achieve this objective and these proposals
could emerge from an open discussion in the field and from the work of others.
On the other hand, reducing everything to a contrast on what is (or pretend to
be) an interval scale or whether all or none evaluation measures are interval
scales may be more a barrier from than a help in progressing towards this goal.
