  Developing the required technology to assist medical experts in their
everyday activities is currently a hot topic in the Artificial Intelligence
research field. Thus, a number of large language models (LLMs) and automated
benchmarks have recently been proposed with the aim of facilitating information
extraction in Evidence-Based Medicine (EBM) using natural language as a tool
for mediating in human-AI interaction. The most representative benchmarks are
limited to either multiple-choice or long-form answers and are available only
in English. In order to address these shortcomings, in this paper we present a
new dataset which, unlike previous work: (i) includes not only explanatory
arguments for the correct answer, but also arguments to reason why the
incorrect answers are not correct; (ii) the explanations are written originally
by medical doctors to answer questions from the Spanish Residency Medical
Exams. Furthermore, this new benchmark allows us to setup a novel extractive
task which consists of identifying the explanation of the correct answer
written by medical doctors. An additional benefit of our setting is that we can
leverage the extractive QA paradigm to automatically evaluate performance of
LLMs without resorting to costly manual evaluation by medical experts.
Comprehensive experimentation with language models for Spanish shows that
sometimes multilingual models fare better than monolingual ones, even
outperforming models which have been adapted to the medical domain.
Furthermore, results across the monolingual models are mixed, with supposedly
smaller and inferior models performing competitively. In any case, the obtained
results show that our novel dataset and approach can be an effective technique
to help medical practitioners in identifying relevant evidence-based
explanations for medical questions.
