  Pretrained contextualized representations offer great success for many
downstream tasks, including document ranking. The multilingual versions of such
pretrained representations provide a possibility of jointly learning many
languages with the same model. Although it is expected to gain big with such
joint training, in the case of cross lingual information retrieval (CLIR), the
models under a multilingual setting are not achieving the same level of
performance as those under a monolingual setting. We hypothesize that the
performance drop is due to the translation gap between query and documents. In
the monolingual retrieval task, because of the same lexical inputs, it is
easier for model to identify the query terms that occurred in documents.
However, in the multilingual pretrained models that the words in different
languages are projected into the same hyperspace, the model tends to translate
query terms into related terms, i.e., terms that appear in a similar context,
in addition to or sometimes rather than synonyms in the target language. This
property is creating difficulties for the model to connect terms that cooccur
in both query and document. To address this issue, we propose a novel Mixed
Attention Transformer (MAT) that incorporates external word level knowledge,
such as a dictionary or translation table. We design a sandwich like
architecture to embed MAT into the recent transformer based deep neural models.
By encoding the translation knowledge into an attention matrix, the model with
MAT is able to focus on the mutually translated words in the input sequence.
Experimental results demonstrate the effectiveness of the external knowledge
and the significant improvement of MAT embedded neural reranking model on CLIR
task.
