  Passage retrieval is a fundamental task in many information systems, such as
web search and question answering, where both efficiency and effectiveness are
critical concerns. In recent years, neural retrievers based on pre-trained
language models (PLM), such as dual-encoders, have achieved huge success. Yet,
studies have found that the performance of dual-encoders are often limited due
to the neglecting of the interaction information between queries and candidate
passages. Therefore, various interaction paradigms have been proposed to
improve the performance of vanilla dual-encoders. Particularly, recent
state-of-the-art methods often introduce late-interaction during the model
inference process. However, such late-interaction based methods usually bring
extensive computation and storage cost on large corpus. Despite their
effectiveness, the concern of efficiency and space footprint is still an
important factor that limits the application of interaction-based neural
retrieval models. To tackle this issue, we incorporate implicit interaction
into dual-encoders, and propose I^3 retriever. In particular, our implicit
interaction paradigm leverages generated pseudo-queries to simulate
query-passage interaction, which jointly optimizes with query and passage
encoders in an end-to-end manner. It can be fully pre-computed and cached, and
its inference process only involves simple dot product operation of the query
vector and passage vector, which makes it as efficient as the vanilla dual
encoders. We conduct comprehensive experiments on MSMARCO and TREC2019 Deep
Learning Datasets, demonstrating the I^3 retriever's superiority in terms of
both effectiveness and efficiency. Moreover, the proposed implicit interaction
is compatible with special pre-training and knowledge distillation for passage
retrieval, which brings a new state-of-the-art performance.
