Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for
  Improved Cross-Modal Retrieval   Current state-of-the-art approaches to cross-modal retrieval process text and
visual input jointly, relying on Transformer-based architectures with
cross-attention mechanisms that attend over all words and objects in an image.
While offering unmatched retrieval performance, such models: 1) are typically
pretrained from scratch and thus less scalable, 2) suffer from huge retrieval
latency and inefficiency issues, which makes them impractical in realistic
applications. To address these crucial gaps towards both improved and efficient
cross-modal retrieval, we propose a novel fine-tuning framework that turns any
pretrained text-image multi-modal model into an efficient retrieval model. The
framework is based on a cooperative retrieve-and-rerank approach which
combines: 1) twin networks (i.e., a bi-encoder) to separately encode all items
of a corpus, enabling efficient initial retrieval, and 2) a cross-encoder
component for a more nuanced (i.e., smarter) ranking of the retrieved small set
of items. We also propose to jointly fine-tune the two components with shared
weights, yielding a more parameter-efficient model. Our experiments on a series
of standard cross-modal retrieval benchmarks in monolingual, multilingual, and
zero-shot setups, demonstrate improved accuracy and huge efficiency benefits
over the state-of-the-art cross-encoders.

35
The categories of this paper are: mtrl-th
15
The categories of this paper are: nucl-ex
26
The categories of this paper are: bayes-an
32
The categories of this paper are: atom-ph
25
The categories of this paper are: patt-sol
31
The categories of this paper are: ao-sci
14
The categories of this paper are: math-ph
27
The categories of this paper are: comp-gas
19
The categories of this paper are: eess
4
The categories of this paper are: cond-mat
18
The categories of this paper are: econ
13
The categories of this paper are: quant-ph
30
The categories of this paper are: q-alg
0
The categories of this paper are: hep-ph
9
The categories of this paper are: nlin
22
The categories of this paper are: chao-dyn