  Although the vision-and-language pretraining (VLP) equipped cross-modal
image-text retrieval (ITR) has achieved remarkable progress in the past two
years, it suffers from a major drawback: the ever-increasing size of VLP models
restricts its deployment to real-world search scenarios (where the high latency
is unacceptable). To alleviate this problem, we present a novel plug-in dynamic
contrastive distillation (DCD) framework to compress the large VLP models for
the ITR task. Technically, we face the following two challenges: 1) the typical
uni-modal metric learning approach is difficult to directly apply to the
cross-modal tasks, due to the limited GPU memory to optimize too many negative
samples during handling cross-modal fusion features. 2) it is inefficient to
static optimize the student network from different hard samples, which have
different effects on distillation learning and student network optimization. We
try to overcome these challenges from two points. First, to achieve multi-modal
contrastive learning, and balance the training costs and effects, we propose to
use a teacher network to estimate the difficult samples for students, making
the students absorb the powerful knowledge from pre-trained teachers, and
master the knowledge from hard samples. Second, to dynamic learn from hard
sample pairs, we propose dynamic distillation to dynamically learn samples of
different difficulties, from the perspective of better balancing the difficulty
of knowledge and students' self-learning ability. We successfully apply our
proposed DCD strategy to two state-of-the-art vision-language pretrained
models, i.e. ViLT and METER. Extensive experiments on MS-COCO and Flickr30K
benchmarks show the effectiveness and efficiency of our DCD framework.
Encouragingly, we can speed up the inference at least 129$\times$ compared to
the existing ITR models.
