  With deep learning becoming the dominant approach in computer vision, the use
of representations extracted from Convolutional Neural Nets (CNNs) is quickly
gaining ground on Fisher Vectors (FVs) as favoured state-of-the-art global
image descriptors for image instance retrieval. While the good performance of
CNNs for image classification are unambiguously recognised, which of the two
has the upper hand in the image retrieval context is not entirely clear yet. In
this work, we propose a comprehensive study that systematically evaluates FVs
and CNNs for image retrieval. The first part compares the performances of FVs
and CNNs on multiple publicly available data sets. We investigate a number of
details specific to each method. For FVs, we compare sparse descriptors based
on interest point detectors with dense single-scale and multi-scale variants.
For CNNs, we focus on understanding the impact of depth, architecture and
training data on retrieval results. Our study shows that no descriptor is
systematically better than the other and that performance gains can usually be
obtained by using both types together. The second part of the study focuses on
the impact of geometrical transformations such as rotations and scale changes.
FVs based on interest point detectors are intrinsically resilient to such
transformations while CNNs do not have a built-in mechanism to ensure such
invariance. We show that performance of CNNs can quickly degrade in presence of
rotations while they are far less affected by changes in scale. We then propose
a number of ways to incorporate the required invariances in the CNN pipeline.
Overall, our work is intended as a reference guide offering practically useful
and simply implementable guidelines to anyone looking for state-of-the-art
global descriptors best suited to their specific image instance retrieval
problem.
