\section{Introduction}

Please follow the steps outlined below when submitting your manuscript to
the IEEE Computer Society Press.  This style guide now has several
important modifications (for example, you are no longer warned against the
use of sticky tape to attach your artwork to the paper), so all authors
should read this new version.

\subsection{Language}

All manuscripts must be in English.

\subsection{Dual submission}

Please refer to the author guidelines on the ICCV 2019 web page for a
discussion of the policy on dual submissions.

\subsection{Paper length}
Papers, excluding the references section,
must be no longer than eight pages in length. The references section
will not be included in the page count, and there is no limit on the
length of the references section. For example, a paper of eight pages
with two pages of references would have a total length of 10 pages.

Overlength papers will simply not be reviewed.  This includes papers
where the margins and formatting are deemed to have been significantly
altered from those laid down by this style guide.  Note that this
\LaTeX\ guide already sets figure captions and references in a smaller font.
The reason such papers will not be reviewed is that there is no provision for
supervised revisions of manuscripts.  The reviewing process cannot determine
the suitability of the paper for presentation in eight pages if it is
reviewed in eleven.

\subsection{The ruler}
The \LaTeX\ style defines a printed ruler which should be present in the
version submitted for review.  The ruler is provided in order that
reviewers may comment on particular lines in the paper without
circumlocution.  If you are preparing a document using a non-\LaTeX\
document preparation system, please arrange for an equivalent ruler to
appear on the final output pages.  The presence or absence of the ruler
should not change the appearance of any other content on the page.  The
camera ready copy should not contain a ruler. (\LaTeX\ users may uncomment
the \verb'\iccvfinalcopy' command in the document preamble.)  Reviewers:
note that the ruler measurements do not align well with lines in the paper
--- this turns out to be very difficult to do well when the paper contains
many figures and equations, and, when done, looks ugly.  Just use fractional
references (e.g.\ this line is $095.5$), although in most cases one would
expect that the approximate location will be adequate.

\subsection{Mathematics}

Please number all of your sections and displayed equations.  It is
important for readers to be able to refer to any particular equation.  Just
because you didn't refer to it in the text doesn't mean some future reader
might not need to refer to it.  It is cumbersome to have to use
circumlocutions like ``the equation second from the top of page 3 column
1''.  (Note that the ruler will not be present in the final copy, so is not
an alternative to equation numbers).  All authors will benefit from reading
Mermin's description of how to write mathematics:
\url{http://www.pamitc.org/documents/mermin.pdf}.

\subsection{Blind review}

Many authors misunderstand the concept of anonymizing for blind
review.  Blind review does not mean that one must remove
citations to one's own work---in fact it is often impossible to
review a paper unless the previous citations are known and
available.

Blind review means that you do not use the words ``my'' or ``our''
when citing previous work.  That is all.  (But see below for
techreports.)

Saying ``this builds on the work of Lucy Smith [1]'' does not say
that you are Lucy Smith; it says that you are building on her
work.  If you are Smith and Jones, do not say ``as we show in
[7]'', say ``as Smith and Jones show in [7]'' and at the end of the
paper, include reference 7 as you would any other cited work.

An example of a bad paper just asking to be rejected:
\begin{quote}
\begin{center}
    An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of our
   previous paper [1], and show it to be inferior to all
   previously known methods.  Why the previous paper was
   accepted without this analysis is beyond me.

   [1] Removed for blind review
\end{quote}

An example of an acceptable paper:

\begin{quote}
\begin{center}
     An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of the
   paper of Smith \emph{et al}\onedot [1], and show it to be inferior to
   all previously known methods.  Why the previous paper
   was accepted without this analysis is beyond me.

   [1] Smith, L and Jones, C. ``The frobnicatable foo
   filter, a fundamental contribution to human knowledge''.
   Nature 381(12), 1-213.
\end{quote}

If you are making a submission to another conference at the same time,
which covers similar or overlapping material, you may need to refer to that
submission in order to explain the differences, just as you would if you
had previously published related work.  In such cases, include the
anonymized parallel submission~\cite{Authors14} as additional material and
cite it as
\begin{quote}
[1] Authors. ``The frobnicatable foo filter'', F\&G 2014 Submission ID 324,
Supplied as additional material {\tt fg324.pdf}.
\end{quote}

Finally, you may feel you need to tell the reader that more details can be
found elsewhere, and refer them to a technical report.  For conference
submissions, the paper must stand on its own, and not {\em require} the
reviewer to go to a techreport for further details.  Thus, you may say in
the body of the paper ``further details may be found
in~\cite{Authors14b}''.  Then submit the techreport as additional material.
Again, you may not assume the reviewers will read this material.

Sometimes your paper is about a problem which you tested using a tool which
is widely known to be restricted to a single institution.  For example,
let's say it's 1969, you have solved a key problem on the Apollo lander,
and you believe that the ICCV70 audience would like to hear about your
solution.  The work is a development of your celebrated 1968 paper entitled
``Zero-g frobnication: How being the only people in the world with access to
the Apollo lander source code makes us a wow at parties'', by Zeus \emph{et al}\onedot.

You can handle this paper like any other.  Don't write ``We show how to
improve our previous work [Anonymous, 1968].  This time we tested the
algorithm on a lunar lander [name of lander removed for blind review]''.
That would be silly, and would immediately identify the authors. Instead
write the following:
\begin{quotation}
\noindent
   We describe a system for zero-g frobnication.  This
   system is new because it handles the following cases:
   A, B.  Previous systems [Zeus et al. 1968] didn't
   handle case B properly.  Ours handles it by including
   a foo term in the bar integral.

   ...

   The proposed system was integrated with the Apollo
   lunar lander, and went all the way to the moon, don't
   you know.  It displayed the following behaviours
   which show how well we solved cases A and B: ...
\end{quotation}
As you can see, the above text follows standard scientific convention,
reads better than the first version, and does not explicitly name you as
the authors.  A reviewer might think it likely that the new paper was
written by Zeus \emph{et al}\onedot, but cannot make any decision based on that guess.
He or she would have to be sure that no other authors could have been
contracted to solve problem B.

\noindent
FAQ\medskip\\
{\bf Q:} Are acknowledgements OK?\\
{\bf A:} No.  Leave them for the final copy.\medskip\\
{\bf Q:} How do I cite my results reported in open challenges?
{\bf A:} To conform with the double blind review policy, you can report results of other challenge participants together with your results in your paper. For your results, however, you should not identify yourself and should not mention your participation in the challenge. Instead present your results referring to the method proposed in your paper and draw conclusions based on the experimental comparison to other results.\medskip\\

\begin{figure}[t]
\begin{center}
\fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}

\end{center}
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
\label{fig:long}
\label{fig:onecol}
\end{figure}

\subsection{Miscellaneous}

\noindent
Compare the following:\\
\begin{tabular}{ll}
 \verb'$conf_a$' &  $conf_a$ \\
 \verb'$\mathit{conf}_a$' & $\mathit{conf}_a$
\end{tabular}\\
See The \TeX book, p165.

The space after \eg, meaning ``for example'', should not be a
sentence-ending space. So \eg is correct, {\em e.g.} is not.  The provided
\verb'\eg' macro takes care of this.

When citing a multi-author paper, you may save space by using ``et alia'',
shortened to ``\emph{et al}\onedot'' (not ``{\em et.\ al.}'' as ``{\em et}'' is a complete word.)
However, use it only when there are three or more authors.  Thus, the
following is correct: ``
   Frobnication has been trendy lately.
   It was introduced by Alpher~\cite{Alpher02}, and subsequently developed by
   Alpher and Fotheringham-Smythe~\cite{Alpher03}, and Alpher \emph{et al}\onedot~\cite{Alpher04}.''

This is incorrect: ``... subsequently developed by Alpher \emph{et al}\onedot~\cite{Alpher03} ...''
because reference~\cite{Alpher03} has just two authors.  If you use the
\verb'\emph{et al}\onedot' macro provided, then you need not worry about double periods
when used at the end of a sentence as in Alpher \emph{et al}\onedot.

For this citation style, keep multiple citations in numerical (not
chronological) order, so prefer \cite{Alpher03,Alpher02,Authors14} to
\cite{Alpher02,Alpher03,Authors14}.

\begin{figure*}
\begin{center}
\fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
\end{center}
   \caption{Example of a short caption, which should be centered.}
\label{fig:short}
\end{figure*}

\section{Formatting your paper}

All text must be in a two-column format. The total allowable width of the
text area is $6\frac78$ inches (17.5 cm) wide by $8\frac78$ inches (22.54
cm) high. Columns are to be $3\frac14$ inches (8.25 cm) wide, with a
$\frac{5}{16}$ inch (0.8 cm) space between them. The main title (on the
first page) should begin 1.0 inch (2.54 cm) from the top edge of the
page. The second and following pages should begin 1.0 inch (2.54 cm) from
the top edge. On all pages, the bottom margin should be 1-1/8 inches (2.86
cm) from the bottom edge of the page for $8.5 \times 11$-inch paper; for A4
paper, approximately 1-5/8 inches (4.13 cm) from the bottom edge of the
page.

\subsection{Margins and page numbering}

All printed material, including text, illustrations, and charts, must be kept
within a print area 6-7/8 inches (17.5 cm) wide by 8-7/8 inches (22.54 cm)
high.
Page numbers should be in footer with page numbers, centered and .75
inches from the bottom of the page and make it start at the correct page
number rather than the 4321 in the example.  To do this fine the line (around
line 23)
\begin{verbatim}
\setcounter{page}{4321}
\end{verbatim}
where the number 4321 is your assigned starting page.

Make sure the first page is numbered by commenting out the first page being
empty on line 46
\begin{verbatim}
\end{verbatim}

\subsection{Type-style and fonts}

Wherever Times is specified, Times Roman may also be used. If neither is
available on your word processor, please use the font closest in
appearance to Times to which you have access.

MAIN TITLE. Center the title 1-3/8 inches (3.49 cm) from the top edge of
the first page. The title should be in Times 14-point, boldface type.
Capitalize the first letter of nouns, pronouns, verbs, adjectives, and
adverbs; do not capitalize articles, coordinate conjunctions, or
prepositions (unless the title begins with such a word). Leave two blank
lines after the title.

AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title
and printed in Times 12-point, non-boldface type. This information is to
be followed by two blank lines.

The ABSTRACT and MAIN TEXT are to be in a two-column format.

MAIN TEXT. Type main text in 10-point Times, single-spaced. Do NOT use
double-spacing. All paragraphs should be indented 1 pica (approx. 1/6
inch or 0.422 cm). Make sure your text is fully justified---that is,
flush left and flush right. Please do not place any additional blank
lines between paragraphs.

Figure and table captions should be 9-point Roman type as in
Figures~\ref{fig:onecol} and~\ref{fig:short}.  Short captions should be centred.

\noindent Callouts should be 9-point Helvetica, non-boldface type.
Initially capitalize only the first word of section titles and first-,
second-, and third-order headings.

FIRST-ORDER HEADINGS. (For example, {\large \bf 1. Introduction})
should be Times 12-point boldface, initially capitalized, flush left,
with one blank line before, and one blank line after.

SECOND-ORDER HEADINGS. (For example, { \bf 1.1. Database elements})
should be Times 11-point boldface, initially capitalized, flush left,
with one blank line before, and one after. If you require a third-order
heading (we discourage it), use 10-point Times, boldface, initially
capitalized, flush left, preceded by one blank line, followed by a period
and your text on the same line.

\subsection{Footnotes}

Please use footnotes\footnote {This is what a footnote looks like.  It
often distracts the reader from the main flow of the argument.} sparingly.
Indeed, try to avoid footnotes altogether and include necessary peripheral
observations in
the text (within parentheses, if you prefer, as in this sentence).  If you
wish to use a footnote, place it at the bottom of the column on the page on
which it is referenced. Use Times 8-point type, single-spaced.

\subsection{References}

List and number all bibliographical references in 9-point Times,
single-spaced, at the end of your paper. When referenced in the text,
enclose the citation number in square brackets, for
example~\cite{Authors14}.  Where appropriate, include the name(s) of
editors of referenced books.

\begin{table}
\begin{center}
\begin{tabular}{|l|c|}
\hline
Method & Frobnability \\
\hline\hline
Theirs & Frumpy \\
Yours & Frobbly \\
Ours & Makes one's heart Frob\\
\hline
\end{tabular}
\end{center}
\caption{Results.   Ours is better.}
\end{table}

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.  Please ensure that any point you wish to
make is resolvable in a printed copy of the paper.  Resize fonts in figures
to match the font in the body text, and choose line widths which render
effectively in print.  Many readers (and reviewers), even of an electronic
copy, will choose to print your paper in order to read it.  You cannot
insist that they do otherwise, and therefore must not assume that they can
zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it's almost always best to use
\verb+\includegraphics+, and to specify the  figure width as a multiple of
the line width as in the example below
{\small\begin{verbatim}
   \usepackage[dvips]{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.eps}
\end{verbatim}
}

\subsection{Color}

Please refer to the author guidelines on the ICCV 2019 web page for a discussion
of the use of color in your document.

\section{Final copy}

You must include your signed IEEE copyright release form when you submit
your finished paper. We MUST have this form before your paper can be
published in the proceedings.

{\small
\bibliographystyle{ieee}

\section{Introduction}

Please follow the steps outlined below when submitting your manuscript to
the IEEE Computer Society Press.  This style guide now has several
important modifications (for example, you are no longer warned against the
use of sticky tape to attach your artwork to the paper), so all authors
should read this new version.

\subsection{Language}

All manuscripts must be in English.

\subsection{Dual submission}

Please refer to the author guidelines on the ICCV 2019 web page for a
discussion of the policy on dual submissions.

\subsection{Paper length}
Papers, excluding the references section,
must be no longer than eight pages in length. The references section
will not be included in the page count, and there is no limit on the
length of the references section. For example, a paper of eight pages
with two pages of references would have a total length of 10 pages.

Overlength papers will simply not be reviewed.  This includes papers
where the margins and formatting are deemed to have been significantly
altered from those laid down by this style guide.  Note that this
\LaTeX\ guide already sets figure captions and references in a smaller font.
The reason such papers will not be reviewed is that there is no provision for
supervised revisions of manuscripts.  The reviewing process cannot determine
the suitability of the paper for presentation in eight pages if it is
reviewed in eleven.

\subsection{The ruler}
The \LaTeX\ style defines a printed ruler which should be present in the
version submitted for review.  The ruler is provided in order that
reviewers may comment on particular lines in the paper without
circumlocution.  If you are preparing a document using a non-\LaTeX\
document preparation system, please arrange for an equivalent ruler to
appear on the final output pages.  The presence or absence of the ruler
should not change the appearance of any other content on the page.  The
camera ready copy should not contain a ruler. (\LaTeX\ users may uncomment
the \verb'\iccvfinalcopy' command in the document preamble.)  Reviewers:
note that the ruler measurements do not align well with lines in the paper
--- this turns out to be very difficult to do well when the paper contains
many figures and equations, and, when done, looks ugly.  Just use fractional
references (e.g.\ this line is $095.5$), although in most cases one would
expect that the approximate location will be adequate.

\subsection{Mathematics}

Please number all of your sections and displayed equations.  It is
important for readers to be able to refer to any particular equation.  Just
because you didn't refer to it in the text doesn't mean some future reader
might not need to refer to it.  It is cumbersome to have to use
circumlocutions like ``the equation second from the top of page 3 column
1''.  (Note that the ruler will not be present in the final copy, so is not
an alternative to equation numbers).  All authors will benefit from reading
Mermin's description of how to write mathematics:
\url{http://www.pamitc.org/documents/mermin.pdf}.

\subsection{Blind review}

Many authors misunderstand the concept of anonymizing for blind
review.  Blind review does not mean that one must remove
citations to one's own work---in fact it is often impossible to
review a paper unless the previous citations are known and
available.

Blind review means that you do not use the words ``my'' or ``our''
when citing previous work.  That is all.  (But see below for
techreports.)

Saying ``this builds on the work of Lucy Smith [1]'' does not say
that you are Lucy Smith; it says that you are building on her
work.  If you are Smith and Jones, do not say ``as we show in
[7]'', say ``as Smith and Jones show in [7]'' and at the end of the
paper, include reference 7 as you would any other cited work.

An example of a bad paper just asking to be rejected:
\begin{quote}
\begin{center}
    An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of our
   previous paper [1], and show it to be inferior to all
   previously known methods.  Why the previous paper was
   accepted without this analysis is beyond me.

   [1] Removed for blind review
\end{quote}

An example of an acceptable paper:

\begin{quote}
\begin{center}
     An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of the
   paper of Smith \emph{et al}\onedot [1], and show it to be inferior to
   all previously known methods.  Why the previous paper
   was accepted without this analysis is beyond me.

   [1] Smith, L and Jones, C. ``The frobnicatable foo
   filter, a fundamental contribution to human knowledge''.
   Nature 381(12), 1-213.
\end{quote}

If you are making a submission to another conference at the same time,
which covers similar or overlapping material, you may need to refer to that
submission in order to explain the differences, just as you would if you
had previously published related work.  In such cases, include the
anonymized parallel submission~\cite{Authors14} as additional material and
cite it as
\begin{quote}
[1] Authors. ``The frobnicatable foo filter'', F\&G 2014 Submission ID 324,
Supplied as additional material {\tt fg324.pdf}.
\end{quote}

Finally, you may feel you need to tell the reader that more details can be
found elsewhere, and refer them to a technical report.  For conference
submissions, the paper must stand on its own, and not {\em require} the
reviewer to go to a techreport for further details.  Thus, you may say in
the body of the paper ``further details may be found
in~\cite{Authors14b}''.  Then submit the techreport as additional material.
Again, you may not assume the reviewers will read this material.

Sometimes your paper is about a problem which you tested using a tool which
is widely known to be restricted to a single institution.  For example,
let's say it's 1969, you have solved a key problem on the Apollo lander,
and you believe that the ICCV70 audience would like to hear about your
solution.  The work is a development of your celebrated 1968 paper entitled
``Zero-g frobnication: How being the only people in the world with access to
the Apollo lander source code makes us a wow at parties'', by Zeus \emph{et al}\onedot.

You can handle this paper like any other.  Don't write ``We show how to
improve our previous work [Anonymous, 1968].  This time we tested the
algorithm on a lunar lander [name of lander removed for blind review]''.
That would be silly, and would immediately identify the authors. Instead
write the following:
\begin{quotation}
\noindent
   We describe a system for zero-g frobnication.  This
   system is new because it handles the following cases:
   A, B.  Previous systems [Zeus et al. 1968] didn't
   handle case B properly.  Ours handles it by including
   a foo term in the bar integral.

   ...

   The proposed system was integrated with the Apollo
   lunar lander, and went all the way to the moon, don't
   you know.  It displayed the following behaviours
   which show how well we solved cases A and B: ...
\end{quotation}
As you can see, the above text follows standard scientific convention,
reads better than the first version, and does not explicitly name you as
the authors.  A reviewer might think it likely that the new paper was
written by Zeus \emph{et al}\onedot, but cannot make any decision based on that guess.
He or she would have to be sure that no other authors could have been
contracted to solve problem B.

\noindent
FAQ\medskip\\
{\bf Q:} Are acknowledgements OK?\\
{\bf A:} No.  Leave them for the final copy.\medskip\\
{\bf Q:} How do I cite my results reported in open challenges?
{\bf A:} To conform with the double blind review policy, you can report results of other challenge participants together with your results in your paper. For your results, however, you should not identify yourself and should not mention your participation in the challenge. Instead present your results referring to the method proposed in your paper and draw conclusions based on the experimental comparison to other results.\medskip\\

\begin{figure}[t]
\begin{center}
\fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}

\end{center}
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
\label{fig:long}
\label{fig:onecol}
\end{figure}

\subsection{Miscellaneous}

\noindent
Compare the following:\\
\begin{tabular}{ll}
 \verb'$conf_a$' &  $conf_a$ \\
 \verb'$\mathit{conf}_a$' & $\mathit{conf}_a$
\end{tabular}\\
See The \TeX book, p165.

The space after \eg, meaning ``for example'', should not be a
sentence-ending space. So \eg is correct, {\em e.g.} is not.  The provided
\verb'\eg' macro takes care of this.

When citing a multi-author paper, you may save space by using ``et alia'',
shortened to ``\emph{et al}\onedot'' (not ``{\em et.\ al.}'' as ``{\em et}'' is a complete word.)
However, use it only when there are three or more authors.  Thus, the
following is correct: ``
   Frobnication has been trendy lately.
   It was introduced by Alpher~\cite{Alpher02}, and subsequently developed by
   Alpher and Fotheringham-Smythe~\cite{Alpher03}, and Alpher \emph{et al}\onedot~\cite{Alpher04}.''

This is incorrect: ``... subsequently developed by Alpher \emph{et al}\onedot~\cite{Alpher03} ...''
because reference~\cite{Alpher03} has just two authors.  If you use the
\verb'\emph{et al}\onedot' macro provided, then you need not worry about double periods
when used at the end of a sentence as in Alpher \emph{et al}\onedot.

For this citation style, keep multiple citations in numerical (not
chronological) order, so prefer \cite{Alpher03,Alpher02,Authors14} to
\cite{Alpher02,Alpher03,Authors14}.

\begin{figure*}
\begin{center}
\fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
\end{center}
   \caption{Example of a short caption, which should be centered.}
\label{fig:short}
\end{figure*}

\section{Formatting your paper}

All text must be in a two-column format. The total allowable width of the
text area is $6\frac78$ inches (17.5 cm) wide by $8\frac78$ inches (22.54
cm) high. Columns are to be $3\frac14$ inches (8.25 cm) wide, with a
$\frac{5}{16}$ inch (0.8 cm) space between them. The main title (on the
first page) should begin 1.0 inch (2.54 cm) from the top edge of the
page. The second and following pages should begin 1.0 inch (2.54 cm) from
the top edge. On all pages, the bottom margin should be 1-1/8 inches (2.86
cm) from the bottom edge of the page for $8.5 \times 11$-inch paper; for A4
paper, approximately 1-5/8 inches (4.13 cm) from the bottom edge of the
page.

\subsection{Margins and page numbering}

All printed material, including text, illustrations, and charts, must be kept
within a print area 6-7/8 inches (17.5 cm) wide by 8-7/8 inches (22.54 cm)
high.

\subsection{Type-style and fonts}

Wherever Times is specified, Times Roman may also be used. If neither is
available on your word processor, please use the font closest in
appearance to Times to which you have access.

MAIN TITLE. Center the title 1-3/8 inches (3.49 cm) from the top edge of
the first page. The title should be in Times 14-point, boldface type.
Capitalize the first letter of nouns, pronouns, verbs, adjectives, and
adverbs; do not capitalize articles, coordinate conjunctions, or
prepositions (unless the title begins with such a word). Leave two blank
lines after the title.

AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title
and printed in Times 12-point, non-boldface type. This information is to
be followed by two blank lines.

The ABSTRACT and MAIN TEXT are to be in a two-column format.

MAIN TEXT. Type main text in 10-point Times, single-spaced. Do NOT use
double-spacing. All paragraphs should be indented 1 pica (approx. 1/6
inch or 0.422 cm). Make sure your text is fully justified---that is,
flush left and flush right. Please do not place any additional blank
lines between paragraphs.

Figure and table captions should be 9-point Roman type as in
Figures~\ref{fig:onecol} and~\ref{fig:short}.  Short captions should be centred.

\noindent Callouts should be 9-point Helvetica, non-boldface type.
Initially capitalize only the first word of section titles and first-,
second-, and third-order headings.

FIRST-ORDER HEADINGS. (For example, {\large \bf 1. Introduction})
should be Times 12-point boldface, initially capitalized, flush left,
with one blank line before, and one blank line after.

SECOND-ORDER HEADINGS. (For example, { \bf 1.1. Database elements})
should be Times 11-point boldface, initially capitalized, flush left,
with one blank line before, and one after. If you require a third-order
heading (we discourage it), use 10-point Times, boldface, initially
capitalized, flush left, preceded by one blank line, followed by a period
and your text on the same line.

\subsection{Footnotes}

Please use footnotes\footnote {This is what a footnote looks like.  It
often distracts the reader from the main flow of the argument.} sparingly.
Indeed, try to avoid footnotes altogether and include necessary peripheral
observations in
the text (within parentheses, if you prefer, as in this sentence).  If you
wish to use a footnote, place it at the bottom of the column on the page on
which it is referenced. Use Times 8-point type, single-spaced.

\subsection{References}

List and number all bibliographical references in 9-point Times,
single-spaced, at the end of your paper. When referenced in the text,
enclose the citation number in square brackets, for
example~\cite{Authors14}.  Where appropriate, include the name(s) of
editors of referenced books.

\begin{table}
\begin{center}
\begin{tabular}{|l|c|}
\hline
Method & Frobnability \\
\hline\hline
Theirs & Frumpy \\
Yours & Frobbly \\
Ours & Makes one's heart Frob\\
\hline
\end{tabular}
\end{center}
\caption{Results.   Ours is better.}
\end{table}

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.  Please ensure that any point you wish to
make is resolvable in a printed copy of the paper.  Resize fonts in figures
to match the font in the body text, and choose line widths which render
effectively in print.  Many readers (and reviewers), even of an electronic
copy, will choose to print your paper in order to read it.  You cannot
insist that they do otherwise, and therefore must not assume that they can
zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it's almost always best to use
\verb+\includegraphics+, and to specify the  figure width as a multiple of
the line width as in the example below
{\small\begin{verbatim}
   \usepackage[dvips]{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.eps}
\end{verbatim}
}

\subsection{Color}

Please refer to the author guidelines on the ICCV 2019 web page for a discussion
of the use of color in your document.

\section{Final copy}

You must include your signed IEEE copyright release form when you submit
your finished paper. We MUST have this form before your paper can be
published in the proceedings.

{\small
\bibliographystyle{ieee}

\subsection{Evaluating unsupervised features}
We evaluate the quality of the features extracted from a convnet trained with DeeperCluster\xspace on YFCC100M by considering several standard transfer learning tasks, namely image classification, object detection and scene classification.

\begin{table}[t!]
  \centering
  \setlength{\tabcolsep}{1pt}
  \begin{tabular}{@{}lc@{}c@{}c@{\hspace{0.2em}}c@{}c@{}c@{}c@{}}
    \toprule
	  & &\phantom{ee}& \multicolumn{2}{c}{Classif.} &\phantom{ee}& \multicolumn{2}{c}{Detect.} \\
                      \cmidrule{4-5} \cmidrule{7-8}
    Method & Data && \textsc{fc68} & \textsc{all} && \textsc{fc68} & \textsc{all} \\
    \midrule
    ImageNet labels                                 & INet        && $89.3^{\phantom{\dagger}}$ & $89.2^{\phantom{\dagger}}$  && $66.3^{\phantom{\dagger}}$  & $70.3^{\phantom{\dagger}}$ \\
	  Random                                          & --          && $10.1^{\phantom{\dagger}}$  & $49.6^{\phantom{\dagger}}$  && $\phantom{0}5.4^{\phantom{\dagger}}$  & $55.6^{\phantom{\dagger}}$ \\

    \midrule
	  \multicolumn{6}{l}{\textit{Unsupervised on curated data}}
    \vspace{0.3em} \\

	  Larsson~\emph{et al}\onedot~\cite{larsson2017colorization}    & INet+Pl.      && -- & $77.2^\dagger$    && $49.2^{\phantom{\dagger}}$ & $59.7^{\phantom{\dagger}}$ \\
	  Wu~\emph{et al}\onedot~\cite{wu2018unsupervised}              & INet          && -- & --       && -- & $60.5^\dagger$ \\
	  Doersh~\emph{et al}\onedot~\cite{doersch2015unsupervised}     & INet          && $54.6^{\phantom{\dagger}}$   & $78.5^{\phantom{\dagger}}$         && $38.0^{{\phantom{\dagger}}}$ & $62.7^{{\phantom{\dagger}}}$ \\
	  Caron~\emph{et al}\onedot~\cite{caron2018deep}                & INet          && $78.5^{\phantom{\dagger}}$ & $82.5^{\phantom{\dagger}}$ && $58.7^{\phantom{\dagger}}$  & $65.9^\dagger$ \\
    \midrule
	  \multicolumn{6}{l}{\textit{Unsupervised on non-curated data}}
    \vspace{0.3em} \\

	  Mahendran~\emph{et al}\onedot~\cite{mahendran2018cross}       & YFCCv   && -- & $76.4^\dagger$  && -- & -- \\
	  Wang and Gupta~\cite{wang2015unsupervised}      & YT8M        && -- & --        && -- & $60.2^\dagger$ \\
	  Wang~\emph{et al}\onedot~\cite{wang2017transitive}            & YT9M        && $59.4^{\phantom{\dagger}}$   & $79.6^{\phantom{\dagger}}$          && $40.9^{\phantom{\dagger}}$ & $63.2^\dagger$ \\
    \midrule
    DeeperCluster\xspace                                           & YFCC         && $\mathbf{79.7}^{\phantom{\dagger}}$ & $\mathbf{84.3}^{\phantom{\dagger}}$ && $\mathbf{60.5}^{\phantom{\dagger}}$ & $\mathbf{67.8}^{\phantom{\dagger}}$ \\
    \bottomrule
  \end{tabular}
  \caption{
Comparison of DeeperCluster\xspace to state-of-the-art unsupervised feature learning on classification and detection on \textsc{Pascal} VOC $2007$.
We disassociate methods using curated datasets and methods using non-curated datasets.
We selected hyper-parameters for each transfer task on the validation set, and then retrain on both training and validation sets.
We report results on the test set averaged over $5$ runs.
``YFFCv'' stands for the videos contained in YFFC100M dataset.
$^\dagger$ numbers from their original paper.
}
  \label{tab:voc}
\end{table}

\paragraph{Pascal VOC 2007~\cite{everingham2010pascal}.}
This dataset has small training and validation sets ($2.5$k images each), making it close to the setting of real applications where models trained using large computational resources are adapted to a new task with a small number of instances.
We report numbers on the classification and detection tasks with finetuning (``\textsc{all}'') or by only retraining the last three fully connected layers of the network (``\textsc{fc68}'').
The \textsc{fc68} setting gives a better measure of the quality of the evaluated features since fewer parameters are retrained.
For classification, we use the code of Caron~\emph{et al}\onedot~\cite{caron2018deep}\footnote{\scriptsize\url{github.com/facebookresearch/deepcluster}} and for detection, \texttt{fast-rcnn}~\cite{girshick2015fast}\footnote{\scriptsize\url{github.com/rbgirshick/py-faster-rcnn}}.
For classification, we train the models for $150k$ iterations, starting with a learning rate of $0.002$ decayed by a factor $10$ every $20k$ iterations, and we report results averaged over $10$ random crops.
For object detection, we train our network for $150k$ iterations, dividing the step-size by $10$ after the first $50k$ steps with an initial learning rate of $0.01$ (\textsc{fc68}) or $0.002$ (\textsc{all}) and a weight decay of $0.0001$.
Following Doersch~\emph{et al}\onedot~\cite{doersch2015unsupervised}, we use the multiscale configuration, with scales $[400,500,600,700]$ for training and $[400,500,600]$ for testing.
In Table~\ref{tab:voc}, we compare DeeperCluster\xspace with two sets of unsupervised methods that use a VGG-16 network: those trained on curated datasets and those trained on non-curated datasets.
Previous unsupervised methods that worked on unucurated datasets with a VGG-16 use videos: Youtube8M (``YT8M''), Youtube9M (``YT9M'') or the videos from YFCC100M (``YFFCv'').
Our approach achieves state-of-the-art performance among all the unsupervised method that uses a VGG-16 architecture, even those that use ImageNet as a training set.
The gap with a supervised network is still important when we freeze the convolutions ($6\%$ for detection and $10\%$ for classification) but drops to less than $5\%$ for both tasks with finetuning.

\paragraph{Linear classifiers on ImageNet~\cite{deng2009imagenet} and Places205~\cite{zhou2014learning}.}
ImageNet (``INet'') and Places205 (``Pl.'') are two large scale image classification datasets: ImageNet's domain covers objects and animals ($1.3$M images) and Places205's domain covers indoor and outdoor scenes ($2.5$M images).
We train linear classifiers with a logistic loss on top of frozen convolutional layers at different depths.
To reduce influence of feature dimension in the comparison, we average-pool the features until their dimension is below $10k$~\cite{zhang2016colorful}.
This experiment probes the quality of the features extracted at each convolutional layer.
In Figure~\ref{fig:layers}, we observe that DeeperCluster\xspace matches the performance of a supervised network for all layers on Places205.
On ImageNet, it also matches supervised features up to the $4$th convolutional block; then the gap suddenly increases to around $20\%$.
It is not surprising since the supervised features are trained on ImageNet itself, while ours are trained on YFCC100M.

\begin{figure}[t]
	\centering
  \includegraphics[width=0.48\linewidth]{figure_linear.pdf}
  \includegraphics[width=0.48\linewidth]{figure_linear_places.pdf}
	\caption{
Accuracy of linear classifiers on ImageNet and Places205 using the activations from different layers as features.
We compare a VGG-16 trained with supervision on ImageNet to VGG-16 trained with either RotNet or DeeperCluster\xspace on YFCC100M.
Exact numbers are in Appendix.
	}
	\label{fig:layers}
\end{figure}

\subsection{Pre-training for ImageNet}
In the previous section, we can observe that a VGG-16 trained on YFCC100M has similar or better low level features than the same network trained on ImageNet with supervision.
In this experiment, we want to check whether these low-level features pre-trained on YFCC100M without supervision can serve as a good initialization for fully-supervised ImageNet classification.
To this end, we pre-train a VGG-16 on YFCC100M using either DeeperCluster\xspace or RotNet.
The resulting weights are then used as initialization for the training of a network on ImageNet with supervision.
We merge the Sobel weights of the network pre-trained with DeeperCluster\xspace with the first convolutional layer during the initialization.
We then train the networks on ImageNet with mini-batch SGD for $100$ epochs, a learning rate of $0.1$, a weight decay of $0.0001$, a batch size of $256$ and dropout of $0.5$.
We reduce the learning rate by a factor of $0.2$ every $20$ epochs.
Note that this learning rate decay schedule slightly differs from the ImageNet classification PyTorch default implementation\footnote{\scriptsize\url{github.com/pytorch/examples/blob/master/imagenet/}} where they train for $90$ epochs and decay the learning rate by $0.1$ at epochs $30$ and $60$.
We give in Appendix the results with this default schedule (with unchanged conclusions).
In Table~\ref{tab:pretrain}, we compare the performance of a network trained with a standard intialization (``Supervised'') to one initialized with a pre-training obtained from either DeeperCluster\xspace (``Supervised + DeeperCluster\xspace pre-training'') or RotNet (``Supervised + RotNet pre-training'') on YFCC100M.
We see that our pre-training improves the performance of a supervised network by $+0.8\%$, leading to $74.9\%$ top-1 accuracy.
This means that our pre-training captures important statistics from YFCC100M that transfers well to ImageNet.

\begin{table}[t!]
\centering
  \setlength{\tabcolsep}{3.5pt}
    \begin{tabular}{@{}l c cc @{}}
      \toprule
      ImageNet &~~~&top-$1$ & top-$5$ \\
      \midrule
      Supervised (PyTorch documentation\footnote{\scriptsize\url{pytorch.org/docs/stable/torchvision/models}}) && $73.4$ & $91.5$ \\
      Supervised (our code)  && $74.1$ & $91.8$ \\
      Supervised + RotNet pre-training && $74.5$ & $92.0$ \\
      Supervised + DeeperCluster\xspace pre-training && $\mathbf{74.9}$ & $\mathbf{92.3}$ \\
      \bottomrule
    \end{tabular}
    \caption{
      Accuracy on the validation set of ImageNet classification for a supervised VGG-16 trained with different initializations:
      we compare a network trained from a standard initialization to networks trained from pre-trained weights using either DeeperCluster\xspace or RotNet on YFCC100M.
    }
    \label{tab:pretrain}
\end{table}

\subsection{Model analysis}
In this final set of experiments, we analyze some components of our model.
Since DeeperCluster\xspace derives from RotNet and DeepCluster,
we first look at the difference between these methods and ours, when trained on curated and non-curated datasets.
We then report quantitative and qualitative evaluations of the clusters obtained with DeeperCluster\xspace.

\paragraph{Comparison with RotNet and DeepCluster.}
In Table~\ref{tab:linear}, we compare DeeperCluster\xspace with DeepCluster and RotNet when a linear classifier is trained on top of the last convolutional layer of a VGG-16 on several datasets.
For reference, we also report previously published numbers~\cite{wu2018unsupervised} with a VGG-$16$ architecture.
We average-pool the features of the last layer resulting in representations of $8192$ dimensions.
Our approach outperforms both RotNet and DeepCluster, even when they are trained on curated datasets (except for ImageNet classification task where DeepCluster trained on ImageNet yields the best performance).
More interestingly, we see that the quality of the dataset or its scale has little impact on RotNet while it has on DeepCluster.
This is confirming that self-supervised methods are more robust than clustering to a change of dataset distribution.

\begin{table}[t!]
\centering
  \setlength{\tabcolsep}{3.5pt}
    \begin{tabular}{@{}l c c c c@{}}
      \toprule
      Method & Data & ImageNet & Places & VOC2007 \\
      \midrule
      Supervised  & ImageNet  & $70.2$ & $45.9$  & $84.8$ \\
      \midrule
      Wu~\emph{et al}\onedot~\cite{wu2018unsupervised} & ImageNet & $39.2$ & $36.3$ & - \\
      \midrule
      RotNet      & ImageNet & $32.7$ & $32.6$ &  $60.9$ \\
      DeepCluster & ImageNet & $\mathbf{48.4}$ & $37.9$ &  $71.9$ \\
      \midrule
      RotNet      & YFCC100M & $33.0$ & $35.5$ &  $62.2$ \\
      DeepCluster & YFCC100M & $34.1$ & $35.4$ &  $63.9$ \\
      \midrule
      DeeperCluster\xspace & YFCC100M & $45.6$ & $\mathbf{42.1}$  &  $\mathbf{73.0}$ \\
      \bottomrule
    \end{tabular}
    \caption{
      Comparaison between DeeperCluster\xspace, RotNet and DeepCluster when pre-trained on curated and non-curated dataset.
      We report the accuracy on several datasets of a linear classifier trained on top of features of the last convolutional layer.
      All the methods use the same architecture.
      DeepCluster does not scale to the full YFCC100M dataset, we thus train it on a random subset of $1.3$M images.
    }
    \label{tab:linear}
\end{table}

\begin{figure*}[t]
	\centering
    \includegraphics[height=0.19\linewidth]{figure_nmi_tag.pdf}
    \includegraphics[height=0.19\linewidth]{figure_nmi_user.pdf}
    \includegraphics[height=0.19\linewidth]{figure_nmi_gps.pdf}
    \includegraphics[height=0.19\linewidth]{figure_nmi_camera.pdf}
    \includegraphics[height=0.19\linewidth]{figure_nmi_imnet.pdf}
  \caption{Normalized mutual information between our clustering and different sorts of metadata: hashtags, user IDs, geographic coordinates, and device types.
   We also plot the NMI with an ImageNet classifier labeling.}
  \label{fig:nmi}
\end{figure*}

\begin{figure*}[t!]
  \centering
  \begin{tabular}{ccccccc}
	  {tag: \itshape cat} & {tag: \itshape elephantparadelondon} & {tag: \itshape always} & {device: \itshape CanoScan}
  \\
  \includegraphics[width=0.225\linewidth]{cluster21642.jpeg}&
  \includegraphics[width=0.225\linewidth]{cluster28828.jpeg}&
  \includegraphics[width=0.225\linewidth]{cluster3551.jpeg}&
  \includegraphics[width=0.225\linewidth]{cluster5841.jpeg}
  \\
  {GPS: ($43$, $10$)} & {GPS: ($-34$, $-151$)} & {GPS: ($64$, $-20$)} & {GPS: ($43$, $-104$)}
  \\
  \includegraphics[width=0.225\linewidth]{cluster30986.jpeg}&
  \includegraphics[width=0.225\linewidth]{cluster380.jpeg}&
  \includegraphics[width=0.225\linewidth]{cluster22739.jpeg}&
  \includegraphics[width=0.225\linewidth]{cluster16433.jpeg}
  \end{tabular}
  \caption{We randomly select $9$ images per cluster and indicate the dominant cluster metadata.
The bottom row depicts clusters pure for GPS coordinates but unpure for user IDs.
As expected, they turn out to correlate with tourist landmarks.
No metadata is used during training.
For copyright reasons, we provide in Appendix the photographer username for each image.
  }
  \label{fig:cluster}
\end{figure*}

\paragraph{Influence of dataset size and number of clusters.}
\label{sec:exp-k}
To measure the influence of the number of images on features, we train models with $1$M, $4$M, $20$M, and $96$M images and report their accuracy on the validation set of the Pascal VOC 2007 classification task (\textsc{fc68} setting).
We also train models on $20$M images with a number of clusters that varies from $10$k to $160$k.
For the experiment with a total of $160$k clusters, we choose $m=2$ which results in $8$ super-classes.
In Figure~\ref{fig:k}, we observe that the quality of our features improves when scaling both in terms of images and clusters.
Interestingly, between $4$M and $20$M of YFCC100M images are needed to meet the performance of our method on ImageNet.
Augmenting the number of images has a bigger impact than the number of clusters.
Yet, this improvement is significant since it corresponds to a reduction of more than $10\%$ of the relative error w.r.t. the supervised model.

\paragraph{Quality of the clusters.}
In addition to features, our method provides a clustering of the input images.
We evaluate the quality of these clusters by measuring their correlation with existing partitions of the data.
In particular, YFCC100M comes with many different metadata.
We consider hashtags, users, camera and GPS coordinates.
If an image has several hashtags, we pick as label the least frequent one in the total hashtag distribution.
We also measure the correlation of ours clusters with labels predicted by a classifier trained on ImageNet categories.
We use a ResNet-$50$ network~\cite{he2016deep}, pre-trained on ImageNet, to classify the YFCC100M images and we select those for which the confidence in prediction is higher than $75\%$.
This evaluation omits a large amount of the data but gives some insight about the quality of our clustering in object classification.

In Figure~\ref{fig:nmi}, we show the evolution during training of the normalized mutual information (NMI) between our clustering and different metadata, and the predicted labels from ImageNet.
The higher the NMI, the more correlated our clusters are to the considered partition.
For reference, we compute the NMI for a clustering of RotNet features (as it corresponds to weights at initialization) and of a supervised model.
First, it is interesting to observe that our clustering is improving over time for every type of metadata.
One important factor is that most of these commodities are correlated since a given user takes pictures in specific places with probably a single camera and use a preferred fixed set of hashtags.
Yet, these plots show that our model captures in the input signal enough information to predict these metadata at least as well as the features trained with supervision.

We visually assess the consistency of our clusters in Figure~\ref{fig:cluster}.
We display $9$ random images from $8$ manually picked clusters.
The first two clusters contain a majority of images associated with tag from the head (first cluster) and from the tail (second cluster) in the YFC100M dataset.
Indeed, $418.538$ YFC100M images are associated with the tag \textit{cat} whereas only $384$ images contain the tag \textit{elephantparadelondon} ($0.0004\%$ of the dataset).
We also show a cluster for which the dominant hashtag does not corrolate visually with the content of the cluster.
As already mentioned, this database is non-curated and contains images that basically do not depict anything semantic.
The dominant metadata of the last cluster in the top row is the device ID \textit{CanoScan}.
As this cluster is about drawings, its images have been mainly taken with a scanner.
Finally, the bottom row depict clusters that are pure for GPS coordinates but unpure for user IDs.
It results in clusters of images taken by many different users in the same place: tourist landmarks.

\section{Introduction}
\input{intro.tex}

\section{Related Work}
\input{related.tex}

\input{method.tex}

\section{Experiments}
\input{experiments.tex}

\section{Conclusion}
In this paper, we present an unsupervised approach specifically designed to deal with large amount of non-curated data.
Our method is well-suited for distributed training, which allows training on large datasets with $96M$ of images.
With such amount of data, our approach surpasses unsupervised methods trained on curated datasets, which validates the potential of unsupervised learning in applications where annotations are scarce or curation is not trivial.
Finally, we show that unsupervised pre-training improves the performance of a network trained on ImageNet.

\paragraph{Acknowledgement.}
We thank Thomas Lucas, Matthijs Douze, Francisco Massa and the rest of Thoth and FAIR teams for their help and fruitful discussions.
We also thank the anonymous reviewers for their thoughtful feedback.
Julien Mairal was funded by the ERC grant number 714381 (SOLARIS project).

{\small
\bibliographystyle{ieee_fullname}

\section{Preliminaries}
In this work, we refer to the vector obtained at the penultimate layer of the convnet as a \emph{feature} or \emph{representation}.
We denote by $f_\theta$ the feature-extracting function, parametrized by a set of parameters $\theta$.
Given a set of images, our goal is then to learn a ``good'' mapping $f_{\theta^*}$.
By ``good'', we mean a function that produces general-purpose visual features that are useful on downstream tasks.

\subsection{Self-supervision}
In self-supervised learning, a pretext task is used to extract target labels directly from data~\cite{doersch2015unsupervised}.
These targets can take a variety of forms.
They can be categorical labels associated with a multiclass problem, as when predicting the transformation of an image~\cite{gidaris2018unsupervised,zhang2019aet} or the ordering of a set of patches~\cite{noroozi2016unsupervised}.
Or they can be continuous variables associated with a regression problem, as when predicting image color~\cite{zhang2016colorful} or surrounding patches~\cite{pathak2016context}.
In this work, we are interested in the former.
We suppose that we are given a set of $N$ images $\{x_1, \dots, x_N\}$ and we assign a pseudo-label $y_n$ in $\mathcal{Y}$ to each input $x_n$.
Given these pseudo-labels, we learn the parameters $\theta$ of the convet jointly with a linear classifier $V$ to predict pseudo-labels by solving the problem
\begin{equation}
  \label{eq:selfsup}
  \min_{\theta, V} \frac{1}{N} \sum_{n=1}^N \ell( y_n, V f_\theta(x_n)),
\end{equation}
where $\ell$ is a loss function.
The pseudo-labels $y_n$ are fixed during the optimization and the quality of the learned features entirely depends on their relevance.

\paragraph{Rotation as self-supervision.}
Gidaris~\emph{et al}\onedot~\cite{gidaris2018unsupervised} have recently shown that good features can be obtained when training a convnet to discriminate between different image rotations.
In this work, we focus on their pretext task, \textit{RotNet}, since its performance on standard evaluation benchmarks is among the best in self-supervised learning.
This pretext task corresponds to a multiclass classification problem with four categories: rotations in $\{0\degree, 90\degree, 180\degree, 270\degree\}$.
Each input $x_n$ in Eq.~(\ref{eq:selfsup}) is randomly rotated and associated with a target~$y_n$ that represents the angle of the applied rotation.

\subsection{Deep clustering}
Clustering-based approaches for deep networks typically build target classes by clustering visual features produced by convnets.
As a consequence, the targets are updated during training along with the representations and are potentially different at each epoch.
In this context, we define a latent pseudo-label $z_n$ in $\mathcal{Z}$ for each image $n$ as well as a corresponding linear classifier $W$.
These clustering-based methods alternate between learning the parameters $\theta$ and $W$ and updating the pseudo-labels $z_n$.
Between two reassignments, the pseudo-labels $z_n$ are fixed, and the parameters and classifier are optimized by solving
\begin{equation}
  \label{eq:mstep}
  \min_{\theta, W} \frac{1}{N} \sum_{n=1}^N \ell( z_n, W f_\theta(x_n)),
\end{equation}
which is of the same form as Eq.~(\ref{eq:selfsup}).
Then, the pseudo-labels $z_n$ can be reassigned by minimizing an auxiliary loss function.
This loss sometimes coincides with Eq.~(\ref{eq:mstep})~\cite{bojanowski2017unsupervised,xie2016unsupervised} but some works proposed to use another objective~\cite{caron2018deep,yang2016joint}.

\paragraph{Updating the targets with $k$-means.}
In this work, we focus on the framework of Caron~\emph{et al}\onedot~\cite{caron2018deep}, \textit{DeepCluster}, where latent targets are obtained by clustering the activations with $k$-means.
More precisely, the targets $z_n$ are updated by solving the following optimization problem:
\begin{equation}\label{eq:clustering}
  \min_{C\in\mathbb{R}^{d\times k}}  \sum_{n=1}^N \left[\min_{z_n\in \{0,1\}^k    ~\text{s.t.}~ z_n^\top {\mathbf 1} = 1} \| C z_n - f_\theta(x_n)\|_2^2 \right],
\end{equation}
$C$ is the matrix where each column corresponds to a centroid, $k$ is the number of centroids, and $z_n$ is a binary vector with a single non-zero entry.
This approach assumes that the number of clusters $k$ is known \emph{a priori}; in practice, we set it by validation on a downstream task (see Sec.~\ref{sec:exp-k}).
The latent targets are updated every $T$ epochs of stochastic gradient descent steps when minimizing the objective~(\ref{eq:mstep}).

Note that this alternate optimization scheme is prone to trivial solutions and controlling the way optimization procedures of both objectives interact is crucial.
Re-assigning empty clusters and performing a batch-sampling based on an uniform distribution over the cluster assignments are workarounds to avoid trivial parametrization~\cite{caron2018deep}.

\section{Method}
\label{sec:method}

In this section, we describe how we combine self-supervised learning with deep clustering in order to scale up to large numbers of images and targets.

\subsection{Combining self-supervision and clustering}

We assume that the inputs $x_1,\dots,x_N$ are rotated images, each associated with a target label $y_n$ encoding its rotation angle and a cluster assignment~$z_n$.
The cluster assignment changes during training along with the visual representations.
We denote by $\mathcal{Y}$ the set of possible rotation angles and by $\mathcal{Z}$, the set of possible cluster assignments.
A way of combining self-supervision with deep clustering is to add the losses defined in Eq.~(\ref{eq:selfsup}) and Eq.~(\ref{eq:mstep}).
However, summing these losses implicitly assumes that classifying rotations and cluster memberships are two independent tasks, which may limit the signal that can be captured.
Instead, we work with the Cartesian product space $\mathcal{Y} \times \mathcal{Z}$, which can potentially capture richer interactions between the two tasks.
We get the following optimization problem:
\begin{equation}
  \label{eq:naive}
  \min_{\theta, W} \frac{1}{N} \sum_{n=1}^N \ell(  y_n \otimes z_n, W f_\theta(x_n)).
\end{equation}
Note that any clustering or self-supervised approach with a multiclass objective can be combined with this formulation.
For example, we could use a self-supervision task that captures information about tiles permutations~\cite{noroozi2016unsupervised} or frame ordering in a video~\cite{wang2015unsupervised}.
However, this formulation does not scale in the number of combined targets, i.e., its complexity is $O(|\mathcal{Y}||\mathcal{Z}|)$.
This limits the use of a large number of cluster or a self-supervised task with a large output space~\cite{zhang2019aet}.
In particular, if we want to capture information contained in the tail of the distribution of non-curated dataset, we may need a large number of clusters.
We thus propose an approximation of our formulation based on a scalable hierarchical loss that it is designed to suit distributed training.

\subsection{Scaling up to large number of targets}

Hierarchical losses are commonly used in language modeling where the goal is to predict a word out of a large vocabulary~\cite{brown1992class}.
Instead of making one decision over the full vocabulary, these approaches split the process in a hierarchy of decisions, each with a smaller output space.
For example, the vocabulary can be split into clusters of semantically similar words, and the hierarchical process would first select a cluster and then a word within this cluster.

Following this line of work, we partition the target labels into a $2$-level hierarchy where we first predict a super-class and then a sub-class among its associated target labels.
The first level is a partition of the images into $S$ super-classes and we denote by $y_n$ the super-class assignment vector in $\{0,1\}^S$ of the image $n$ and by $y_{ns}$ the $s$-th entry of $y_n$.
This super-class assignment is made with a linear classifier~$V$ on top of the features.
The second-level of the hierarchy is obtained by partitioning \emph{within each super-class}.
We denote by $z^s_n$ the vector in $\{0,1\}^{k_s}$ of the assignment into~$k_s$ sub-classes for an image $n$ belonging to super-class $s$.
There are $S$ sub-class classifiers $W_1,\dots,W_S$, each predicting the sub-class memberships within a super-class $s$.
The parameters of the linear classifiers~$(V, W_1, \dots, W_S)$ and $\theta$ are jointly learned by minimizing the following loss function:
\begin{equation}\label{eq:sup}
\hspace{-5pt}\frac{1}{N} \sum_{n=1}^N \left[\ell\big(V f_\theta(x_n), y_n\big) {+}\sum_{s=1}^S y_{ns} \ell\left(W_s f_\theta(x_n) , z^s_n\right)\right],
\end{equation}
where $\ell$ is the negative log-softmax function. Note that an image that does not belong to the super-class $s$ does not belong either to any of its $k_s$ sub-classes.

\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth]{pull-figure-3.pdf}
  \caption{DeeperCluster alternates between a hierachical clustering of the features and learning the parameters of a convnet by predicting both the rotation angle and the cluster assignments in a single hierachical loss.}
   \vspace{-1em}
  \label{fig:method}
\end{figure}

\paragraph{Choice of super-classes.}
A natural partition would be to define the super-classes based on the target labels from the self-supervised task and the sub-classes as the labels produced by clustering.
However, this would mean that each image of the entire dataset would be present in each super-class (with a different rotation), which does not take advantage of the hierarchical structure to use a bigger number of clusters.

Instead, we split the dataset into $m$ sets by running $k$-means with $m$ centroids on the full dataset every $T$ epochs.
We then use the Cartesian product between the assignment to these $m$ clusters and the angle rotation classes to form the super-classes.
There are $4m$ super-classes, each associated with the subset of data belonging to the corresponding cluster ($N/m$ images if the clustering is perfectly balanced).
These subsets are then further split with $k$-means into $k$ subclasses.
This is equivalent to running a hierarchical $k$-means with rotation constraints on the full datasets to form our hierarchical loss.
We typically use $m=4$ and $k=80$k, leading to a total of $320$k different clusters split in $4$ subsets.
Our approach, ``DeeperCluster\xspace'', shares similarities with DeepCluster but is designed to scale to larger datasets.
We alternate between clustering the non-rotated images features and training the network to predict both the rotation applied to the input data and its cluster assignment amongst the clusters corresponding to this rotation (Figure~\ref{fig:method}).

\paragraph{Distributed training.}
Building the super-classes based on data splits lends itself to a distributed implementation that scales well in the number of images.
Specifically, when optimizing Eq.~(\ref{eq:sup}), we form as many distributed communication groups of $p$ GPUs as the number of super-classes, i.e., $G=4m$.
Different communication groups share the parameters $\theta$ and the super-class classifier $V$, while the parameters of the sub-class classifiers $W_1,\dots,W_S$ are only shared within a communication group.
Each communication group $s$ deals only with the subset of images and the rotation angle associated with the super-class $s$.

\paragraph{Distributed $k$-means.}
Every $T$ epochs, we recompute the super and sub-class assignments by running two consecutive $k$-means on the entire dataset.
This is achieved by first randomly splitting the dataset across different GPUs.
Each GPU is in charge of computing cluster assignments for its partition, whereas
centroids are updated across GPUs.
We reduce communication between GPUs by sharing only the number of assigned elements for each cluster and the sum of their features.
The new centroids are then computed from these statistics.
We observe empirically that $k$-means converges in $10$ iterations.
We cluster $96$M features of dimension $4096$ into $m=4$ clusters using $64$ GPUs ($1$ minute per iteration).
Then, we split this pool of GPUs into $4$ groups of $16$ GPUs.
Each group clusters around $23$M features into $80$k clusters ($4$ minutes per iteration).

\subsection{Implementation details}

The loss in Eq.~(\ref{eq:sup}) is minimized with mini-batch stochastic gradient descent~\cite{bottou2012stochastic}.
Each mini-batch contains $3072$ instances distributed accross $64$ GPUs, leading to $48$ instances per GPU per minibatch~\cite{goyal2017accurate}.
We use dropout, weight decay, momentum and a constant learning rate of $0.1$.
We reassign clusters every $3$ epochs.
We use the Pascal VOC $2007$ classification task without finetuning as a downstream task to select hyper-parameters.
In order to speed up experimentations, we initialize the network with RotNet trained on YFCC100M.
Before clustering, we perform a whitening of the activations and $\ell_2$-normalize each of them.
We use standard data augmentations, i.e., cropping of random sizes and aspect ratios and horizontal flips~\cite{krizhevsky2012imagenet}).
We use the VGG-$16$ architecture~\cite{simonyan2014very} with batch normalization layers.
Following~\cite{bojanowski2017unsupervised,caron2018deep,paulin2015local}, we pre-process images with a Sobel filtering.
We train our models on the $96$M images from YFCC100M~\cite{thomee2015yfcc100m} that we managed to download.
We use this publicly available dataset for research purposes only.

\section*{\refname}}{}{}{}

\input{template_sierra.tex}

\usepackage{geometry}
\geometry{
 a0paper,
 left=0mm,
 right=0mm,
 top=0mm,
 }
\usepackage{tikzscale}
\usepackage{booktabs}
\usepackage{algorithmic}
\usepackage{float}
\usepackage{subcaption}

\renewcommand{\baselinestretch}{1.1}
\def\emph{et al}\onedot{\emph{et al}.}

\begin{document}

\sffamily

\postertitle{Leveraging Large-Scale Uncurated Data for Unsupervised Learning of Visual Features} {Mathilde Caron, Piotr Bojanowski, Armand Joulin and Julien Mairal} {Facebook AI Research, INRIA}

\vspace{-1em}
\setlength\columnsep{-25pt}
\begin{multicols}{3}

\large

\begin{center}

\block[orangeinria][1]{Overview}
{
\begin{itemize}
	\begin{Large}
	\item \textcolor{orangeinria}{\textbf{Goal}}\\
	Learning general-purpose visual features with convnets on large-scale unsupervised and uncurated datasets.
	\item \textcolor{orangeinria}{\textbf{Motivation}}\\
	\begin{itemize}
		\item bridge the performance gap between unsupervised methods trained on curated data, which are costly to obtain, and massive raw datasets that are easily available;
		\item new unsupervised approach which leverages self-supervision and clustering to capture complementary statistics from large-scale data.
	\end{itemize}
	\item \textcolor{orangeinria}{\textbf{Method}}\\
	Our approah, iterates between:
	\begin{itemize}
		\item hierarchical clustering of the features;
		\item updating convnet weights by predicting both rotation angle and cluster assignment in a single hierarchical loss.
	\end{itemize}
	\item \textcolor{orangeinria}{\textbf{Results}}\\
	Features pre-trained on $95$M images from YFCC100M with state-of-the-art performance on standard evaluation benchmarks with VGG-$16$.
	\end{Large}
	\vspace{0.5em}
\end{itemize}
}

\block[orangeinria][1]{Illustration of our approach}
{\includegraphics[width=1.\linewidth]{figures/pull-figure-3.pdf}}

\block{Method}
{
\begin{itemize}
	\item  A large set of unlabelled images~$\{ x_1, \ldots, x_N \}$, $x_i$ in $\mathbb{R}^{3 \times 224 \times 224}$.
	\item $f_\theta$ is the convnet mapping (with $\theta$ the set of corresponding parameters).
	\item We partition the target labels into a $2$-level hierarchy:
		\begin{enumerate}
			\item \textbf{Super-classes}: $y_n$ the super-class assignment vector in $\{0,1\}^S$ of the image $n$;
			\item \textbf{Sub-classes}: partitioning \emph{within each super-class}. $z^s_n$ is the vector in $\{0,1\}^{k_s}$ of the assignment into~$k_s$ sub-classes for an image $n$ belonging to super-class $s$.
		\end{enumerate}
	\item Parameters of linear classifiers~$(V, W_1, \dots, W_S)$ and $\theta$ are learned by minimizing:
\begin{Large}
\[
\frac{1}{N} \sum_{n=1}^N \left[\ell\big(V f_\theta(x_n), y_n\big) {+}\sum_{s=1}^S y_{ns} \ell\left(W_s f_\theta(x_n) , z^s_n\right)\right],
\]
\end{Large}
where $\ell$ is the negative log-softmax function.
\end{itemize}
} \par

\block{Transfer learning to Pascal VOC 2007}{
  \begin{tabular}{@{}lc@{}c@{}c@{\hspace{0.2em}}c@{}c@{}c@{}c@{}}
	  & &\phantom{ee}& \multicolumn{2}{c}{Classif.} &\phantom{ee}& \multicolumn{2}{c}{Detect.} \\
                      \cmidrule{4-5} \cmidrule{7-8}
    Method & Data && \textsc{fc68} & \textsc{all} && \textsc{fc68} & \textsc{all} \\
    \midrule
	  ImageNet labels                                 & ImageNet        && $89.3$ & $86.9$  && $57.0$  & $67.3$ \\

    \midrule
	  \multicolumn{6}{l}{\textit{Unsupervised on curated data}}
    \vspace{0.3em} \\

	  Larsson~\emph{et al}\onedot~\cite{larsson2017colorization}    & ImageNet + Places      && -- & $77.2$    && $45.6$ & $59.7$ \\
	  Doersh~\emph{et al}\onedot~\cite{doersch2015unsupervised}     & ImageNet          && $54.6$   & $78.5$         && $38.0$ & $62.7$ \\
	  Caron~\emph{et al}\onedot~\cite{caron2018deep}                & ImageNet          && $78.5$ & $82.3$ && $\mathbf{57.1}$  & $65.9$ \\
    \midrule
	  \multicolumn{6}{l}{\textit{Unsupervised on uncurated data}}
    \vspace{0.3em} \\

	  Mahendran~\emph{et al}\onedot~\cite{mahendran2018cross}       & YFCC100M videos   && -- & $76.4$  && -- & -- \\
	  Wang and Gupta~\cite{wang2015unsupervised}      & Youtube8M        && -- & --        && -- & $60.2$ \\
	  Wang~\emph{et al}\onedot~\cite{wang2017transitive}            & Youtube9M        && $59.4$   & $79.6$          && $40.9$ & $63.2$ \\
    \midrule
    Our method                                           & YFCC100M         && $\mathbf{79.9}$ & $\mathbf{83.8}$ && $56.9$ & $\mathbf{67.5}$ \\
    \bottomrule
  \end{tabular}
}

\block{Comparing with methods on YFCC100M}{
\begin{itemize}
	\item[] We train logistic regressions on top of frozen convolutional layers at different depths.
\end{itemize}
\includegraphics[width=0.48\linewidth]{figures/figure_linear.pdf}
\includegraphics[width=0.48\linewidth]{figures/figure_linear_places.pdf}
} \par

\block{Amounts of images and clusters}
{
\begin{itemize}
	\item[] We report validation mAP on Pascal VOC classification task (\textsc{fc68} setting).
\end{itemize}
  \includegraphics[width=0.495\linewidth]{figures/figure_size_dataset.pdf}
  \includegraphics[width=0.495\linewidth]{figures/figure_k.pdf}
}

\block{Clustering quality}
{
\begin{itemize}
	\item[] We display $9$ random images for clusters pure for a certain metadata. The bottom row depicts clusters that are pure for GPS coordinates but unpure for user IDs.
\end{itemize}
{
  \begin{tabular}{ccccccc}
  {tag: \itshape cat} & {tag: \small \itshape elephantparadelondon} & {tag: \itshape always} & {device: \itshape CanoScan}
  \\
  \includegraphics[width=0.24\linewidth]{images/cluster21642.jpeg}&
  \includegraphics[width=0.24\linewidth]{images/cluster28828.jpeg}&
  \includegraphics[width=0.24\linewidth]{images/cluster3551.jpeg}&
  \includegraphics[width=0.24\linewidth]{images/cluster5841.jpeg}
  \\
  {GPS: ($43$, $10$)} & {GPS: ($-34$, $-151$)} & {GPS: ($64$, $-20$)} & {GPS: ($43$, $-104$)}
  \\
  \includegraphics[width=0.24\linewidth]{images/cluster30986.jpeg}&
  \includegraphics[width=0.24\linewidth]{images/cluster380.jpeg}&
  \includegraphics[width=0.24\linewidth]{images/cluster22739.jpeg}&
  \includegraphics[width=0.24\linewidth]{images/cluster16433.jpeg}
  \end{tabular}
  }
} \par

\block{References}
{
{\small
\bibliographystyle{ieee}

\section*{\textbf{\LARGE{Appendix}}}

\begin{table*}[h]
\centering
  \setlength{\tabcolsep}{3.5pt}
    \begin{tabular}{@{}l ccccccccccccc@{}}
      \toprule
      Method & \footnotesize \texttt{conv1} & \footnotesize \texttt{conv2} & \footnotesize \texttt{conv3} & \footnotesize \texttt{conv4} & \footnotesize \texttt{conv5} & \footnotesize \texttt{conv6} & \footnotesize \texttt{conv7} & \footnotesize \texttt{conv8} & \footnotesize \texttt{conv9} & \footnotesize \texttt{conv10} & \footnotesize \texttt{conv11} & \footnotesize \texttt{conv12} & \footnotesize \texttt{conv13} \\
      \midrule
      \textit{ImageNet} \\
      \midrule
      Supervised & $7.8 $ & $12.3$ & $15.6$ & $21.4$ & $24.4$ & $24.1$ & $33.4$ & $41.1$ & $44.7$ & $49.6$ & $61.2$ & $66.0$ & $70.2$ \\
      RotNet & $10.9$ & $15.7$ & $17.2$ & $21.0$ & $27.0$ & $26.6$ & $26.7$ & $33.5$ & $35.2$ & $33.5$ & $39.6$ & $38.2$ & $33.0$ \\
      DeeperCluster\xspace & $7.4$ & $9.6$ & $14.9$ & $16.8$ & $26.1$ & $29.2$ & $34.2$ & $41.6$ & $43.4$ & $45.5$ & $49.0$ & $49.2$ & $45.6$ \\
      \midrule
      \textit{Places205} \\
      \midrule
      Supervised & $10.5$ & $16.4$ & $20.7$ & $24.7$ & $30.3$ & $31.3$ & $35.0$ & $38.1$ & $39.5$ & $40.8$ & $45.4$ & $45.3$ & $45.9$ \\
      RotNet & $13.9$ & $19.1$ & $22.5$ & $24.8$ & $29.9$ & $30.8$ & $32.5$ & $35.3$ & $36.0$ & $36.1$ & $38.8$ & $37.9$ & $35.5$ \\
      DeeperCluster\xspace & $12.7$ & $14.8$ & $21.2$ & $23.3$ & $30.5$ & $32.6$ & $34.8$ & $39.5$ & $40.8$ & $41.6$ & $44.0$ & $44.0$ & $42.1$ \\
      \bottomrule
    \end{tabular}
    \caption{
    Accuracy of linear classifiers on ImageNet and Places205 using the activations from different layers as features.
    We train a linear classifier on top of frozen convolutional  layers at different depths.
    We compare a VGG-16 trained with supervision on ImageNet to VGG-16s trained with either RotNet or our approach on YFCC100M.
    }
    \label{tab:layers}
\end{table*}

\setcounter{section}{0}

\section{Evaluating unsupervised features}
Here we provide numbers from Figure~$2$ in Table~\ref{tab:layers}.

\section{YFCC100M and Imagenet label distribution}

YFCC100M dataset contains social media from the Flickr website.
The content of this dataset is very unbalanced, with a ``long-tail'' distribution of hashtags contrasting with the well-behaved label distribution of ImageNet as can be seen in Figure~\ref{fig:tag_dist}.
For example, \textit{guenon} and \textit{baseball} correspond to labels with $1300$ associated images in ImageNet, while there are respectively $226$ and $256,758$ images associated with these hashtags in YFCC100M.

\section{Pre-training for ImageNet}

\begin{figure}[t]
  \centering
  \includegraphics[scale=0.3]{figure_tag_dist.pdf}
  \caption{
    Comparison of the hashtag distribution in YFCC100M with the label distribution in ImageNet.
  }
  \label{fig:tag_dist}
\end{figure}

\begin{table}[h]
\centering
  \setlength{\tabcolsep}{3.5pt}
    \begin{tabular}{@{}l cc @{}}
      \toprule
	    & \footnotesize PyTorch doc & \footnotesize Our \\
	    & \footnotesize hyperparam & \footnotesize hyperparam \\
      \midrule
      	    \small Supervised (PyTorch documentation\footnote{\url{pytorch.org/docs/stable/torchvision/models}}) & $73.4$ & - \\
	    \small Supervised (our code)  & $73.3$ & $74.1$ \\
	    \small Supervised + RotNet pre-training & $73.7$ & $74.5$ \\
	    \small Supervised + DeeperCluster\xspace pre-training & $74.3$ & $74.9$ \\
      \bottomrule
    \end{tabular}
    \caption{
      Top-$1$ accuracy on validation set of a VGG-16 trained on ImageNet with supervision with different initializations.
      We compare a network initialized randomly to networks pre-trained with our unsupervised method or with RotNet on YFCC100M.
    }
    \label{tab:pretrain90}
\end{table}
In Table~\ref{tab:pretrain90}, we compare the performance of a network trained with supervision on ImageNet with a standard intialization (``Supervised'') to one pre-trained with DeeperCluster\xspace (``Supervised + DeeperCluster\xspace pre-training'') and to one pre-trained with RotNet (``Supervised + RotNet pre-training'').
The convnet is finetuned on ImageNet with supervision with mini-batch SGD following the hyperparameters of the ImageNet classification example implementation from PyTorch documentation\footnote{\url{github.com/pytorch/examples/blob/master/imagenet/main.py}}).
Indeed, we train for $90$ epochs (instead of $100$ epochs in Table~$3$ of the main paper).
We use a learning rate of $0.1$, a weight decay of $0.0001$, a batch size of $256$ and dropout of $0.5$.
We reduce the learning rate by a factor of $0.1$ at epochs $30$ and $60$ (instead of decaying the learning rate with a factor $0.2$ every $20$ epochs in Table~$3$ of the main paper).
This setting is unfair towards the supervised from scratch baseline since as we start the optimization with a good initialization we arrive at convergence earlier.
Indeed, we observe that the gap between our pretraining and the baseline shrinks from $1.0$ to $0.8$ when evaluating at convergence instead of evaluating before convergence.
As a matter of fact, the gap for the RotNet pretraining with the baseline remains the same: $0.4$.

\section{Model analysis}
\subsection{Instance retrieval}

\begin{table}[h]
\centering
    \begin{tabular}{@{}lc c c@{}}
      \toprule
	    Method & Pretraining & Oxford$5$K & Paris$6$K \\
      \midrule
      ImageNet labels & ImageNet  & $72.4$ & $81.5$  \\
      Random          & -         & $\phantom{0}6.9$  & $22.0$ \\
      \midrule
      Doersch~\emph{et al}\onedot~\cite{doersch2015unsupervised} & ImageNet & $35.4$ & $53.1$ \\
      Wang~\emph{et al}\onedot~\cite{wang2017transitive} & Youtube $9$M & $42.3$ & $58.0$ \\
      \midrule
      RotNet         & ImageNet & $48.2$ & $61.1$\\
      DeepCluster     & ImageNet & $\mathbf{61.1}$ & $\mathbf{74.9}$\\
      \midrule
      RotNet          & YFCC100M & $46.5$ & $59.2$ \\
      DeepCluster     & YFCC100M & $57.2$ & $74.6$ \\
      \midrule
      DeeperCluster\xspace           & YFCC100M & $55.8$ & $73.4$ \\
      \bottomrule
    \end{tabular}
    \caption{
      mAP on instance-level image retrieval on Oxford and Paris dataset.
      We apply R-MAC with a resolution of $1024$ pixels and $3$ grid levels~\cite{tolias2015particular}.
      We disassociate the methods using unsupervised ImageNet and the methods using non-curated datasets.
      DeepCluster does not scale to the full YFCC100M dataset, we thus train it on a random subset of $1.3$M images.
    }
    \label{tab:retrieval}
\end{table}

Instance retrieval consists of retrieving from a corpus the most similar images to a given a query.
We follow the experimental setting of Tolias~\emph{et al}\onedot~\cite{tolias2015particular}:
we apply R-MAC with a resolution of $1024$ pixels and $3$ grid levels and we report mAP on instance-level image retrieval on Oxford Buildings~\cite{philbin2007object} and Paris~\cite{philbin2008lost} datasets.

As described by Dosovitskiy~\emph{et al}\onedot~\cite{dosovitskiy2016discriminative}, class-level supervision induces invariance to semantic categories.
This property may not be beneficial for other computer vision tasks such as instance-level recognition.
For that reason, descriptor matching and instance retrieval are tasks for which unsupervised feature learning might provide performance improvements.
Moreover, these tasks constitute evaluations that do not require any additionnal training step, allowing a straightforward comparison accross different methods.
We evaluate our method and compare it to previous work following the experimental setup proposed by Caron~\emph{et al}\onedot~\cite{caron2018deep}.
We report results for the instance retrieval task in Table~\ref{tab:retrieval}.

We observe that features trained with RotNet have significantly worse performance than DeepCluster both on Oxford$5$K and Paris$6$K.
This performance discrepancy means that properties acquired by classifying large rotations are not relevant to instance retrieval.
An explanation is that all images in Oxford$5$k and Paris$6$k have the same orientation as they picture buildings and landmarks.
As our method is a combination of the two paradigms, it suffers an important performance loss on Oxfork$5$K, but is not affected much on Paris$6$k.
These results emphasize the importance of having a diverse set of benchmarks to evaluate the quality of features produced by unsupervised learning methods.

\begin{figure}[ht]
\centering
\includegraphics[width=0.7\linewidth]{figure_color.pdf}
\caption{Sorted standard deviations to clusters mean colors.
If the standard deviation of a cluster to its mean color is low, the images of this cluster have a similar colorization.}
\label{fig:color_std}
\end{figure}

\subsection{Influence of data pre-processing}
In this section we experiment with our method on raw RGB inputs.
We provide some insights into the reasons why sobel filtering is crucial to obtain good performance with our method.

First, in Figure~\ref{fig:color_std}, we randomly select a subset of $3000$ clusters and sort them by standard deviation to their mean color.
If the standard deviation of a cluster to its mean color is low, it means that the images of this cluster tend to have a similar colorization.
Moreover, we show in Figure~\ref{fig:color_cluster} some clusters with a low standard deviation to the mean color.
We observe in Figure~\ref{fig:color_std} that the clustering on features learned with our method focuses more on color than the clustering on RotNet features.
Indeed, clustering by color and low-level information produces balanced clusters that can easily be predicted by a convnet.
Clustering by color is a solution to our formulation.
However, as we want to avoid an uninformative clustering essentially based on colors, we remove some part of the input information by feeding the network with the image gradients instead of the raw RGB image (see Figure~\ref{fig:preprocess}).
This allows to greatly improve the performance of our features when evaluated on downstream tasks as it can be seen in Table~\ref{tab:rgb}.
We observe that Sobel filter improves slightly RotNet features as well.

\begin{figure}[ht]
\centering
  \begin{tabular}{cccc}
\includegraphics[width=0.2\linewidth]{color_153.jpeg}&
\includegraphics[width=0.2\linewidth]{color_171.jpeg}&
\includegraphics[width=0.2\linewidth]{color_437.jpeg}&
\includegraphics[width=0.2\linewidth]{color_319.jpeg}
\\
\\
\includegraphics[width=0.1\linewidth]{color153.jpeg}&
\includegraphics[width=0.1\linewidth]{color171.jpeg}&
\includegraphics[width=0.1\linewidth]{color437.jpeg}&
\includegraphics[width=0.1\linewidth]{color319.jpeg}
\end{tabular}
\caption{We show clusters with an uniform colorization accross their images.
For each cluster, we show the mean color of the cluster.}
\label{fig:color_cluster}
\end{figure}

\begin{figure}[t]
\centering
  \begin{tabular}{c cc}
  RGB & \multicolumn{2}{c}{Sobel}\\
  \midrule
\includegraphics[width=0.27\linewidth]{img2-RGB.jpeg}&
\includegraphics[width=0.27\linewidth]{img2-sobx.jpeg}&
\includegraphics[width=0.27\linewidth]{img2-soby.jpeg}
\\
  \midrule
\includegraphics[width=0.27\linewidth]{img3-RGB.jpeg}&
\includegraphics[width=0.27\linewidth]{img3-sobx.jpeg}&
\includegraphics[width=0.27\linewidth]{img3-soby.jpeg}
\end{tabular}
\caption{Visualization of two images preprocessed with Sobel filter.
Sobel gives a $2$ channels output which at each point contain the vertical and horizontal derivative approximations.
Photographer usernames of these two YFCC100M RGB images are respectively \textit{booledozer} and \textit{nathalie.cone}.
}
\label{fig:preprocess}
\end{figure}

\begin{table}[t]
\centering
  \setlength{\tabcolsep}{3.5pt}
    \begin{tabular}{@{}l c c c@{}}
      \toprule
      Method & Data & RGB & Sobel \\
      \midrule
      RotNet      & YFCC 1M & $69.8$ & $70.4$ \\
      \midrule
      DeeperCluster\xspace & YFCC 20M & $71.6$ & $76.1$ \\
      \bottomrule
    \end{tabular}
    \caption{
Influence of applying Sobel filter or using raw RGB input on the features quality.
We report validation mAP on Pascal VOC classification task (\textsc{fc68} setting).
    }
    \label{tab:rgb}
\end{table}

\section{Hyperparameters}
In this section, we detail our different hyperparameter choices.
Images are rescaled to $3 \times 224 \times 224$.
Note that for each network we choose the best performing hyperparameters by evaluating on Pascal VOC $2007$ classification task without finetuning.
\begin{itemize}
\item \textbf{RotNet YFCC100M}: we train with a total batch-size of $512$, a learning rate of $0.05$, weight decay of $0.00001$ and dropout of $0.3$.
\item \textbf{RotNet ImageNet}: we train with a total batch-size of $512$, a learning rate of $0.05$, weight decay of $0.00001$ and dropout of $0.3$.
\item \textbf{DeepCluster YFCC100M 1.3M images}: we train with a total batch-size of $256$, a learning rate of $0.05$, weight decay of $0.00001$ and dropout of $0.5$.
A sobel filter is used in preprocessing step. We cluster the pca-reduced to $256$ dimensions, whitened and normalized features with $k$-means into $10.000$ clusters every $2$ epochs of training.
\item \textbf{DeeperCluster YFCC100M}: we train with a total batch-size of $3072$, a learning rate of $0.1$, weight decay of $0.00001$ and dropout of $0.5$.
A sobel filter is used in preprocessing step.
We cluster the whitened and normalized features (of dimension $4096$) of the non-rotated images with hierarchical $k$-means into $320.000$ clusters ($4$ clusterings in $80.000$ clusters each) every $3$ epochs of training.
\item \textbf{DeeperCluster ImageNet}: we train with a total batch-size of $748$, a learning rate of $0.1$, weight decay of $0.00001$ and dropout of $0.5$.
A sobel filter is used in preprocessing step. We cluster the whitened and normalized features (of dimension $4096$) of the non-rotated images with $k$-means into $10.000$ clusters every $5$ epochs of training.
\end{itemize}
For all methods, we use stochastic gradient descent with a momentum of $0.9$.
We stop training as soon as performance on Pascal VOC $2007$ classification task saturates.
We use PyTorch version 1.0 for all our experiments.

\section{Usernames of cluster visualization images}
For copyright reason, we give here the Flickr user names of the images from Figure~$5$.
For each cluster, the user names are listed from left to right and from top to bottom.
Photographers of images in cluster \textit{cat} are sun\_summer, savasavasava, windy\_sydney, ironsalchicha, Chiang Kai Yen, habigu, Crackers93, rikkis\_refuge and rabidgamer.
Photographers of images in custer \textit{elephantparadelondon} are Karen Roe, asw909, Matt From London, jorgeleria, Loz Flowers, Loz Flowers, Deck Accessory, Maxwell Hamilton and Melinda 26 Cristiano.
Photographers of images in custer \textit{always} are troutproject, elandru, vlauria, Raymond Yee, tsupo543, masatsu, robotson, edgoubert and troutproject.
Photographers of images in custer \textit{CanoScan} are what-i-found, what-i-found, allthepreciousthings, carbonated, what-i-found, what-i-found, what-i-found, what-i-found and what-i-found.
Photographers of images in custer \textit{GPS: (43, 10)} are bloke, garysoccer1, macpalm, M A T T E O 1 2 3, coder11, Johan.dk, chrissmallwood, markomni and xiquinhosilva.
Photographers of images in custer \textit{GPS: (-34, -151)} are asamiToku, Scott R Frost, BeauGiles, MEADEN, chaitanyakuber, mathias Straumann, jeroenvanlieshout, jamespia and Bastard Sheep.
Photographers of images in custer \textit{GPS(64, -20)} are arrygj, Bsivad, Powys Walker, Maria Grazia Dal Pra27, Sterling College, roundedbygravity, johnmcga, MuddyRavine and El coleccionista de instantes.
Photographers of images in custer \textit{GPS: (43, -104)} are dodds, eric.terry.kc, Lodahln, wmamurphy, purza7, jfhatesmustard, Marcel B., Silly America and Liralen Li.