  Merchandise categories inherently form a semantic hierarchy with different
levels of concept abstraction, especially for fine-grained categories. This
hierarchy encodes rich correlations among various categories across different
levels, which can effectively regularize the semantic space and thus make
predictions less ambiguous. However, previous studies of fine-grained image
retrieval primarily focus on semantic similarities or visual similarities. In a
real application, merely using visual similarity may not satisfy the need of
consumers to search merchandise with real-life images, e.g., given a red coat
as a query image, we might get a red suit in recall results only based on
visual similarity since they are visually similar. But the users actually want
a coat rather than suit even the coat is with different color or texture
attributes. We introduce this new problem based on photoshopping in real
practice. That's why semantic information are integrated to regularize the
margins to make "semantic" prior to "visual". To solve this new problem, we
propose a hierarchical adaptive semantic-visual tree (ASVT) to depict the
architecture of merchandise categories, which evaluates semantic similarities
between different semantic levels and visual similarities within the same
semantic class simultaneously. The semantic information satisfies the demand of
consumers for similar merchandise with the query while the visual information
optimizes the correlations within the semantic class. At each level, we set
different margins based on the semantic hierarchy and incorporate them as prior
information to learn a fine-grained feature embedding. To evaluate our
framework, we propose a new dataset named JDProduct, with hierarchical labels
collected from actual image queries and official merchandise images on an
online shopping application. Extensive experimental results on the public
CARS196 and CUB-
