Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: No Token Left Behind: Explainability-Aided Image Classification and
  Generation   The application of zero-shot learning in computer vision has been
revolutionized by the use of image-text matching models. The most notable
example, CLIP, has been widely used for both zero-shot classification and
guiding generative models with a text prompt. However, the zero-shot use of
CLIP is unstable with respect to the phrasing of the input text, making it
necessary to carefully engineer the prompts used. We find that this instability
stems from a selective similarity score, which is based only on a subset of the
semantically meaningful input tokens. To mitigate it, we present a novel
explainability-based approach, which adds a loss term to ensure that CLIP
focuses on all relevant semantic parts of the input, in addition to employing
the CLIP similarity loss used in previous works. When applied to one-shot
classification through prompt engineering, our method yields an improvement in
the recognition rate, without additional training or fine-tuning. Additionally,
we show that CLIP guidance of generative models using our method significantly
improves the generated images. Finally, we demonstrate a novel use of CLIP
guidance for text-based image generation with spatial conditioning on object
location, by requiring the image explainability heatmap for each object to be
confined to a pre-determined bounding box.

29
The categories of this paper are: funct-an
30
The categories of this paper are: q-alg
33
The categories of this paper are: plasm-ph
6
The categories of this paper are: astro-ph
21
The categories of this paper are: adap-org
27
The categories of this paper are: comp-gas
12
The categories of this paper are: nucl-th
3
The categories of this paper are: physics
37
The categories of this paper are: supr-con
2
The categories of this paper are: math
25
The categories of this paper are: patt-sol
15
The categories of this paper are: nucl-ex
31
The categories of this paper are: ao-sci
23
The categories of this paper are: solv-int
36
The categories of this paper are: cmp-lg
16
The categories of this paper are: stat