Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: VisualGPT: Data-efficient Adaptation of Pretrained Language Models for
  Image Captioning   The ability to quickly learn from a small quantity oftraining data widens the
range of machine learning applications. In this paper, we propose a
data-efficient image captioning model, VisualGPT, which leverages the
linguistic knowledge from a large pretrained language model(LM). A crucial
challenge is to balance between the use of visual information in the image and
prior linguistic knowledge acquired from pretraining. We designed a novel
self-resurrecting encoder-decoder attention mechanism to quickly adapt the
pretrained LM as the language decoder ona small amount of in-domain training
data. The proposed self-resurrecting activation unit produces sparse
activations but has reduced susceptibility to zero gradients. We train the
proposed model, VisualGPT, on 0.1%, 0.5% and 1% of MSCOCO and Conceptual
Captions training data. Under these conditions, we outperform the best baseline
model by up to 10.8% CIDEr on MS COCO and upto 5.4% CIDEr on Conceptual
Captions. Further, Visual-GPT achieves the state-of-the-art result on IU X-ray,
a medical report generation dataset. To the best of our knowledge, this is the
first work that improves data efficiency of image captioning by utilizing LM
pretrained on unimodal data. Our code is available at:
https://github.com/Vision-CAIR/VisualGPT.

23
The categories of this paper are: solv-int
5
The categories of this paper are: gr-qc
31
The categories of this paper are: ao-sci
6
The categories of this paper are: astro-ph
32
The categories of this paper are: atom-ph
21
The categories of this paper are: adap-org
18
The categories of this paper are: econ
35
The categories of this paper are: mtrl-th
15
The categories of this paper are: nucl-ex
26
The categories of this paper are: bayes-an
3
The categories of this paper are: physics
36
The categories of this paper are: cmp-lg
9
The categories of this paper are: nlin
4
The categories of this paper are: cond-mat
37
The categories of this paper are: supr-con
33
The categories of this paper are: plasm-ph