Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Multi-modal Understanding and Generation for Medical Images and Text via
  Vision-Language Pre-Training   Recently a number of studies demonstrated impressive performance on diverse
vision-language multimodal tasks such as image captioning and visual question
answering by extending the self-attention based Transformer architecture with
multimodal pre-training objectives. Despite its huge potential, vision-language
multimodal pre-training in the medical domain has only recently received
attention, and only demonstrated improved diagnosis accuracy of vision-language
pre-trained models. In this work we explore a broad set of multimodal
representation learning tasks in the medical domain, specifically using
radiology images and the unstructured report. We propose a new model which
adopts a Transformer based architecture combined with a novel multimodal
attention masking scheme to maximize generalization performance for both
vision-language understanding task (e.g., diagnosis classification) and
vision-language generation task (e.g., radiology report generation). By
rigorously evaluating the proposed model on four downstream tasks with three
radiographic image-text datasets (MIMIC-CXR, Open-I, and VQA-RAD), we
empirically demonstrate the superior downstream task performance and generality
of our model against various baselines including task specific architectures.
In addition, we qualitatively analyze our model by showing the results of
retrieved image-report pairs, the attention map visualization, and generated
reports. Our proposed multimodal pre-training model could flexibly adapt to
multiple downstream tasks of vision-language understanding and generation with
a novel self-attention scheme. We believe that our approach can provide the
basis for a wide range of interpretations of vision-language multimodal in the
medical domain.

3
The categories of this paper are: physics
27
The categories of this paper are: comp-gas
5
The categories of this paper are: gr-qc
33
The categories of this paper are: plasm-ph
26
The categories of this paper are: bayes-an
24
The categories of this paper are: dg-ga
37
The categories of this paper are: supr-con
4
The categories of this paper are: cond-mat
12
The categories of this paper are: nucl-th
22
The categories of this paper are: chao-dyn
31
The categories of this paper are: ao-sci
8
The categories of this paper are: hep-ex
0
The categories of this paper are: hep-ph
36
The categories of this paper are: cmp-lg
13
The categories of this paper are: quant-ph
30
The categories of this paper are: q-alg