Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Gaining Extra Supervision via Multi-task learning for Multi-Modal Video
  Question Answering   This paper proposes a method to gain extra supervision via multi-task
learning for multi-modal video question answering. Multi-modal video question
answering is an important task that aims at the joint understanding of vision
and language. However, establishing large scale dataset for multi-modal video
question answering is expensive and the existing benchmarks are relatively
small to provide sufficient supervision. To overcome this challenge, this paper
proposes a multi-task learning method which is composed of three main
components: (1) multi-modal video question answering network that answers the
question based on the both video and subtitle feature, (2) temporal retrieval
network that predicts the time in the video clip where the question was
generated from and (3) modality alignment network that solves metric learning
problem to find correct association of video and subtitle modalities. By
simultaneously solving related auxiliary tasks with hierarchically shared
intermediate layers, the extra synergistic supervisions are provided. Motivated
by curriculum learning, multi task ratio scheduling is proposed to learn easier
task earlier to set inductive bias at the beginning of the training. The
experiments on publicly available dataset TVQA shows state-of-the-art results,
and ablation studies are conducted to prove the statistical validity.

8
The categories of this paper are: hep-ex
19
The categories of this paper are: eess
26
The categories of this paper are: bayes-an
21
The categories of this paper are: adap-org
14
The categories of this paper are: math-ph
24
The categories of this paper are: dg-ga
36
The categories of this paper are: cmp-lg
23
The categories of this paper are: solv-int
31
The categories of this paper are: ao-sci
13
The categories of this paper are: quant-ph
22
The categories of this paper are: chao-dyn
27
The categories of this paper are: comp-gas
20
The categories of this paper are: acc-phys
7
The categories of this paper are: hep-th
35
The categories of this paper are: mtrl-th
10
The categories of this paper are: q-bio