Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Listen, Look and Deliberate: Visual context-aware speech recognition
  using pre-trained text-video representations   In this study, we try to address the problem of leveraging visual signals to
improve Automatic Speech Recognition (ASR), also known as visual context-aware
ASR (VC-ASR). We explore novel VC-ASR approaches to leverage video and text
representations extracted by a self-supervised pre-trained text-video embedding
model. Firstly, we propose a multi-stream attention architecture to leverage
signals from both audio and video modalities. This architecture consists of
separate encoders for the two modalities and a single decoder that attends over
them. We show that this architecture is better than fusing modalities at the
signal level. Additionally, we also explore leveraging the visual information
in a second pass model, which has also been referred to as a `deliberation
model'. The deliberation model accepts audio representations and text
hypotheses from the first pass ASR and combines them with a visual stream for
an improved visual context-aware recognition. The proposed deliberation scheme
can work on top of any well trained ASR and also enabled us to leverage the
pre-trained text model to ground the hypotheses with the visual features. Our
experiments on HOW2 dataset show that multi-stream and deliberation
architectures are very effective at the VC-ASR task. We evaluate the proposed
models for two scenarios; clean audio stream and distorted audio in which we
mask out some specific words in the audio. The deliberation model outperforms
the multi-stream model and achieves a relative WER improvement of 6% and 8.7%
for the clean and masked data, respectively, compared to an audio-only model.
The deliberation model also improves recovering the masked words by 59%
relative.

4
The categories of this paper are: cond-mat
20
The categories of this paper are: acc-phys
7
The categories of this paper are: hep-th
16
The categories of this paper are: stat
6
The categories of this paper are: astro-ph
12
The categories of this paper are: nucl-th
34
The categories of this paper are: chem-ph
30
The categories of this paper are: q-alg
8
The categories of this paper are: hep-ex
5
The categories of this paper are: gr-qc
29
The categories of this paper are: funct-an
36
The categories of this paper are: cmp-lg
10
The categories of this paper are: q-bio
0
The categories of this paper are: hep-ph
21
The categories of this paper are: adap-org
18
The categories of this paper are: econ