Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Looking Enhances Listening: Recovering Missing Speech Using Images   Speech is understood better by using visual context; for this reason, there
have been many attempts to use images to adapt automatic speech recognition
(ASR) systems. Current work, however, has shown that visually adapted ASR
models only use images as a regularization signal, while completely ignoring
their semantic content. In this paper, we present a set of experiments where we
show the utility of the visual modality under noisy conditions. Our results
show that multimodal ASR models can recover words which are masked in the input
acoustic signal, by grounding its transcriptions using the visual
representations. We observe that integrating visual context can result in up to
35% relative improvement in masked word recovery. These results demonstrate
that end-to-end multimodal ASR systems can become more robust to noise by
leveraging the visual context.

15
The categories of this paper are: nucl-ex
9
The categories of this paper are: nlin
10
The categories of this paper are: q-bio
33
The categories of this paper are: plasm-ph
14
The categories of this paper are: math-ph
27
The categories of this paper are: comp-gas
20
The categories of this paper are: acc-phys
4
The categories of this paper are: cond-mat
25
The categories of this paper are: patt-sol
37
The categories of this paper are: supr-con
6
The categories of this paper are: astro-ph
24
The categories of this paper are: dg-ga
29
The categories of this paper are: funct-an
34
The categories of this paper are: chem-ph
0
The categories of this paper are: hep-ph
36
The categories of this paper are: cmp-lg