Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Aligning Subtitles in Sign Language Videos   The goal of this work is to temporally align asynchronous subtitles in sign
language videos. In particular, we focus on sign-language interpreted TV
broadcast data comprising (i) a video of continuous signing, and (ii) subtitles
corresponding to the audio content. Previous work exploiting such
weakly-aligned data only considered finding keyword-sign correspondences,
whereas we aim to localise a complete subtitle text in continuous signing. We
propose a Transformer architecture tailored for this task, which we train on
manually annotated alignments covering over 15K subtitles that span 17.7 hours
of video. We use BERT subtitle embeddings and CNN video representations learned
for sign recognition to encode the two signals, which interact through a series
of attention layers. Our model outputs frame-level predictions, i.e., for each
video frame, whether it belongs to the queried subtitle or not. Through
extensive evaluations, we show substantial improvements over existing alignment
baselines that do not make use of subtitle text embeddings for learning. Our
automatic alignment model opens up possibilities for advancing machine
translation of sign languages via providing continuously synchronized
video-text data.

3
The categories of this paper are: physics
15
The categories of this paper are: nucl-ex
37
The categories of this paper are: supr-con
4
The categories of this paper are: cond-mat
36
The categories of this paper are: cmp-lg
25
The categories of this paper are: patt-sol
24
The categories of this paper are: dg-ga
28
The categories of this paper are: alg-geom
19
The categories of this paper are: eess
23
The categories of this paper are: solv-int
13
The categories of this paper are: quant-ph
0
The categories of this paper are: hep-ph
9
The categories of this paper are: nlin
30
The categories of this paper are: q-alg
16
The categories of this paper are: stat
31
The categories of this paper are: ao-sci