Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Towards the evaluation of automatic simultaneous speech translation from
  a communicative perspective   In recent years, automatic speech-to-speech and speech-to-text translation
has gained momentum thanks to advances in artificial intelligence, especially
in the domains of speech recognition and machine translation. The quality of
such applications is commonly tested with automatic metrics, such as BLEU,
primarily with the goal of assessing improvements of releases or in the context
of evaluation campaigns. However, little is known about how the output of such
systems is perceived by end users or how they compare to human performances in
similar communicative tasks.
  In this paper, we present the results of an experiment aimed at evaluating
the quality of a real-time speech translation engine by comparing it to the
performance of professional simultaneous interpreters. To do so, we adopt a
framework developed for the assessment of human interpreters and use it to
perform a manual evaluation on both human and machine performances. In our
sample, we found better performance for the human interpreters in terms of
intelligibility, while the machine performs slightly better in terms of
informativeness. The limitations of the study and the possible enhancements of
the chosen framework are discussed. Despite its intrinsic limitations, the use
of this framework represents a first step towards a user-centric and
communication-oriented methodology for evaluating real-time automatic speech
translation.

19
The categories of this paper are: eess
4
The categories of this paper are: cond-mat
29
The categories of this paper are: funct-an
6
The categories of this paper are: astro-ph
3
The categories of this paper are: physics
7
The categories of this paper are: hep-th
24
The categories of this paper are: dg-ga
13
The categories of this paper are: quant-ph
5
The categories of this paper are: gr-qc
32
The categories of this paper are: atom-ph
37
The categories of this paper are: supr-con
20
The categories of this paper are: acc-phys
9
The categories of this paper are: nlin
8
The categories of this paper are: hep-ex
23
The categories of this paper are: solv-int
18
The categories of this paper are: econ