  We have been investigating rakugo speech synthesis as a challenging example
of speech synthesis that entertains audiences. Rakugo is a traditional Japanese
form of verbal entertainment similar to a combination of one-person stand-up
comedy and comic storytelling and is popular even today. In rakugo, a performer
plays multiple characters, and conversations or dialogues between the
characters make the story progress. To investigate how close the quality of
synthesized rakugo speech can approach that of professionals' speech, we
modeled rakugo speech using Tacotron 2, a state-of-the-art speech synthesis
system that can produce speech that sounds as natural as human speech albeit
under limited conditions, and an enhanced version of it with self-attention to
better consider long-term dependencies. We also used global style tokens and
manually labeled context features to enrich speaking styles. Through a
listening test, we measured not only naturalness but also distinguishability of
characters, understandability of the content, and the degree of entertainment.
Although we found that the speech synthesis models could not yet reach the
professional level, the results of the listening test provided interesting
insights: 1) we should not focus only on the naturalness of synthesized speech
but also the distinguishability of characters and the understandability of the
content to further entertain audiences; 2) the fundamental frequency (fo)
expressions of synthesized speech are poorer than those of human speech, and
more entertaining speech should have richer fo expression. Although there is
room for improvement, we believe this is an important stepping stone toward
achieving entertaining speech synthesis at the professional level.
