  Speech emotions are crucial in human communication and are extensively used
in fields like speech synthesis and natural language understanding. Most prior
studies, such as speech emotion recognition, have categorized speech emotions
into a fixed set of classes. Yet, emotions expressed in human speech are often
complex, and categorizing them into predefined groups can be insufficient to
adequately represent speech emotions. On the contrary, describing speech
emotions directly by means of natural language may be a more effective
approach. Regrettably, there are not many studies available that have focused
on this direction. Therefore, this paper proposes a speech emotion captioning
framework named SECap, aiming at effectively describing speech emotions using
natural language. Owing to the impressive capabilities of large language models
in language comprehension and text generation, SECap employs LLaMA as the text
decoder to allow the production of coherent speech emotion captions. In
addition, SECap leverages HuBERT as the audio encoder to extract general speech
features and Q-Former as the Bridge-Net to provide LLaMA with emotion-related
speech features. To accomplish this, Q-Former utilizes mutual information
learning to disentangle emotion-related speech features and speech contents,
while implementing contrastive learning to extract more emotion-related speech
features. The results of objective and subjective evaluations demonstrate that:
1) the SECap framework outperforms the HTSAT-BART baseline in all objective
evaluations; 2) SECap can generate high-quality speech emotion captions that
attain performance on par with human annotators in subjective mean opinion
score tests.
