  Humour is a substantial element of human affect and cognition. Its automatic
understanding can facilitate a more naturalistic human-device interaction and
the humanisation of artificial intelligence. Current methods of humour
detection are solely based on staged data making them inadequate for
'real-world' applications. We address this deficiency by introducing the novel
Passau-Spontaneous Football Coach Humour (Passau-SFCH) dataset, comprising of
about 11 hours of recordings. The Passau-SFCH dataset is annotated for the
presence of humour and its dimensions (sentiment and direction) as proposed in
Martin's Humor Style Questionnaire. We conduct a series of experiments,
employing pretrained Transformers, convolutional neural networks, and
expert-designed features. The performance of each modality (text, audio, video)
for spontaneous humour recognition is analysed and their complementarity is
investigated. Our findings suggest that for the automatic analysis of humour
and its sentiment, facial expressions are most promising, while humour
direction can be best modelled via text-based features. The results reveal
considerable differences among various subjects, highlighting the individuality
of humour usage and style. Further, we observe that a decision-level fusion
yields the best recognition result. Finally, we make our code publicly
available at https://www.github.com/EIHW/passau-sfch. The Passau-SFCH dataset
is available upon request.
