  The ability of artificial intelligence (AI) systems to perceive and
comprehend audio signals is crucial for many applications. Although significant
progress has been made in this area since the development of AudioSet, most
existing models are designed to map audio inputs to pre-defined, discrete sound
label sets. In contrast, humans possess the ability to not only classify sounds
into general categories, but also to listen to the finer details of the sounds,
explain the reason for the predictions, think about what the sound infers, and
understand the scene and what action needs to be taken, if any. Such
capabilities beyond perception are not yet present in existing audio models. On
the other hand, modern large language models (LLMs) exhibit emerging reasoning
ability but they lack audio perception capabilities. Therefore, we ask the
question: can we build a model that has both audio perception and a reasoning
ability?
  In this paper, we propose a new audio foundation model, called LTU (Listen,
Think, and Understand). To train LTU, we created a new OpenAQA-5M dataset
consisting of 1.9 million closed-ended and 3.7 million open-ended, diverse
(audio, question, answer) tuples, and have used an autoregressive training
framework with a perception-to-understanding curriculum. LTU demonstrates
strong performance and generalization ability on conventional audio tasks such
as classification and captioning. More importantly, it exhibits emerging audio
reasoning and comprehension abilities that are absent in existing audio models.
To the best of our knowledge, LTU is one of the first multimodal large language
models that focus on general audio (rather than just speech) understanding.
