  Vision transformers, which were originally developed for natural language
processing, have recently generated significant interest in the computer vision
and audio communities due to their flexibility in learning long-range
relationships. Constrained by data hungry nature of transformers and limited
labelled data most transformer-based models for audio tasks are finetuned from
ImageNet pretrained models, despite the huge gap between the natural images
domain and audio domain. This has motivated the research in self-supervised
pretraining of audio transformers, which reduces the dependency on large
amounts of labeled data and focuses on extracting concise representation of the
audio spectrograms. In this paper, we propose ASiT, a novel self-supervised
transformer for general audio representations that captures local and global
contextual information employing group masked model learning and
self-distillation. We evaluate our pretrained models on both audio and speech
classification tasks including audio event classification, keyword spotting,
and speaker identification. We further conduct comprehensive ablation studies,
including evaluations of different pretraining strategies. The proposed ASiT
framework significantly boosts the performance on all tasks and sets a new
state-of-the-art performance on five audio and speech classification tasks,
outperforming recent methods, including the approaches that use additional
datasets for pretraining. The code and pretrained weights will be made publicly
available for the scientific community.
