  The learning of interpretable representations from raw data presents
significant challenges for time series data like speech. In this work, we
propose a relevance weighting scheme that allows the interpretation of the
speech representations during the forward propagation of the model itself. The
relevance weighting is achieved using a sub-network approach that performs the
task of feature selection. A relevance sub-network, applied on the output of
first layer of a convolutional neural network model operating on raw speech
signals, acts as an acoustic filterbank (FB) layer with relevance weighting. A
similar relevance sub-network applied on the second convolutional layer
performs modulation filterbank learning with relevance weighting. The full
acoustic model consisting of relevance sub-networks, convolutional layers and
feed-forward layers is trained for a speech recognition task on noisy and
reverberant speech in the Aurora-4, CHiME-3 and VOiCES datasets. The proposed
representation learning framework is also applied for the task of sound
classification in the UrbanSound8K dataset. A detailed analysis of the
relevance weights learned by the model reveals that the relevance weights
capture information regarding the underlying speech/audio content. In addition,
speech recognition and sound classification experiments reveal that the
incorporation of relevance weighting in the neural network architecture
improves the performance significantly.
