text: We propose a novel method for the blind separation of audio signals produced by musical instruments. While the approach of applying non-negative matrix factorization (NMF) has been studied in many papers, it does not make use of the pitch-invariance that the sounds of instruments exhibit. This limitation can be overcome by using tensor factorization, in which context the use of log-frequency spectrograms was initiated, but this still requires the specific tuning of the instruments to be hard-coded into the algorithm. We develop a time-frequency representation that is both shift-invariant and frequency-aligned, with a variant that can also be used for wideband signals. Our separation algorithm exploits this shift-invariance in order to find patterns of peaks related to specific instruments, while non-linear optimization enables it to represent arbitrary frequencies and incorporate inharmonicity, and the reasonability of the representation is ensured by a sparsity condition. The relative amplitudes of the harmonics are saved in a dictionary, which is trained via a modified version of ADAM. For a realistic monaural piece with acoustic recorder and violin, we achieve qualitatively good separation with a signal-to-distortion ratio (SDR) of 12.5 dB, a signal-to-interference ratio (SIR) of 25.7 dB, and a signal-to-artifacts ratio (SAR) of 12.7 dB, averaged.
  @cite_19: We present a methodology for analyzing polyphonic musical passages comprised of notes that exhibit a harmonically fixed spectral profile (such as piano notes). Taking advantage of this unique note structure, we can model the audio content of the musical passage by a linear basis transform and use non-negative matrix decomposition methods to estimate the spectral profile and the temporal information of every note. This approach results in a very simple and compact system that is not knowledge-based, but rather learns notes by observation.
  @cite_27: 
  @cite_22: Is perception of the whole based on perception of its parts? There is psychological1 and physiological2,3 evidence for parts-based representations in the brain, and certain computational theories of object recognition rely on such representations4,5. But little is known about how brains or computers might learn the parts of objects. Here we demonstrate an algorithm for non-negative matrix factorization that is able to learn parts of faces and semantic features of text. This is in contrast to other methods, such as principal components analysis and vector quantization, that learn holistic, not parts-based, representations. Non-negative matrix factorization is distinguished from the other methods by its use of non-negativity constraints. These constraints lead to a parts-based representation because they allow only additive, not subtractive, combinations. When non-negative matrix factorization is implemented as a neural network, parts-based representations emerge by virtue of two properties: the firing rates of neurons are never negative and synaptic strengths do not change sign.
  @cite_8: Time-frequency analysis is a modern branch of harmonic analysis. It com- prises all those parts of mathematics and its applications that use the struc- ture of translations and modulations (or time-frequency shifts) for the anal- ysis of functions and operators. Time-frequency analysis is a form of local Fourier analysis that treats time and frequency simultaneously and sym- metrically. My goal is a systematic exposition of the foundations of time-frequency analysis, whence the title of the book. The topics range from the elemen- tary theory of the short-time Fourier transform and classical results about the Wigner distribution via the recent theory of Gabor frames to quantita- tive methods in time-frequency analysis and the theory of pseudodifferential operators. This book is motivated by applications in signal analysis and quantum mechanics, but it is not about these applications. The main ori- entation is toward the detailed mathematical investigation of the rich and elegant structures underlying time-frequency analysis. Time-frequency analysis originates in the early development of quantum mechanics by H. Weyl, E. Wigner, and J. von Neumann around 1930, and in the theoretical foundation of information theory and signal analysis by D.
summary: In order to apply dictionary learning on audio data, it is helpful to regard a time-frequency representation which subdivides the problem into smaller time frames and highlights the frequency characteristics of the signal. Classically, such a representation is computed via application of the short-time Fourier transform (STFT) (cf. @cite_8 ) on the audio signal, and the resulting spectrogram (i.e., the magnitude of the STFT) is then decomposed via non-negative matrix factorization (NMF) @cite_22 in order to obtain a dictionary. This approach was initially studied by Smaragdis and Brown @cite_19 for the purpose of polyphonic music transcription and then applied to audio source separation by Wang and Plumbley @cite_27 .