text: Training deep neural networks results in strong learned representations that show good generalization capabilities. In most cases, training involves iterative modification of all weights in the network via back-propagation. In Extreme Learning Machines, it has been suggested to set the first layer of a network to fixed random values instead of learning it. In this paper, we propose to take this approach a step further and fix almost all layers of a deep convolutional neural network, allowing only a small portion of the weights to be learned. As our experiments show, fixing even the majority of the parameters of the network often results in performance which is on par with the performance of learning all of them. The implications of this intriguing property of deep neural networks are discussed and we suggest practical ways to harness it to create more robust and compact representations.
  @cite_9: In Hadamard Matrices and Their Applications, K. J. Horadam provides the first unified account of cocyclic Hadamard matrices and their applications in signal and data processing. This original work is based on the development of an algebraic link between Hadamard matrices and the cohomology of finite groups that was discovered fifteen years ago. The book translates physical applications into terms a pure mathematician will appreciate, and theoretical structures into ones an applied mathematician, computer scientist, or communications engineer can adapt and use. The first half of the book explains the state of our knowledge of Hadamard matrices and two important generalizations: matrices with group entries and multidimensional Hadamard arrays. It focuses on their applications in engineering and computer science, as signal transforms, spreading sequences, error-correcting codes, and cryptographic primitives. The book's second half presents the new results in cocyclic Hadamard matrices and their applications. Full expression of this theory has been realized only recently, in the Five-fold Constellation. This identifies cocyclic generalized Hadamard matrices with particular "stars" in four other areas of mathematics and engineering: group cohomology, incidence structures, combinatorics, and signal correlation. Pointing the way to possible new developments in a field ripe for further research, this book formulates and discusses ninety open questions.
  @cite_6: Neural networks are commonly used as models for classification for a wide variety of tasks. Typically, a learned affine transformation is placed at the end of such models, yielding a per-class value used for classification. This classifier can have a vast number of parameters, which grows linearly with the number of possible classes, thus requiring increasingly more resources. In this work we argue that this classifier can be fixed, up to a global scale constant, with little or no loss of accuracy for most tasks, allowing memory and computational benefits. Moreover, we show that by initializing the classifier with a Hadamard matrix we can speed up inference as well. We discuss the implications for current understanding of neural network models.
summary: A very recent result is that of @cite_6 , showing - quite surprisingly - that using a fixed, Hadamard matrix @cite_9 for a final classification layer does not hinder the performance of a classifier. In contrast, we do not impose any constraints on the values of any of the fixed weights (except drawing them from the same distribution as that of the learned ones), and evaluate the effect of fixing many different subsets of weights throughout the network.