  We propose a new method for creating computationally efficient convolutional
neural networks (CNNs) by using low-rank representations of convolutional
filters. Rather than approximating filters in previously-trained networks with
more efficient versions, we learn a set of small basis filters from scratch;
during training, the network learns to combine these basis filters into more
complex filters that are discriminative for image classification. To train such
networks, a novel weight initialization scheme is used. This allows effective
initialization of connection weights in convolutional layers composed of groups
of differently-shaped filters. We validate our approach by applying it to
several existing CNN architectures and training these networks from scratch
using the CIFAR, ILSVRC and MIT Places datasets. Our results show similar or
higher accuracy than conventional CNNs with much less compute. Applying our
method to an improved version of VGG-11 network using global max-pooling, we
achieve comparable validation accuracy using 41% less compute and only 24% of
the original VGG-11 model parameters; another variant of our method gives a 1
percentage point increase in accuracy over our improved VGG-11 model, giving a
top-5 center-crop validation accuracy of 89.7% while reducing computation by
16% relative to the original VGG-11 model. Applying our method to the GoogLeNet
architecture for ILSVRC, we achieved comparable accuracy with 26% less compute
and 41% fewer model parameters. Applying our method to a near state-of-the-art
network for CIFAR, we achieved comparable accuracy with 46% less compute and
55% fewer parameters.
