  The success of neural networks has driven a shift in focus from feature
engineering to architecture engineering. However, successful networks today are
constructed using a small and manually defined set of building blocks. Even in
methods of neural architecture search (NAS) the network connectivity patterns
are largely constrained. In this work we propose a method for discovering
neural wirings. We relax the typical notion of layers and instead enable
channels to form connections independent of each other. This allows for a much
larger space of possible networks. The wiring of our network is not fixed
during training -- as we learn the network parameters we also learn the
structure itself. Our experiments demonstrate that our learned connectivity
outperforms hand engineered and randomly wired networks. By learning the
connectivity of MobileNetV1we boost the ImageNet accuracy by 10% at ~41M FLOPs.
Moreover, we show that our method generalizes to recurrent and continuous time
networks. Our work may also be regarded as unifying core aspects of the neural
architecture search problem with sparse neural network learning. As NAS becomes
more fine grained, finding a good architecture is akin to finding a sparse
subnetwork of the complete graph. Accordingly, DNW provides an effective
mechanism for discovering sparse subnetworks of predefined architectures in a
single training run. Though we only ever use a small percentage of the weights
during the forward pass, we still play the so-called initialization lottery
with a combinatorial number of subnetworks. Code and pretrained models are
available at https://github.com/allenai/dnw while additional visualizations may
be found at https://mitchellnw.github.io/blog/2019/dnw/.
