Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: NeuroFabric: Identifying Ideal Topologies for Training A Priori Sparse
  Networks   Long training times of deep neural networks are a bottleneck in machine
learning research. The major impediment to fast training is the quadratic
growth of both memory and compute requirements of dense and convolutional
layers with respect to their information bandwidth. Recently, training `a
priori' sparse networks has been proposed as a method for allowing layers to
retain high information bandwidth, while keeping memory and compute low.
However, the choice of which sparse topology should be used in these networks
is unclear. In this work, we provide a theoretical foundation for the choice of
intra-layer topology. First, we derive a new sparse neural network
initialization scheme that allows us to explore the space of very deep sparse
networks. Next, we evaluate several topologies and show that seemingly similar
topologies can often have a large difference in attainable accuracy. To explain
these differences, we develop a data-free heuristic that can evaluate a
topology independently from the dataset the network will be trained on. We then
derive a set of requirements that make a good topology, and arrive at a single
topology that satisfies all of them.

31
The categories of this paper are: ao-sci
0
The categories of this paper are: hep-ph
36
The categories of this paper are: cmp-lg
11
The categories of this paper are: hep-lat
29
The categories of this paper are: funct-an
19
The categories of this paper are: eess
14
The categories of this paper are: math-ph
3
The categories of this paper are: physics
33
The categories of this paper are: plasm-ph
5
The categories of this paper are: gr-qc
2
The categories of this paper are: math
22
The categories of this paper are: chao-dyn
10
The categories of this paper are: q-bio
35
The categories of this paper are: mtrl-th
28
The categories of this paper are: alg-geom
21
The categories of this paper are: adap-org