  Post-training dropout based approaches achieve high sparsity and are well
established means of deciphering problems relating to computational cost and
overfitting in Neural Network architectures. Contrastingly, pruning at
initialization is still far behind. Initialization pruning is more efficacious
when it comes to scaling computation cost of the network. Furthermore, it
handles overfitting just as well as post training dropout.
  In approbation of the above reasons, the paper presents two approaches to
prune at initialization. The goal is to achieve higher sparsity while
preserving performance. 1) K-starts, begins with k random p-sparse matrices at
initialization. In the first couple of epochs the network then determines the
"fittest" of these p-sparse matrices in an attempt to find the "lottery ticket"
p-sparse network. The approach is adopted from how evolutionary algorithms find
the best individual. Depending on the Neural Network architecture, fitness
criteria can be based on magnitude of network weights, magnitude of gradient
accumulation over an epoch or a combination of both. 2) Dissipating gradients
approach, aims at eliminating weights that remain within a fraction of their
initial value during the first couple of epochs. Removing weights in this
manner despite their magnitude best preserves performance of the network.
Contrarily, the approach also takes the most epochs to achieve higher sparsity.
3) Combination of dissipating gradients and kstarts outperforms either methods
and random dropout consistently.
  The benefits of using the provided pertaining approaches are: 1) They do not
require specific knowledge of the classification task, fixing of dropout
threshold or regularization parameters 2) Retraining of the model is neither
necessary nor affects the performance of the p-sparse network.
