  The increasing computational cost of deep neural network models limits the
applicability of intelligent applications on resource-constrained edge devices.
While a number of neural network pruning methods have been proposed to compress
the models, prevailing approaches focus only on parametric operators (e.g.,
convolution), which may miss optimization opportunities. In this paper, we
present a novel fusion-catalyzed pruning approach, called FuPruner, which
simultaneously optimizes the parametric and non-parametric operators for
accelerating neural networks. We introduce an aggressive fusion method to
equivalently transform a model, which extends the optimization space of pruning
and enables non-parametric operators to be pruned in a similar manner as
parametric operators, and a dynamic filter pruning method is applied to
decrease the computational cost of models while retaining the accuracy
requirement. Moreover, FuPruner provides configurable optimization options for
controlling fusion and pruning, allowing much more flexible
performance-accuracy trade-offs to be made. Evaluation with state-of-the-art
residual neural networks on five representative intelligent edge platforms,
Jetson TX2, Jetson Nano, Edge TPU, NCS, and NCS2, demonstrates the
effectiveness of our approach, which can accelerate the inference of models on
CIFAR-10 and ImageNet datasets.
