  Deep neural network (DNN) has emerged as the most important and popular
artificial intelligent (AI) technique. The growth of model size poses a key
energy efficiency challenge for the underlying computing platform. Thus, model
compression becomes a crucial problem. However, the current approaches are
limited by various drawbacks. Specifically, network sparsification approach
suffers from irregularity, heuristic nature and large indexing overhead. On the
other hand, the recent structured matrix-based approach (i.e., CirCNN) is
limited by the relatively complex arithmetic computation (i.e., FFT), less
flexible compression ratio, and its inability to fully utilize input sparsity.
To address these drawbacks, this paper proposes PermDNN, a novel approach to
generate and execute hardware-friendly structured sparse DNN models using
permuted diagonal matrices. Compared with unstructured sparsification approach,
PermDNN eliminates the drawbacks of indexing overhead, non-heuristic
compression effects and time-consuming retraining. Compared with circulant
structure-imposing approach, PermDNN enjoys the benefits of higher reduction in
computational complexity, flexible compression ratio, simple arithmetic
computation and full utilization of input sparsity. We propose PermDNN
architecture, a multi-processing element (PE) fully-connected (FC)
layer-targeted computing engine. The entire architecture is highly scalable and
flexible, and hence it can support the needs of different applications with
different model configurations. We implement a 32-PE design using CMOS 28nm
technology. Compared with EIE, PermDNN achieves 3.3x~4.8x higher throughout,
5.9x~8.5x better area efficiency and 2.8x~4.0x better energy efficiency on
different workloads. Compared with CirCNN, PermDNN achieves 11.51x higher
throughput and 3.89x better energy efficiency.
