  Existing deep convolutional neural networks (CNNs) generate massive
interlayer feature data during network inference. To maintain real-time
processing in embedded systems, large on-chip memory is required to buffer the
interlayer feature maps. In this paper, we propose an efficient hardware
accelerator with an interlayer feature compression technique to significantly
reduce the required on-chip memory size and off-chip memory access bandwidth.
The accelerator compresses interlayer feature maps through transforming the
stored data into frequency domain using hardware-implemented 8x8 discrete
cosine transform (DCT). The high-frequency components are removed after the DCT
through quantization. Sparse matrix compression is utilized to further compress
the interlayer feature maps. The on-chip memory allocation scheme is designed
to support dynamic configuration of the feature map buffer size and scratch pad
size according to different network-layer requirements. The hardware
accelerator combines compression, decompression, and CNN acceleration into one
computing stream, achieving minimal compressing and processing delay. A
prototype accelerator is implemented on an FPGA platform and also synthesized
in TSMC 28-nm COMS technology. It achieves 403GOPS peak throughput and
1.4x~3.3x interlayer feature map reduction by adding light hardware area
overhead, making it a promising hardware accelerator for intelligent IoT
devices.
