  Deep Neural Networks (DNNs) have achieved extraordinary performance in
various application domains. To support diverse DNN models, efficient
implementations of DNN inference on edge-computing platforms, e.g., ASICs,
FPGAs, and embedded systems, are extensively investigated. Due to the huge
model size and computation amount, model compression is a critical step to
deploy DNN models on edge devices. This paper focuses on weight quantization, a
hardware-friendly model compression approach that is complementary to weight
pruning. Unlike existing methods that use the same quantization scheme for all
weights, we propose the first solution that applies different quantization
schemes for different rows of the weight matrix. It is motivated by (1) the
distribution of the weights in the different rows are not the same; and (2) the
potential of achieving better utilization of heterogeneous FPGA hardware
resources. To achieve that, we first propose a hardware-friendly quantization
scheme named sum-of-power-of-2 (SP2) suitable for Gaussian-like weight
distribution, in which the multiplication arithmetic can be replaced with logic
shifter and adder, thereby enabling highly efficient implementations with the
FPGA LUT resources. In contrast, the existing fixed-point quantization is
suitable for Uniform-like weight distribution and can be implemented
efficiently by DSP. Then to fully explore the resources, we propose an
FPGA-centric mixed scheme quantization (MSQ) with an ensemble of the proposed
SP2 and the fixed-point schemes. Combining the two schemes can maintain, or
even increase accuracy due to better matching with weight distributions.
