  Recent advancements in quantization and mixed-precision techniques offer
significant promise for improving the run-time and energy efficiency of neural
networks. In this work, we further showed that neural networks, wherein
individual parameters or activations can take on different precisions ranging
between 1 and 4 bits, can achieve accuracies comparable to or exceeding the
full-precision counterparts. However, the deployment of such networks poses
numerous challenges, stemming from the necessity to manage and control the
compute/communication/storage requirements associated with these extremely
fine-grained mixed precisions for each piece of data. There is a lack of
existing efficient hardware and system-level support tailored to these unique
and challenging requirements. Our research introduces the first novel holistic
hardware-software co-design approach for these networks, which enables a
continuous feedback loop between hardware design, training, and inference to
facilitate systematic design exploration. As a proof-of-concept, we illustrate
this co-design approach by designing new, configurable CPU SIMD architectures
tailored for these networks, tightly integrating the architecture with new
system-aware training and inference techniques. We perform systematic design
space exploration using this framework to analyze various tradeoffs. The design
for mixed-precision networks that achieves optimized tradeoffs corresponds to
an architecture that supports 1, 2, and 4-bit fixed-point operations with four
configurable precision patterns, when coupled with system-aware training and
inference optimization -- networks trained for this design achieve accuracies
that closely match full-precision accuracies, while compressing and improving
run-time efficiency of the neural networks drastically by 10-20x, compared to
full-precision networks.
