  Post-Training Quantization (PTQ) is a powerful technique for model
compression, reducing the precision of neural networks without additional
training overhead. Recent works have investigated adopting 8-bit floating-point
quantization (FP8) in the context of PTQ for model inference. However, the
exploration of floating-point formats smaller than 8 bits and their comparison
with integer quantization remains relatively limited. In this work, we present
minifloats, which are reduced-precision floating-point formats capable of
further reducing the memory footprint, latency, and energy cost of a model
while approaching full-precision model accuracy. Our work presents a novel PTQ
design-space exploration, comparing minifloat and integer quantization schemes
across a range of 3 to 8 bits for both weights and activations. We examine the
applicability of various PTQ techniques to minifloats, including weight
equalization, bias correction, SmoothQuant, gradient-based learned rounding,
and the GPTQ method. Our experiments validate the effectiveness of
low-precision minifloats when compared to their integer counterparts across a
spectrum of accuracy-precision trade-offs on a set of reference deep learning
vision workloads. Finally, we evaluate our results against an FPGA-based
hardware cost model, showing that integer quantization often remains the
Pareto-optimal option, given its relatively smaller hardware resource
footprint.
