  Custom dataflow Convolutional Neural Network (CNN) inference accelerators on
FPGA are tailored to a specific CNN topology and store parameters in On-Chip
Memory (OCM), resulting in high energy efficiency and low inference latency.
However, in these accelerators the shapes of parameter memories are dictated by
throughput constraints and do not map well to the underlying OCM, which becomes
an implementation bottleneck. In this work, we propose an accelerator design
methodology - Frequency Compensated Memory Packing (FCMP) - which improves the
OCM utilization efficiency of dataflow accelerators with minimal reduction in
throughput and no modifications to the physical structure of FPGA OCM. To
validate our methodology, we apply it to several realizations of medium-sized
CIFAR-10 inference accelerators and demonstrate up to 30% reduction in OCM
utilization without loss of inference throughput, allowing us to port the
accelerators from Xilinx Zynq 7020 to 7012S, reducing application cost. We also
implement a custom dataflow FPGA inference accelerator for a quantized
ResNet-50 CNN, utilizing on-chip weights, the largest topology ever implemented
with this accelerator architecture. We demonstrate that by applying FCMP to the
ResNet accelerator, the OCM bottleneck is alleviated which enables the
accelerator to be ported from Alveo U250 to the smaller Alveo U280 board with
less throughput loss compared to alternative techniques. By providing a
finer-grained trade off between throughput and OCM requirements, FCMP increases
the flexibility of custom dataflow CNN inference designs on FPGA.
