  Convolutional neural network (CNN) accelerators are being widely used for
their efficiency, but they require a large amount of memory, leading to the use
of a slow and power consuming external memory. This paper exploits two schemes
to reduce the required memory amount and ultimately to implement a CNN of
reasonable performance only with on-chip memory of a practical device like a
low-end FPGA. To reduce the memory amount of the intermediate data, a
stream-based line-buffer architecture and a dataflow for the architecture are
proposed instead of the conventional frame-based architecture, where the amount
of the intermediate data memory is proportional to the square of the input
image size. The architecture consists of layer-dedicated blocks operating in a
pipelined way with the input and output streams. Each convolutional layer block
has a line buffer storing just a few rows of input data. The sizes of the line
buffers are proportional to the width of the input image, so the architecture
requires less intermediate data storage than the conventional frame-based
architecture, especially in the trend of getting larger input size in modern
object detection CNNs. In addition to the reduced intermediate data storage,
the weight memory is reduced by the accelerator-aware pruning. The experimental
results show that a whole object detection CNN can be implemented even on a
low-end FPGA without an external memory. Compared to previous accelerators with
similar object detection accuracy, the proposed accelerator reaches much higher
throughput even with less FPGA resources of LUTs, registers, and DSPs, showing
much higher efficiency. The trained models and implemented bit files are
available at https://github.com/HyeongjuKang/accelerator-aware-pruning and
https://github.com/HyeongjuKang/aocstream.
