  3D reconstruction from videos has become increasingly popular for various
applications, including navigation for autonomous driving of robots and drones,
augmented reality (AR), and 3D modeling. This task often combines traditional
image/video processing algorithms and deep neural networks (DNNs). Although
recent developments in deep learning have improved the accuracy of the task,
the large number of calculations involved results in low computation speed and
high power consumption. Although there are various domain-specific hardware
accelerators for DNNs, it is not easy to accelerate the entire process of
applications that alternate between traditional image/video processing
algorithms and DNNs. Thus, FPGA-based end-to-end acceleration is required for
such complicated applications in low-power embedded environments.
  This paper proposes a novel FPGA-based accelerator for DeepVideoMVS, a
DNN-based depth estimation method for 3D reconstruction. We employ HW/SW
co-design to appropriately utilize heterogeneous components in modern SoC
FPGAs, such as programmable logic (PL) and CPU, according to the inherent
characteristics of the method. As some operations are unsuitable for hardware
implementation, we determine the operations to be implemented in software
through analyzing the number of times each operation is performed and its
memory access pattern, and then considering comprehensive aspects: the ease of
hardware implementation and degree of expected acceleration by hardware. The
hardware and software implementations are executed in parallel on the PL and
CPU to hide their execution latencies. The proposed accelerator was developed
on a Xilinx ZCU104 board by using NNgen, an open-source high-level synthesis
(HLS) tool. Experiments showed that the proposed accelerator operates 60.2
times faster than the software-only implementation on the same FPGA board with
minimal accuracy degradation.
