  Light-weight convolutional neural networks (CNNs) have small complexity and
are good candidates for low-power, high-throughput inference. Such networks are
heterogeneous in terms of computation-to-communication (CTC) ratios and
computation patterns between layers, especially for different layer types. Yet,
existing AI processors either use homogeneous processing elements (PEs),
resulting in low runtime PE efficiency, or run different layers on
heterogeneous PEs in sequential, introducing resource redundancy. This paper
proposes a heterogeneous dual-core architecture (dual-OPU), where one core is
optimized for regular convolution layers and the other for depthwise
convolution layers. PEs are homogeneous with each core. To make full use of
dual-core parallelism, we develop a scheduling algorithm to concurrently
execute layers for different input images on dual-core and balance parallel
workload. Meanwhile, we automatically tune the PE number for a core and tune
the input size for each PE to maximize throughput. Compared with a single-core
processor with the same area for a single network, heterogeneous dual-OPU on
average improves runtime PE efficiency and throughput by 11% and 31%,
respectively. For a workload of multiple networks, dual-OPU improves average
throughput by 11% compared with the state-of-the-art processors scaled to the
same area. To the best of our knowledge, it is the first in-depth study on the
heterogeneous dual-core processor for light-weight CNNs.
