  Deployment of modern TinyML tasks on small battery-constrained IoT devices
requires high computational energy efficiency. Analog In-Memory Computing (IMC)
using non-volatile memory (NVM) promises major efficiency improvements in deep
neural network (DNN) inference and serves as on-chip memory storage for DNN
weights. However, IMC's functional flexibility limitations and their impact on
performance, energy, and area efficiency are not yet fully understood at the
system level. To target practical end-to-end IoT applications, IMC arrays must
be enclosed in heterogeneous programmable systems, introducing new system-level
challenges which we aim at addressing in this work. We present a heterogeneous
tightly-coupled clustered architecture integrating 8 RISC-V cores, an in-memory
computing accelerator (IMA), and digital accelerators. We benchmark the system
on a highly heterogeneous workload such as the Bottleneck layer from a
MobileNetV2, showing 11.5x performance and 9.5x energy efficiency improvements,
compared to highly optimized parallel execution on the cores. Furthermore, we
explore the requirements for end-to-end inference of a full mobile-grade DNN
(MobileNetV2) in terms of IMC array resources, by scaling up our heterogeneous
architecture to a multi-array accelerator. Our results show that our solution,
on the end-to-end inference of the MobileNetV2, is one order of magnitude
better in terms of execution latency than existing programmable architectures
and two orders of magnitude better than state-of-the-art heterogeneous
solutions integrating in-memory computing analog cores.
