  To keep up with the ever-growing performance demand of neural networks,
specialized hardware (HW) accelerators are shifting towards multi-core and
chiplet architectures. So far, these multi-accelerator systems exploit the
increased parallelism by pipelining different NN layers across input batches on
different cores to increase throughput. Yet, when pursuing this with
non-batched layer-by-layer scheduling of latency-critical applications, this
fails to fully exploit the available HW resources towards energy-efficient
execution at the edge.
  This work, therefore, enables fine-grained depth-first scheduling of
layer-fused DNNs onto multi-core architectures through an open-source modeling
framework called Stream. Stream is capable of representing a wide range of
scheduling granularities and HW architectures and optimizes execution schedules
towards minimal energy, minimal latency and/or minimal memory footprint for
constrained edge devices. We validate against three SotA HW implementations
employing layer-fused scheduling showing tight matching with measured
efficiencies. Using Stream in further explorations, we demonstrate that
high-level architectural decisions greatly impact hardware efficiency under the
fine-grained scheduling paradigm, reducing the energy-delay product from 2.4x
for single-core architectures to up to 30x for heterogeneous multi-core
architectures compared to the traditional scheduling at layer granularity.
