  Deep neural network (DNN) inference is increasingly being executed on mobile
and embedded platforms due to low latency and better privacy. However,
efficient deployment on these platforms is challenging due to the intensive
computation and memory access. We propose a holistic system design for DNN
performance and energy optimisation, combining the trade-off opportunities in
both algorithms and hardware. The system can be viewed as three abstract
layers: the device layer contains heterogeneous computing resources; the
application layer has multiple concurrent workloads; and the runtime resource
management layer monitors the dynamically changing algorithms' performance
targets as well as hardware resources and constraints, and tries to meet them
by tuning the algorithm and hardware at the same time. Moreover, We illustrate
the runtime approach through a dynamic version of 'once-for-all network'
(namely Dynamic-OFA), which can scale the ConvNet architecture to fit
heterogeneous computing resources efficiently and has good generalisation for
different model architectures such as Transformer. Compared to the
state-of-the-art Dynamic DNNs, our experimental results using ImageNet on a
Jetson Xavier NX show that the Dynamic-OFA is up to 3.5x (CPU), 2.4x (GPU)
faster for similar ImageNet Top-1 accuracy, or 3.8% (CPU), 5.1% (GPU) higher
accuracy at similar latency. Furthermore, compared with Linux governor (e.g.
performance, schedutil), our runtime approach reduces the energy consumption by
16.5% at similar latency.
