  The recent ground-breaking advances in deep learning networks ( DNNs ) make
them attractive for embedded systems. However, it can take a long time for DNNs
to make an inference on resource-limited embedded devices. Offloading the
computation into the cloud is often infeasible due to privacy concerns, high
latency, or the lack of connectivity. As such, there is a critical need to find
a way to effectively execute the DNN models locally on the devices. This paper
presents an adaptive scheme to determine which DNN model to use for a given
input, by considering the desired accuracy and inference time. Our approach
employs machine learning to develop a predictive model to quickly select a
pre-trained DNN to use for a given input and the optimization constraint. We
achieve this by first training off-line a predictive model, and then use the
learnt model to select a DNN model to use for new, unseen inputs. We apply our
approach to the image classification task and evaluate it on a Jetson TX2
embedded deep learning platform using the ImageNet ILSVRC 2012 validation
dataset. We consider a range of influential DNN models. Experimental results
show that our approach achieves a 7.52% improvement in inference accuracy, and
a 1.8x reduction in inference time over the most-capable single DNN model.
