  A rising research challenge is running costly machine learning (ML) networks
locally on resource-constrained edge devices. ML networks with large
convolutional layers can easily exceed available memory, increasing latency due
to excessive OS swapping. Previous memory reduction techniques such as pruning
and quantization reduce model accuracy and often require retraining.
Alternatively, distributed methods partition the convolutions into equivalent
smaller sub-computations, but the implementations introduce communication costs
and require a network of devices. Distributed partitioning approaches can,
however, also be used to run in a reduced memory footprint on a single device
by subdividing the network into smaller operations. In this paper, we extend
prior work on distributed partitioning into a memory-aware execution on a
single device. Our approach extends prior fusing strategies to allow for
multiple groups of convolutional layers that are fused and tiled independently.
This enables trading off overhead versus data reuse in order to specifically
reduces memory footprint. We propose a memory usage predictor coupled with a
search algorithm to provide optimized fusing and tiling configurations for an
arbitrary set of convolutional layers. When applied to the YOLOv2 object
detection network, results show that our approach can run in less than half the
memory, and with a speedup of up to 2.78 under severe memory constraints.
Additionally, our algorithm will return a configuration with a latency that is
within 6% of the best latency measured in a manual search.
