Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Accelerating Deep Learning Inference via Freezing   Over the last few years, Deep Neural Networks (DNNs) have become ubiquitous
owing to their high accuracy on real-world tasks. However, this increase in
accuracy comes at the cost of computationally expensive models leading to
higher prediction latencies. Prior efforts to reduce this latency such as
quantization, model distillation, and any-time prediction models typically
trade-off accuracy for performance. In this work, we observe that caching
intermediate layer outputs can help us avoid running all the layers of a DNN
for a sizeable fraction of inference requests. We find that this can
potentially reduce the number of effective layers by half for 91.58% of
CIFAR-10 requests run on ResNet-18. We present Freeze Inference, a system that
introduces approximate caching at each intermediate layer and we discuss
techniques to reduce the cache size and improve the cache hit rate. Finally, we
discuss some of the open research challenges in realizing such a design.

18
The categories of this paper are: econ
33
The categories of this paper are: plasm-ph
6
The categories of this paper are: astro-ph
20
The categories of this paper are: acc-phys
5
The categories of this paper are: gr-qc
10
The categories of this paper are: q-bio
29
The categories of this paper are: funct-an
13
The categories of this paper are: quant-ph
28
The categories of this paper are: alg-geom
17
The categories of this paper are: q-fin
31
The categories of this paper are: ao-sci
24
The categories of this paper are: dg-ga
3
The categories of this paper are: physics
32
The categories of this paper are: atom-ph
21
The categories of this paper are: adap-org
2
The categories of this paper are: math