Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: TinyTL: Reduce Activations, Not Trainable Parameters for Efficient
  On-Device Learning   On-device learning enables edge devices to continually adapt the AI models to
new data, which requires a small memory footprint to fit the tight memory
constraint of edge devices. Existing work solves this problem by reducing the
number of trainable parameters. However, this doesn't directly translate to
memory saving since the major bottleneck is the activations, not parameters. In
this work, we present Tiny-Transfer-Learning (TinyTL) for memory-efficient
on-device learning. TinyTL freezes the weights while only learns the bias
modules, thus no need to store the intermediate activations. To maintain the
adaptation capacity, we introduce a new memory-efficient bias module, the lite
residual module, to refine the feature extractor by learning small residual
feature maps adding only 3.8% memory overhead. Extensive experiments show that
TinyTL significantly saves the memory (up to 6.5x) with little accuracy loss
compared to fine-tuning the full network. Compared to fine-tuning the last
layer, TinyTL provides significant accuracy improvements (up to 34.1%) with
little memory overhead. Furthermore, combined with feature extractor
adaptation, TinyTL provides 7.3-12.9x memory saving without sacrificing
accuracy compared to fine-tuning the full Inception-V3.

25
The categories of this paper are: patt-sol
19
The categories of this paper are: eess
13
The categories of this paper are: quant-ph
17
The categories of this paper are: q-fin
3
The categories of this paper are: physics
15
The categories of this paper are: nucl-ex
0
The categories of this paper are: hep-ph
31
The categories of this paper are: ao-sci
26
The categories of this paper are: bayes-an
21
The categories of this paper are: adap-org
16
The categories of this paper are: stat
22
The categories of this paper are: chao-dyn
23
The categories of this paper are: solv-int
12
The categories of this paper are: nucl-th
28
The categories of this paper are: alg-geom
29
The categories of this paper are: funct-an