Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: LiST: Lite Self-training Makes Efficient Few-shot Learners   We present a new method LiST for efficient fine-tuning of large pre-trained
language models (PLMs) in few-shot learning settings. LiST significantly
improves over recent methods that adopt prompt fine-tuning using two key
techniques. The first one is the use of self-training to leverage large amounts
of unlabeled data for prompt-tuning to significantly boost the model
performance in few-shot settings. We use self-training in conjunction with
meta-learning for re-weighting noisy pseudo-prompt labels. However, traditional
self-training is expensive as it requires updating all the model parameters
repetitively. Therefore, we use a second technique for light-weight fine-tuning
where we introduce a small number of task-specific adapter parameters that are
fine-tuned during self-training while keeping the PLM encoder frozen. This also
significantly reduces the overall model footprint across several tasks that can
now share a common PLM encoder as backbone for inference. Combining the above
techniques, LiST not only improves the model performance for few-shot learning
on target domains but also reduces the model memory footprint. We present a
comprehensive study on six NLU tasks to validate the effectiveness of LiST. The
results show that LiST improves by 35% over classic fine-tuning methods and 6%
over prompt-tuning with 96% reduction in number of trainable parameters when
fine-tuned with no more than 30 labeled examples from each target domain.

23
The categories of this paper are: solv-int
10
The categories of this paper are: q-bio
36
The categories of this paper are: cmp-lg
30
The categories of this paper are: q-alg
4
The categories of this paper are: cond-mat
24
The categories of this paper are: dg-ga
20
The categories of this paper are: acc-phys
19
The categories of this paper are: eess
6
The categories of this paper are: astro-ph
9
The categories of this paper are: nlin
21
The categories of this paper are: adap-org
13
The categories of this paper are: quant-ph
8
The categories of this paper are: hep-ex
29
The categories of this paper are: funct-an
27
The categories of this paper are: comp-gas
18
The categories of this paper are: econ