Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: On Transferability of Prompt Tuning for Natural Language Processing   Prompt tuning (PT) is a promising parameter-efficient method to utilize
extremely large pre-trained language models (PLMs), which can achieve
comparable performance to full-parameter fine-tuning by only tuning a few soft
prompts. However, PT requires much more training time than fine-tuning.
Intuitively, knowledge transfer can help to improve the efficiency. To explore
whether we can improve PT via prompt transfer, we empirically investigate the
transferability of soft prompts across different downstream tasks and PLMs in
this work. We find that (1) in zero-shot setting, trained soft prompts can
effectively transfer to similar tasks on the same PLM and also to other PLMs
with a cross-model projector trained on similar tasks; (2) when used as
initialization, trained soft prompts of similar tasks and projected prompts of
other PLMs can significantly accelerate training and also improve the
performance of PT. Moreover, to explore what decides prompt transferability, we
investigate various transferability indicators and find that the overlapping
rate of activated neurons strongly reflects the transferability, which suggests
how the prompts stimulate PLMs is essential. Our findings show that prompt
transfer is promising for improving PT, and further research shall focus more
on prompts' stimulation to PLMs. The source code can be obtained from
https://github.com/thunlp/Prompt-Transferability.

0
The categories of this paper are: hep-ph
31
The categories of this paper are: ao-sci
27
The categories of this paper are: comp-gas
32
The categories of this paper are: atom-ph
15
The categories of this paper are: nucl-ex
17
The categories of this paper are: q-fin
26
The categories of this paper are: bayes-an
37
The categories of this paper are: supr-con
6
The categories of this paper are: astro-ph
33
The categories of this paper are: plasm-ph
4
The categories of this paper are: cond-mat
24
The categories of this paper are: dg-ga
10
The categories of this paper are: q-bio
2
The categories of this paper are: math
29
The categories of this paper are: funct-an
34
The categories of this paper are: chem-ph