Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer   There has been growing interest in parameter-efficient methods to apply
pre-trained language models to downstream tasks. Building on the Prompt Tuning
approach of Lester et al. (2021), which learns task-specific soft prompts to
condition a frozen pre-trained model to perform different tasks, we propose a
novel prompt-based transfer learning approach called SPoT: Soft Prompt
Transfer. SPoT first learns a prompt on one or more source tasks and then uses
it to initialize the prompt for a target task. We show that SPoT significantly
boosts the performance of Prompt Tuning across many tasks. More remarkably,
across all model sizes, SPoT matches or outperforms standard Model Tuning
(which fine-tunes all model parameters) on the SuperGLUE benchmark, while using
up to 27,000x fewer task-specific parameters. To understand where SPoT is most
effective, we conduct a large-scale study on task transferability with 26 NLP
tasks in 160 combinations, and demonstrate that many tasks can benefit each
other via prompt transfer. Finally, we propose an efficient retrieval approach
that interprets task prompts as task embeddings to identify similar tasks and
predict the most transferable source tasks for a novel target task.

10
The categories of this paper are: q-bio
23
The categories of this paper are: solv-int
21
The categories of this paper are: adap-org
30
The categories of this paper are: q-alg
16
The categories of this paper are: stat
24
The categories of this paper are: dg-ga
18
The categories of this paper are: econ
15
The categories of this paper are: nucl-ex
4
The categories of this paper are: cond-mat
13
The categories of this paper are: quant-ph
28
The categories of this paper are: alg-geom
7
The categories of this paper are: hep-th
6
The categories of this paper are: astro-ph
2
The categories of this paper are: math
14
The categories of this paper are: math-ph
25
The categories of this paper are: patt-sol