Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Parameter-efficient Multi-task Fine-tuning for Transformers via Shared
  Hypernetworks   State-of-the-art parameter-efficient fine-tuning methods rely on introducing
adapter modules between the layers of a pretrained language model. However,
such modules are trained separately for each task and thus do not enable
sharing information across tasks. In this paper, we show that we can learn
adapter parameters for all layers and tasks by generating them using shared
hypernetworks, which condition on task, adapter position, and layer id in a
transformer model. This parameter-efficient multi-task learning framework
allows us to achieve the best of both worlds by sharing knowledge across tasks
via hypernetworks while enabling the model to adapt to each individual task
through task-specific adapters. Experiments on the well-known GLUE benchmark
show improved performance in multi-task learning while adding only 0.29%
parameters per task. We additionally demonstrate substantial performance
improvements in few-shot domain generalization across a variety of tasks. Our
code is publicly available in https://github.com/rabeehk/hyperformer.

8
The categories of this paper are: hep-ex
5
The categories of this paper are: gr-qc
26
The categories of this paper are: bayes-an
28
The categories of this paper are: alg-geom
33
The categories of this paper are: plasm-ph
4
The categories of this paper are: cond-mat
0
The categories of this paper are: hep-ph
32
The categories of this paper are: atom-ph
6
The categories of this paper are: astro-ph
23
The categories of this paper are: solv-int
10
The categories of this paper are: q-bio
9
The categories of this paper are: nlin
15
The categories of this paper are: nucl-ex
35
The categories of this paper are: mtrl-th
16
The categories of this paper are: stat
24
The categories of this paper are: dg-ga