Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Masking as an Efficient Alternative to Finetuning for Pretrained
  Language Models   We present an efficient method of utilizing pretrained language models, where
we learn selective binary masks for pretrained weights in lieu of modifying
them through finetuning. Extensive evaluations of masking BERT and RoBERTa on a
series of NLP tasks show that our masking scheme yields performance comparable
to finetuning, yet has a much smaller memory footprint when several tasks need
to be inferred simultaneously. Through intrinsic evaluations, we show that
representations computed by masked language models encode information necessary
for solving downstream tasks. Analyzing the loss landscape, we show that
masking and finetuning produce models that reside in minima that can be
connected by a line segment with nearly constant test accuracy. This confirms
that masking can be utilized as an efficient alternative to finetuning.

34
The categories of this paper are: chem-ph
21
The categories of this paper are: adap-org
0
The categories of this paper are: hep-ph
18
The categories of this paper are: econ
22
The categories of this paper are: chao-dyn
17
The categories of this paper are: q-fin
11
The categories of this paper are: hep-lat
15
The categories of this paper are: nucl-ex
26
The categories of this paper are: bayes-an
12
The categories of this paper are: nucl-th
19
The categories of this paper are: eess
7
The categories of this paper are: hep-th
5
The categories of this paper are: gr-qc
33
The categories of this paper are: plasm-ph
30
The categories of this paper are: q-alg
4
The categories of this paper are: cond-mat