Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: AUBER: Automated BERT Regularization   How can we effectively regularize BERT? Although BERT proves its
effectiveness in various downstream natural language processing tasks, it often
overfits when there are only a small number of training instances. A promising
direction to regularize BERT is based on pruning its attention heads based on a
proxy score for head importance. However, heuristic-based methods are usually
suboptimal since they predetermine the order by which attention heads are
pruned. In order to overcome such a limitation, we propose AUBER, an effective
regularization method that leverages reinforcement learning to automatically
prune attention heads from BERT. Instead of depending on heuristics or
rule-based policies, AUBER learns a pruning policy that determines which
attention heads should or should not be pruned for regularization. Experimental
results show that AUBER outperforms existing pruning methods by achieving up to
10% better accuracy. In addition, our ablation study empirically demonstrates
the effectiveness of our design choices for AUBER.

5
The categories of this paper are: gr-qc
34
The categories of this paper are: chem-ph
37
The categories of this paper are: supr-con
24
The categories of this paper are: dg-ga
2
The categories of this paper are: math
11
The categories of this paper are: hep-lat
7
The categories of this paper are: hep-th
31
The categories of this paper are: ao-sci
14
The categories of this paper are: math-ph
25
The categories of this paper are: patt-sol
8
The categories of this paper are: hep-ex
28
The categories of this paper are: alg-geom
10
The categories of this paper are: q-bio
21
The categories of this paper are: adap-org
35
The categories of this paper are: mtrl-th
32
The categories of this paper are: atom-ph