Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Not All Attention Is All You Need   Beyond the success story of pre-trained language models (PrLMs) in recent
natural language processing, they are susceptible to over-fitting due to
unusual large model size. To this end, dropout serves as a therapy. However,
existing methods like random-based, knowledge-based and search-based dropout
are more general but less effective onto self-attention based models, which are
broadly chosen as the fundamental architecture of PrLMs. In this paper, we
propose a novel dropout method named AttendOut to let self-attention empowered
PrLMs capable of more robust task-specific tuning. We demonstrate that
state-of-the-art models with elaborate training design may achieve much
stronger results. We verify the universality of our approach on extensive
natural language processing tasks.

23
The categories of this paper are: solv-int
20
The categories of this paper are: acc-phys
3
The categories of this paper are: physics
6
The categories of this paper are: astro-ph
27
The categories of this paper are: comp-gas
34
The categories of this paper are: chem-ph
26
The categories of this paper are: bayes-an
0
The categories of this paper are: hep-ph
33
The categories of this paper are: plasm-ph
29
The categories of this paper are: funct-an
22
The categories of this paper are: chao-dyn
18
The categories of this paper are: econ
8
The categories of this paper are: hep-ex
12
The categories of this paper are: nucl-th
7
The categories of this paper are: hep-th
13
The categories of this paper are: quant-ph