Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: DecBERT: Enhancing the Language Understanding of BERT with Causal
  Attention Masks   Since 2017, the Transformer-based models play critical roles in various
downstream Natural Language Processing tasks. However, a common limitation of
the attention mechanism utilized in Transformer Encoder is that it cannot
automatically capture the information of word order, so explicit position
embeddings are generally required to be fed into the target model. In contrast,
Transformer Decoder with the causal attention masks is naturally sensitive to
the word order. In this work, we focus on improving the position encoding
ability of BERT with the causal attention masks. Furthermore, we propose a new
pre-trained language model DecBERT and evaluate it on the GLUE benchmark.
Experimental results show that (1) the causal attention mask is effective for
BERT on the language understanding tasks; (2) our DecBERT model without
position embeddings achieve comparable performance on the GLUE benchmark; and
(3) our modification accelerates the pre-training process and DecBERT w/ PE
achieves better overall performance than the baseline systems when pre-training
with the same amount of computational resources.

4
The categories of this paper are: cond-mat
26
The categories of this paper are: bayes-an
36
The categories of this paper are: cmp-lg
20
The categories of this paper are: acc-phys
31
The categories of this paper are: ao-sci
9
The categories of this paper are: nlin
17
The categories of this paper are: q-fin
28
The categories of this paper are: alg-geom
30
The categories of this paper are: q-alg
18
The categories of this paper are: econ
21
The categories of this paper are: adap-org
37
The categories of this paper are: supr-con
27
The categories of this paper are: comp-gas
32
The categories of this paper are: atom-ph
8
The categories of this paper are: hep-ex
15
The categories of this paper are: nucl-ex