Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees   Pre-trained language models like BERT achieve superior performances in
various NLP tasks without explicit consideration of syntactic information.
Meanwhile, syntactic information has been proved to be crucial for the success
of NLP applications. However, how to incorporate the syntax trees effectively
and efficiently into pre-trained Transformers is still unsettled. In this
paper, we address this problem by proposing a novel framework named
Syntax-BERT. This framework works in a plug-and-play mode and is applicable to
an arbitrary pre-trained checkpoint based on Transformer architecture.
Experiments on various datasets of natural language understanding verify the
effectiveness of syntax trees and achieve consistent improvement over multiple
pre-trained models, including BERT, RoBERTa, and T5.

37
The categories of this paper are: supr-con
7
The categories of this paper are: hep-th
21
The categories of this paper are: adap-org
2
The categories of this paper are: math
4
The categories of this paper are: cond-mat
12
The categories of this paper are: nucl-th
22
The categories of this paper are: chao-dyn
0
The categories of this paper are: hep-ph
31
The categories of this paper are: ao-sci
36
The categories of this paper are: cmp-lg
26
The categories of this paper are: bayes-an
17
The categories of this paper are: q-fin
30
The categories of this paper are: q-alg
13
The categories of this paper are: quant-ph
3
The categories of this paper are: physics
19
The categories of this paper are: eess