Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: A Comparative Study of Transformers on Word Sense Disambiguation   Recent years of research in Natural Language Processing (NLP) have witnessed
dramatic growth in training large models for generating context-aware language
representations. In this regard, numerous NLP systems have leveraged the power
of neural network-based architectures to incorporate sense information in
embeddings, resulting in Contextualized Word Embeddings (CWEs). Despite this
progress, the NLP community has not witnessed any significant work performing a
comparative study on the contextualization power of such architectures. This
paper presents a comparative study and an extensive analysis of nine widely
adopted Transformer models. These models are BERT, CTRL, DistilBERT,
OpenAI-GPT, OpenAI-GPT2, Transformer-XL, XLNet, ELECTRA, and ALBERT. We
evaluate their contextualization power using two lexical sample Word Sense
Disambiguation (WSD) tasks, SensEval-2 and SensEval-3. We adopt a simple yet
effective approach to WSD that uses a k-Nearest Neighbor (kNN) classification
on CWEs. Experimental results show that the proposed techniques also achieve
superior results over the current state-of-the-art on both the WSD tasks

5
The categories of this paper are: gr-qc
13
The categories of this paper are: quant-ph
0
The categories of this paper are: hep-ph
34
The categories of this paper are: chem-ph
2
The categories of this paper are: math
3
The categories of this paper are: physics
37
The categories of this paper are: supr-con
36
The categories of this paper are: cmp-lg
19
The categories of this paper are: eess
10
The categories of this paper are: q-bio
12
The categories of this paper are: nucl-th
11
The categories of this paper are: hep-lat
27
The categories of this paper are: comp-gas
23
The categories of this paper are: solv-int
15
The categories of this paper are: nucl-ex
18
The categories of this paper are: econ