Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via
  Adaptive Gradient Gating for Rare Token Embeddings   Recent studies have determined that the learned token embeddings of
large-scale neural language models are degenerated to be anisotropic with a
narrow-cone shape. This phenomenon, called the representation degeneration
problem, facilitates an increase in the overall similarity between token
embeddings that negatively affect the performance of the models. Although the
existing methods that address the degeneration problem based on observations of
the phenomenon triggered by the problem improves the performance of the text
generation, the training dynamics of token embeddings behind the degeneration
problem are still not explored. In this study, we analyze the training dynamics
of the token embeddings focusing on rare token embedding. We demonstrate that
the specific part of the gradient for rare token embeddings is the key cause of
the degeneration problem for all tokens during training stage. Based on the
analysis, we propose a novel method called, adaptive gradient gating (AGG). AGG
addresses the degeneration problem by gating the specific part of the gradient
for rare token embeddings. Experimental results from language modeling, word
similarity, and machine translation tasks quantitatively and qualitatively
verify the effectiveness of AGG.

11
The categories of this paper are: hep-lat
25
The categories of this paper are: patt-sol
2
The categories of this paper are: math
3
The categories of this paper are: physics
16
The categories of this paper are: stat
20
The categories of this paper are: acc-phys
8
The categories of this paper are: hep-ex
27
The categories of this paper are: comp-gas
32
The categories of this paper are: atom-ph
37
The categories of this paper are: supr-con
9
The categories of this paper are: nlin
22
The categories of this paper are: chao-dyn
18
The categories of this paper are: econ
19
The categories of this paper are: eess
10
The categories of this paper are: q-bio
35
The categories of this paper are: mtrl-th