Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: A Simple Contrastive Learning Objective for Alleviating Neural Text
  Degeneration   The cross-entropy objective has proved to be an all-purpose training
objective for autoregressive language models (LMs). However, without
considering the penalization of problematic tokens, LMs trained using
cross-entropy exhibit text degeneration. To address this, unlikelihood training
has been proposed to force unlikely tokens to be assigned a low probability by
a LM. But unlikelihood does not consider the relationship between the label
tokens and the unlikely token candidates, thus showing marginal improvements in
degeneration. We propose a new contrastive token learning objective that
inherits the advantages of cross-entropy and unlikelihood training and avoids
their limitations. The key idea is to force a LM to generate high probabilities
for label tokens at each step while low probabilities of negative candidates.
Comprehensive experiments on language modeling and open-domain dialogue
generation tasks show that the proposed contrastive token objective yields less
repetitive texts, with a higher generation quality than unlikelihood training,
achieving the new state-of-the-art performance.

20
The categories of this paper are: acc-phys
25
The categories of this paper are: patt-sol
26
The categories of this paper are: bayes-an
0
The categories of this paper are: hep-ph
31
The categories of this paper are: ao-sci
28
The categories of this paper are: alg-geom
19
The categories of this paper are: eess
17
The categories of this paper are: q-fin
12
The categories of this paper are: nucl-th
14
The categories of this paper are: math-ph
36
The categories of this paper are: cmp-lg
37
The categories of this paper are: supr-con
23
The categories of this paper are: solv-int
3
The categories of this paper are: physics
29
The categories of this paper are: funct-an
11
The categories of this paper are: hep-lat