Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Contrastive Learning with Adversarial Perturbations for Conditional Text
  Generation   Recently, sequence-to-sequence (seq2seq) models with the Transformer
architecture have achieved remarkable performance on various conditional text
generation tasks, such as machine translation. However, most of them are
trained with teacher forcing with the ground truth label given at each time
step, without being exposed to incorrectly generated tokens during training,
which hurts its generalization to unseen inputs, that is known as the "exposure
bias" problem. In this work, we propose to mitigate the conditional text
generation problem by contrasting positive pairs with negative pairs, such that
the model is exposed to various valid or incorrect perturbations of the inputs,
for improved generalization. However, training the model with naive contrastive
learning framework using random non-target sequences as negative examples is
suboptimal, since they are easily distinguishable from the correct output,
especially so with models pretrained with large text corpora. Also, generating
positive examples requires domain-specific augmentation heuristics which may
not generalize over diverse domains. To tackle this problem, we propose a
principled method to generate positive and negative samples for contrastive
learning of seq2seq models. Specifically, we generate negative examples by
adding small perturbations to the input sequence to minimize its conditional
likelihood, and positive examples by adding large perturbations while enforcing
it to have a high conditional likelihood. Such "hard" positive and negative
pairs generated using our method guides the model to better distinguish correct
outputs from incorrect ones. We empirically show that our proposed method
significantly improves the generalization of the seq2seq on three text
generation tasks - machine translation, text summarization, and question
generation.

26
The categories of this paper are: bayes-an
37
The categories of this paper are: supr-con
10
The categories of this paper are: q-bio
18
The categories of this paper are: econ
22
The categories of this paper are: chao-dyn
21
The categories of this paper are: adap-org
27
The categories of this paper are: comp-gas
19
The categories of this paper are: eess
17
The categories of this paper are: q-fin
0
The categories of this paper are: hep-ph
16
The categories of this paper are: stat
14
The categories of this paper are: math-ph
4
The categories of this paper are: cond-mat
11
The categories of this paper are: hep-lat
25
The categories of this paper are: patt-sol
7
The categories of this paper are: hep-th