Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: SupCL-Seq: Supervised Contrastive Learning for Downstream Optimized
  Sequence Representations   While contrastive learning is proven to be an effective training strategy in
computer vision, Natural Language Processing (NLP) is only recently adopting it
as a self-supervised alternative to Masked Language Modeling (MLM) for
improving sequence representations. This paper introduces SupCL-Seq, which
extends the supervised contrastive learning from computer vision to the
optimization of sequence representations in NLP. By altering the dropout mask
probability in standard Transformer architectures, for every representation
(anchor), we generate augmented altered views. A supervised contrastive loss is
then utilized to maximize the system's capability of pulling together similar
samples (e.g., anchors and their altered views) and pushing apart the samples
belonging to the other classes. Despite its simplicity, SupCLSeq leads to large
gains in many sequence classification tasks on the GLUE benchmark compared to a
standard BERTbase, including 6% absolute improvement on CoLA, 5.4% on MRPC,
4.7% on RTE and 2.6% on STSB. We also show consistent gains over self
supervised contrastively learned representations, especially in non-semantic
tasks. Finally we show that these gains are not solely due to augmentation, but
rather to a downstream optimized sequence representation. Code:
https://github.com/hooman650/SupCL-Seq

18
The categories of this paper are: econ
10
The categories of this paper are: q-bio
15
The categories of this paper are: nucl-ex
23
The categories of this paper are: solv-int
17
The categories of this paper are: q-fin
30
The categories of this paper are: q-alg
2
The categories of this paper are: math
35
The categories of this paper are: mtrl-th
27
The categories of this paper are: comp-gas
14
The categories of this paper are: math-ph
20
The categories of this paper are: acc-phys
8
The categories of this paper are: hep-ex
31
The categories of this paper are: ao-sci
25
The categories of this paper are: patt-sol
0
The categories of this paper are: hep-ph
19
The categories of this paper are: eess