Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained
  Language Models   Recent work indicated that pretrained language models (PLMs) such as BERT and
RoBERTa can be transformed into effective sentence and word encoders even via
simple self-supervised techniques. Inspired by this line of work, in this paper
we propose a fully unsupervised approach to improving word-in-context (WiC)
representations in PLMs, achieved via a simple and efficient WiC-targeted
fine-tuning procedure: MirrorWiC. The proposed method leverages only raw texts
sampled from Wikipedia, assuming no sense-annotated data, and learns
context-aware word representations within a standard contrastive learning
setup. We experiment with a series of standard and comprehensive WiC benchmarks
across multiple languages. Our proposed fully unsupervised MirrorWiC models
obtain substantial gains over off-the-shelf PLMs across all monolingual,
multilingual and cross-lingual setups. Moreover, on some standard WiC
benchmarks, MirrorWiC is even on-par with supervised models fine-tuned with
in-task data and sense labels.

23
The categories of this paper are: solv-int
32
The categories of this paper are: atom-ph
30
The categories of this paper are: q-alg
16
The categories of this paper are: stat
6
The categories of this paper are: astro-ph
34
The categories of this paper are: chem-ph
8
The categories of this paper are: hep-ex
37
The categories of this paper are: supr-con
14
The categories of this paper are: math-ph
33
The categories of this paper are: plasm-ph
28
The categories of this paper are: alg-geom
7
The categories of this paper are: hep-th
31
The categories of this paper are: ao-sci
2
The categories of this paper are: math
18
The categories of this paper are: econ
21
The categories of this paper are: adap-org