Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Multi-Level Contrastive Learning for Cross-Lingual Alignment   Cross-language pre-trained models such as multilingual BERT (mBERT) have
achieved significant performance in various cross-lingual downstream NLP tasks.
This paper proposes a multi-level contrastive learning (ML-CTL) framework to
further improve the cross-lingual ability of pre-trained models. The proposed
method uses translated parallel data to encourage the model to generate similar
semantic embeddings for different languages. However, unlike the sentence-level
alignment used in most previous studies, in this paper, we explicitly integrate
the word-level information of each pair of parallel sentences into contrastive
learning. Moreover, cross-zero noise contrastive estimation (CZ-NCE) loss is
proposed to alleviate the impact of the floating-point error in the training
process with a small batch size. The proposed method significantly improves the
cross-lingual transfer ability of our basic model (mBERT) and outperforms on
multiple zero-shot cross-lingual downstream tasks compared to the same-size
models in the Xtreme benchmark.

26
The categories of this paper are: bayes-an
20
The categories of this paper are: acc-phys
23
The categories of this paper are: solv-int
36
The categories of this paper are: cmp-lg
24
The categories of this paper are: dg-ga
9
The categories of this paper are: nlin
11
The categories of this paper are: hep-lat
4
The categories of this paper are: cond-mat
5
The categories of this paper are: gr-qc
0
The categories of this paper are: hep-ph
19
The categories of this paper are: eess
29
The categories of this paper are: funct-an
12
The categories of this paper are: nucl-th
6
The categories of this paper are: astro-ph
25
The categories of this paper are: patt-sol
35
The categories of this paper are: mtrl-th