Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Cross-Thought for Sentence Encoder Pre-training   In this paper, we propose Cross-Thought, a novel approach to pre-training
sequence encoder, which is instrumental in building reusable sequence
embeddings for large-scale NLP tasks such as question answering. Instead of
using the original signals of full sentences, we train a Transformer-based
sequence encoder over a large set of short sequences, which allows the model to
automatically select the most useful information for predicting masked words.
Experiments on question answering and textual entailment tasks demonstrate that
our pre-trained encoder can outperform state-of-the-art encoders trained with
continuous sentence signals as well as traditional masked language modeling
baselines. Our proposed approach also achieves new state of the art on HotpotQA
(full-wiki setting) by improving intermediate information retrieval
performance.

13
The categories of this paper are: quant-ph
27
The categories of this paper are: comp-gas
5
The categories of this paper are: gr-qc
28
The categories of this paper are: alg-geom
32
The categories of this paper are: atom-ph
31
The categories of this paper are: ao-sci
14
The categories of this paper are: math-ph
10
The categories of this paper are: q-bio
9
The categories of this paper are: nlin
22
The categories of this paper are: chao-dyn
25
The categories of this paper are: patt-sol
15
The categories of this paper are: nucl-ex
19
The categories of this paper are: eess
17
The categories of this paper are: q-fin
7
The categories of this paper are: hep-th
3
The categories of this paper are: physics