Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Distributionally Robust Language Modeling   Language models are generally trained on data spanning a wide range of topics
(e.g., news, reviews, fiction), but they might be applied to an a priori
unknown target distribution (e.g., restaurant reviews). In this paper, we first
show that training on text outside the test distribution can degrade test
performance when using standard maximum likelihood (MLE) training. To remedy
this without the knowledge of the test distribution, we propose an approach
which trains a model that performs well over a wide range of potential test
distributions. In particular, we derive a new distributionally robust
optimization (DRO) procedure which minimizes the loss of the model over the
worst-case mixture of topics with sufficient overlap with the training
distribution. Our approach, called topic conditional value at risk (topic
CVaR), obtains a 5.5 point perplexity reduction over MLE when the language
models are trained on a mixture of Yelp reviews and news and tested only on
reviews.

31
The categories of this paper are: ao-sci
5
The categories of this paper are: gr-qc
11
The categories of this paper are: hep-lat
27
The categories of this paper are: comp-gas
35
The categories of this paper are: mtrl-th
22
The categories of this paper are: chao-dyn
25
The categories of this paper are: patt-sol
36
The categories of this paper are: cmp-lg
29
The categories of this paper are: funct-an
7
The categories of this paper are: hep-th
8
The categories of this paper are: hep-ex
19
The categories of this paper are: eess
13
The categories of this paper are: quant-ph
21
The categories of this paper are: adap-org
20
The categories of this paper are: acc-phys
14
The categories of this paper are: math-ph