Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Geographic Adaptation of Pretrained Language Models   Geographic linguistic features are commonly used to improve the performance
of pretrained language models (PLMs) on NLP tasks where geographic knowledge is
intuitively beneficial (e.g., geolocation prediction and dialect feature
prediction). Existing work, however, leverages such geographic information in
task-specific fine-tuning, failing to incorporate it into PLMs' geo-linguistic
knowledge, which would make it transferable across different tasks. In this
work, we introduce an approach to task-agnostic geoadaptation of PLMs that
forces the PLM to learn associations between linguistic phenomena and
geographic locations. More specifically, geoadaptation is an intermediate
training step that couples masked language modeling and geolocation prediction
in a dynamic multitask learning setup. In our experiments, we geoadapt BERTi\'c
-- a PLM for Bosnian, Croatian, Montenegrin, and Serbian (BCMS) -- using a
corpus of geotagged BCMS tweets. Evaluation on three different tasks, namely
unsupervised (zero-shot) and supervised geolocation prediction and
(unsupervised) prediction of dialect features, shows that our geoadaptation
approach is very effective: e.g., we obtain new state-of-the-art performance in
supervised geolocation prediction and report massive gains over geographically
uninformed PLMs on zero-shot geolocation prediction.

21
The categories of this paper are: adap-org
11
The categories of this paper are: hep-lat
28
The categories of this paper are: alg-geom
15
The categories of this paper are: nucl-ex
26
The categories of this paper are: bayes-an
4
The categories of this paper are: cond-mat
9
The categories of this paper are: nlin
36
The categories of this paper are: cmp-lg
32
The categories of this paper are: atom-ph
18
The categories of this paper are: econ
27
The categories of this paper are: comp-gas
37
The categories of this paper are: supr-con
3
The categories of this paper are: physics
34
The categories of this paper are: chem-ph
14
The categories of this paper are: math-ph
29
The categories of this paper are: funct-an