Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Temporal Adaptation of BERT and Performance on Downstream Document
  Classification: Insights from Social Media   Language use differs between domains and even within a domain, language use
changes over time. For pre-trained language models like BERT, domain adaptation
through continued pre-training has been shown to improve performance on
in-domain downstream tasks. In this article, we investigate whether temporal
adaptation can bring additional benefits. For this purpose, we introduce a
corpus of social media comments sampled over three years. It contains
unlabelled data for adaptation and evaluation on an upstream masked language
modelling task as well as labelled data for fine-tuning and evaluation on a
downstream document classification task. We find that temporality matters for
both tasks: temporal adaptation improves upstream and temporal fine-tuning
downstream task performance. Time-specific models generally perform better on
past than on future test sets, which matches evidence on the bursty usage of
topical words. However, adapting BERT to time and domain does not improve
performance on the downstream task over only adapting to domain. Token-level
analysis shows that temporal adaptation captures event-driven changes in
language use in the downstream task, but not those changes that are actually
relevant to task performance. Based on our findings, we discuss when temporal
adaptation may be more effective.

35
The categories of this paper are: mtrl-th
30
The categories of this paper are: q-alg
22
The categories of this paper are: chao-dyn
34
The categories of this paper are: chem-ph
5
The categories of this paper are: gr-qc
37
The categories of this paper are: supr-con
10
The categories of this paper are: q-bio
29
The categories of this paper are: funct-an
2
The categories of this paper are: math
21
The categories of this paper are: adap-org
23
The categories of this paper are: solv-int
11
The categories of this paper are: hep-lat
27
The categories of this paper are: comp-gas
16
The categories of this paper are: stat
26
The categories of this paper are: bayes-an
31
The categories of this paper are: ao-sci