  Advancing the utility of social media data for research applications requires
methods for automatically detecting demographic information about social media
study populations, including users' age. The objective of this study was to
develop and evaluate a method that automatically identifies the exact age of
users based on self-reports in their tweets. Our end-to-end automatic natural
language processing (NLP) pipeline, ReportAGE, includes query patterns to
retrieve tweets that potentially mention an age, a classifier to distinguish
retrieved tweets that self-report the user's exact age ("age" tweets) and those
that do not ("no age" tweets), and rule-based extraction to identify the age.
To develop and evaluate ReportAGE, we manually annotated 11,000 tweets that
matched the query patterns. Based on 1000 tweets that were annotated by all
five annotators, inter-annotator agreement (Fleiss' kappa) was 0.80 for
distinguishing "age" and "no age" tweets, and 0.95 for identifying the exact
age among the "age" tweets on which the annotators agreed. A deep neural
network classifier, based on a RoBERTa-Large pretrained model, achieved the
highest F1-score of 0.914 (precision = 0.905, recall = 0.942) for the "age"
class. When the age extraction was evaluated using the classifier's
predictions, it achieved an F1-score of 0.855 (precision = 0.805, recall =
0.914) for the "age" class. When it was evaluated directly on the held-out test
set, it achieved an F1-score of 0.931 (precision = 0.873, recall = 0.998) for
the "age" class. We deployed ReportAGE on more than 1.2 billion tweets posted
by 245,927 users, and predicted ages for 132,637 (54%) of them. Scaling the
detection of exact age to this large number of users can advance the utility of
social media data for research applications that do not align with the
predefined age groupings of extant binary or multi-class classification
approaches.
