Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Low Frequency Names Exhibit Bias and Overfitting in Contextualizing
  Language Models   We use a dataset of U.S. first names with labels based on predominant gender
and racial group to examine the effect of training corpus frequency on
tokenization, contextualization, similarity to initial representation, and bias
in BERT, GPT-2, T5, and XLNet. We show that predominantly female and non-white
names are less frequent in the training corpora of these four language models.
We find that infrequent names are more self-similar across contexts, with
Spearman's r between frequency and self-similarity as low as -.763. Infrequent
names are also less similar to initial representation, with Spearman's r
between frequency and linear centered kernel alignment (CKA) similarity to
initial representation as high as .702. Moreover, we find Spearman's r between
racial bias and name frequency in BERT of .492, indicating that lower-frequency
minority group names are more associated with unpleasantness. Representations
of infrequent names undergo more processing, but are more self-similar,
indicating that models rely on less context-informed representations of
uncommon and minority names which are overfit to a lower number of observed
contexts.

23
The categories of this paper are: solv-int
25
The categories of this paper are: patt-sol
20
The categories of this paper are: acc-phys
9
The categories of this paper are: nlin
5
The categories of this paper are: gr-qc
28
The categories of this paper are: alg-geom
22
The categories of this paper are: chao-dyn
7
The categories of this paper are: hep-th
32
The categories of this paper are: atom-ph
10
The categories of this paper are: q-bio
24
The categories of this paper are: dg-ga
3
The categories of this paper are: physics
14
The categories of this paper are: math-ph
2
The categories of this paper are: math
4
The categories of this paper are: cond-mat
37
The categories of this paper are: supr-con