Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: The SAME score: Improved cosine based bias score for word embeddings   Over the last years, word and sentence embeddings have established as text
preprocessing for all kinds of NLP tasks and improved performances in these
tasks significantly. Unfortunately, it has also been shown that these
embeddings inherit various kinds of biases from the training data and thereby
pass on biases present in society to NLP solutions. Many papers attempted to
quantify bias in word or sentence embeddings to evaluate debiasing methods or
compare different embedding models, often with cosine-based scores. However,
some works have raised doubts about these scores showing that even though they
report low biases, biases persist and can be shown with other tests. In fact,
there is a great variety of bias scores or tests proposed in the literature
without any consensus on the optimal solutions. We lack works that study the
behavior of bias scores and elaborate their advantages and disadvantages. In
this work, we will explore different cosine-based bias scores. We provide a
bias definition based on the ideas from the literature and derive novel
requirements for bias scores. Furthermore, we thoroughly investigate the
existing cosine-based scores and their limitations in order to show why these
scores fail to report biases in some situations. Finally, we propose a new bias
score, SAME, to address the shortcomings of existing bias scores and show
empirically that SAME is better suited to quantify biases in word embeddings.

34
The categories of this paper are: chem-ph
13
The categories of this paper are: quant-ph
35
The categories of this paper are: mtrl-th
19
The categories of this paper are: eess
27
The categories of this paper are: comp-gas
20
The categories of this paper are: acc-phys
10
The categories of this paper are: q-bio
5
The categories of this paper are: gr-qc
17
The categories of this paper are: q-fin
9
The categories of this paper are: nlin
29
The categories of this paper are: funct-an
2
The categories of this paper are: math
0
The categories of this paper are: hep-ph
14
The categories of this paper are: math-ph
23
The categories of this paper are: solv-int
32
The categories of this paper are: atom-ph