Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: On Transferability of Bias Mitigation Effects in Language Model
  Fine-Tuning   Fine-tuned language models have been shown to exhibit biases against
protected groups in a host of modeling tasks such as text classification and
coreference resolution. Previous works focus on detecting these biases,
reducing bias in data representations, and using auxiliary training objectives
to mitigate bias during fine-tuning. Although these techniques achieve bias
reduction for the task and domain at hand, the effects of bias mitigation may
not directly transfer to new tasks, requiring additional data collection and
customized annotation of sensitive attributes, and re-evaluation of appropriate
fairness metrics. We explore the feasibility and benefits of upstream bias
mitigation (UBM) for reducing bias on downstream tasks, by first applying bias
mitigation to an upstream model through fine-tuning and subsequently using it
for downstream fine-tuning. We find, in extensive experiments across hate
speech detection, toxicity detection, occupation prediction, and coreference
resolution tasks over various bias factors, that the effects of UBM are indeed
transferable to new downstream tasks or domains via fine-tuning, creating less
biased downstream models than directly fine-tuning on the downstream task or
transferring from a vanilla upstream model. Though challenges remain, we show
that UBM promises more efficient and accessible bias mitigation in LM
fine-tuning.

21
The categories of this paper are: adap-org
5
The categories of this paper are: gr-qc
0
The categories of this paper are: hep-ph
34
The categories of this paper are: chem-ph
17
The categories of this paper are: q-fin
25
The categories of this paper are: patt-sol
13
The categories of this paper are: quant-ph
27
The categories of this paper are: comp-gas
12
The categories of this paper are: nucl-th
33
The categories of this paper are: plasm-ph
35
The categories of this paper are: mtrl-th
14
The categories of this paper are: math-ph
31
The categories of this paper are: ao-sci
11
The categories of this paper are: hep-lat
30
The categories of this paper are: q-alg
28
The categories of this paper are: alg-geom