# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# ## Fairness Analysis of NamSor's Gender API Endpoint using Aequitas
#
# Part I: Fairness of Gender API Endpoint by Gender and Ethnicity

# +
import pandas as pd
import seaborn as sns
from aequitas.group import Group
from aequitas.bias import Bias
from aequitas.fairness import Fairness
from aequitas.plotting import Plot
import pdfkit as pdf

# import warnings; warnings.simplefilter('ignore')

# %matplotlib inline
# -

df = pd.read_csv("data/compas_ethnicity_predictions.csv")
df.head()

# Non String columns will lead to problems later so we have to find out if there are any
non_attr_cols = ['id', 'model_id', 'entity_id', 'score', 'label_value', 'rank_abs', 'rank_pct']
attr_cols = df.columns[~df.columns.isin(non_attr_cols)]  # index of the columns that are
df.columns[(df.dtypes != object) & (df.dtypes != str) & (df.columns.isin(attr_cols))]

# And delete them.
df = df.drop(['Unnamed: 0'], axis=1)
df.head()

# we only want to look at ethnicity here, since that is what we calculated label_value for
df = df.drop(['race_pred', 'first', 'last'], axis=1) # if we don't drop the tables, Aequitas thinks these are attributes by which groups should be separated

# now we remove groups that are too small
df = df[df.race.isin(['African-American', 'Caucasian', 'Hispanic'])]

df.shape

df["combined_attribute"] = df["sex"] + " " + df["race"]

df.head()

# ## Group Metrics
# First, let's calculate the group metrics like tp, fn, and so on. For determining which score is considered to be a prediction of the classification being correct we will use a score treshold that is the maximum score, or minimum probabilityCalibrated. We do this, because NamSor does make a prediction starting at that score and we want to look at the raw prediction first. We will look at different scores later on to see which score produces which fairness.

t = df[["score"]].max()[0]
g = Group()
xtab, _ = g.get_crosstabs(df, attr_cols=["sex", "race", "combined_attribute"], score_thresholds= {'score': [t]})
absolute_metrics = g.list_absolute_metrics(xtab)

df[["score"]].max()[0]

xtab[[col for col in xtab.columns if col not in absolute_metrics]]


def _color_red_or_green_or_orange(val): # https://stackoverflow.com/questions/28075699/coloring-cells-in-pandas
    try:
        if(val > 0.8 and val < 1.2):
            color = 'green'
        elif(val < 0.8):
            color = 'orange'
        else: color = 'red'
        return 'color: %s' % color
    except:
        return 'color: black'


xtab.columns

xtab #[['attribute_name', 'attribute_value'] + absolute_metrics]

# ## Disparities of Group Metrics

#aq_palette = sns.diverging_palette(240, 10, n=9) #sns.diverging_palette(225, 35, n=2)
b = Bias()

# +
bdf = b.get_disparity_predefined_groups(xtab, original_df=df, 
                                        ref_groups_dict={'sex':'Male', 'race':'Caucasian', 'combined_attribute':'Male Caucasian'})

bdf_overview = bdf[["attribute_name", "attribute_value", "precision_disparity", "fdr_disparity"]]

bdf_overview.style.applymap(_color_red_or_green_or_orange) #
# -

# ## Fairness Metrics

f = Fairness()
fdf = f.get_group_value_fairness(bdf)


def _color_red_or_green(val): # https://stackoverflow.com/questions/28075699/coloring-cells-in-pandas
    color = 'green' if val else 'red'
    return 'color: %s' % color


fdf[["attribute_name", "attribute_value", "Impact Parity", "Precision Parity", "FDR Parity", "FPR Parity", "TPR Parity", "Equalized Odds", "TypeI Parity"]].style.applymap(_color_red_or_green)

gaf = f.get_group_attribute_fairness(fdf.fillna(True))
gaf[["attribute_name", "Impact Parity", "Precision Parity", "FDR Parity", "FPR Parity", "TPR Parity", "Equalized Odds", "TypeI Parity"]].style.applymap(_color_red_or_green)

gof = f.get_overall_fairness(fdf)
gof

xtab[['attribute_value'] + absolute_metrics].round(2).to_latex('graphics/group_metrics_maxthreshold_ethnicity.tex')
fdf[["attribute_value", "ppr_disparity", "pprev_disparity", "precision_disparity", "fdr_disparity", "fpr_disparity", "tpr_disparity"]].round(2).to_latex('graphics/disparities_maxthreshold_ethnicity.tex')
fdf[["attribute_value", "Impact Parity", "Precision Parity", "FPR Parity", "TPR Parity", "Equalized Odds", "TypeI Parity", "Unsupervised Fairness"]].to_latex('graphics/fairness_maxthreshold_ethnicity.tex')

# ### Analysis
# We find that:
# * PPREV or **Impact Parity**
# * PPV- or Precision- or **Predictive Parity** only on a high level between Men and Women or on subgroup level between Caucasian men and Caucasian women
# * TPR- or Recall Parity or **Equal Opportunity**
# * **Equalized Odds** (TPR and FPR) 
# * **Predictive Equality** (FPR) 
# * **Equal Opportunity** (TPR) 
#
# There is no:
# * PPV- or Precision- or **Predictive Parity** (the colors are confusing here - if there is PPV Parity there has to be FDR Parity also) for groups other than Caucasian
# * Because of Equal Opportunity and limited Predictive Equality there is also limited **Overall Accuracy Equality**
# * Because of PPV/Predictive Parity and limited FPR/Predictive Equality there is also limited **Type I**
# * PPR- or Demographic- or **Statistical Parity**
# * Because of Impact Parity but no Statistical Parity, there is no **Unsupervised Fairness**.
# These last two depend on a representative data set, which we have not. However
#
# The unfairness does not negatively impact the concerned groups but if looking at the disparities, one can see that all groups except Caucasian women obtain **more precise** results than Caucasian men, the reference group. Therefore, one could say that the ethnicity API endpoint is actually biased against Caucasians. It seems to be harder to infer the ethnicity of Caucasians.
#
# Not measured:
# * FOR Parity
# * Type II Parity (FNR/Equal Opportunity and FOR Parity)
# * Supervised Fairness (Type I and II Parity)

# ## Impact of Treshold chosen
# Now we will check whether chosing different tresholds results in different fairness.

# +
from numpy import arange

disparities_res = pd.DataFrame({'model_id' : []})
fairness_res = pd.DataFrame({'model_id' : []})
overall_fairness_res = pd.DataFrame({'model_id' : []})
all_tables = []
complete_tables = []

g = Group()

tresholds = arange(0.1, 0.5, 0.1)
for t in tresholds:
    xtab, _ = g.get_crosstabs(df, 
                              attr_cols=["combined_attribute"], 
                              score_thresholds= {'score': [t]})
    absolute_metrics = g.list_absolute_metrics(xtab)
    
    b = Bias()
    bdf = b.get_disparity_predefined_groups(xtab, original_df=df, 
                                            ref_groups_dict={'combined_attribute':'Male Caucasian'}, 
                                            input_group_metrics=["ppr", "tpr", "fpr", "precision", "tnr", "npv", "pprev", "for", "fdr", "fnr"],
                                            check_significance=False, 
                                            mask_significance=False)
    
    f = Fairness()
    fdf = f.get_group_value_fairness(bdf)
    gaf = f.get_group_attribute_fairness(fdf)
    
    # save results
    fdf['model_id'] = t.round(2)
    fairness_by_group = fdf[['model_id', 'attribute_value'] + f.list_parities(fdf)]
    fairness_res = fairness_res.append(fairness_by_group, sort=False)
    
    gaf['model_id'] = t.round(2)
    overall_fairness_res = overall_fairness_res.append(gaf, sort=False)
    
    bdf['model_id'] = t.round(2)
    disparities_res = disparities_res.append(bdf, sort=False)
    complete_tables.append(bdf)
# -

disparity_overview = disparities_res[["model_id", "attribute_value", "ppr_disparity", "tpr_disparity", "fpr_disparity", "precision_disparity", "tnr_disparity", "npv_disparity", "pprev_disparity", "for_disparity", "fdr_disparity", "fnr_disparity"]].groupby(["model_id", "attribute_value"]).mean()

disparity_overview.style.applymap(_color_red_or_green_or_orange)

fairness_overview = fairness_res[["model_id", "attribute_value"] + f.list_parities(fdf)].groupby(["model_id", "attribute_value"]).max()

fairness_overview.style.applymap(_color_red_or_green)

overall_fairness_res

# For score = 0.1 the groups become too small to return any meaningful results.
#
# We find that:
# * PPV- or Precision- or **Predictive Parity** only on a high level between Men and Women or on subgroup level between Caucasian men and Caucasian women
#
#
# Tendencies:
# * lower thresholds result in lower **Impact Parity**, 
# * lower thresholds result in lower TPR- or Recall Parity or **Equal Opportunity**, **Equalized Odds** (TPR and FPR), **Predictive Equality** (FPR), **Equal Opportunity** (TPR), **Overall Accuracy Equality** except for Hispanics.
# * lower thresholds result in higher **FOR Parity**, **FNR Parity** and thus **Type II Parity**
#
# There is no:
# * PPV- or Precision- or **Predictive Parity** (the colors are confusing here - if there is PPV Parity there has to be FDR Parity also) for groups other than Caucasian
# * **Type I** Parity
# * Supervised Fairness
# * PPR- or Demographic- or **Statistical Parity**
# * Because of Impact Parity but no Statistical Parity, there is no **Unsupervised Fairness**.
# These last two are actually Fairness measures on the data set, which can not be given if the data set is not representational of all groups!
#
# Our observation from before can be confirmed: All groups except Caucasian women obtain **better** results than Caucasian men, the reference group. Therefore, one could say that the ethnicity API endpoint is actually biased against Caucasians. 

disparity_overview.to_html('graphics/disparities_by_threshold_ethnicity.html')
fairness_overview.to_html('graphics/fairness_by_threshold_ethnicity.html')
overall_fairness_res.to_html('graphics/overall_fairness_by_threshold_ethnicity.html')

disparities_res[["model_id", "attribute_value", "pprev_disparity", "npv_disparity", "precision_disparity", "fpr_disparity", "fnr_disparity", "for_disparity"]].round(2).groupby(["model_id", "attribute_value"]).mean()

disparities_res[["model_id", "attribute_value", "pprev_disparity", "npv_disparity", "precision_disparity", "fpr_disparity", "fnr_disparity", "for_disparity"]].round(2).groupby(["model_id", "attribute_value"]).mean().to_latex('graphics/disparities_by_threshold_ethnicity_small.tex')

disparity_overview.to_latex('graphics/disparities_by_threshold_ethnicity.tex')
fairness_overview.to_latex('graphics/fairness_by_threshold_ethnicity.tex')
overall_fairness_res.to_latex('graphics/overall_fairness_by_threshold_ethnicity.tex')

aqp = Plot()
i = 0.1
for table in complete_tables:
    fdf = f.get_group_value_fairness(table)
    plot = aqp.plot_fairness_group_all(fdf, ncols=5, metrics = "all")
    plot.savefig('graphics/ethnicity_endpoint_disparities_{}.pdf'.format(i), format='pdf')
    i = i + 0.1



StarcoderdataJupyterScriptsDedupFiltered