<jupyter_start><jupyter_text>Toxic Comment Classifier---For the full the report please refer to [Toxic Comment Classifier Report](insert link here). The goal of this classifier is to be able to identify if a comment is toxic, severely toxic, obscene, a threat, an insult, and/or identity hate.    Housekeeping In the below section section we'll be importing python modules we'll be using throughout the project. As well as some jupyter notebook specific configuration.<jupyter_code># Import libraries necessary for this project
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt    
from functools import reduce
import seaborn as sns
from io import StringIO
from wordcloud import WordCloud
import re
from sklearn.feature_extraction.text import TfidfVectorizer
from  sklearn.model_selection import train_test_split
import pickle 
import subprocess

sns.set(style="white")

# Pretty display for notebooks
%matplotlib inline

#load the Toxic Comment data 
data = pd.read_csv('./data/train.csv')<jupyter_output><empty_output><jupyter_text>Exploraty Data AnalysisOur first step is to understand the data set we're working with. In this section we want to perfrom the following analysis of dataset: - Dimensions of the data set - Data Inspection- Completeness of data set - Identify the distribution of labels- Frequencies of words used for each label   Dimensions of the data set<jupyter_code>print("The data set has {} rows with {} columns".format(*data.shape))<jupyter_output><empty_output><jupyter_text>Data Inspection<jupyter_code>data.dtypes<jupyter_output><empty_output><jupyter_text>As we can see our dataframe is made of 8 columns. The latter 6 are our target labels. They contain only binary values 0 for false and 1 for true. The `comment_text` column contains the comment as a string.<jupyter_code>data.head(n=10)<jupyter_output><empty_output><jupyter_text>As we can see in the above table comments that contain mixed case characters. As well as escaped characters such as `\n`.   Completeness of data set<jupyter_code>print("Does the data contain missing data? : ")
print(pd.isnull(data).any())
print("Longest comments length: {}".format(max(data["comment_text"].apply(len))))
print("Shortest comments length: {}".format(min(data["comment_text"].apply(len))))
print("Mean comments length: {}".format(np.average(data["comment_text"].apply(len))))
print("")
data[(data["comment_text"].apply(len)) == 6]<jupyter_output>Does the data contain missing data? : 
id               False
comment_text     False
toxic            False
severe_toxic     False
obscene          False
threat           False
insult           False
identity_hate    False
dtype: bool
Longest comments length: 5000
Shortest comments length: 6
Mean comments length: 394.0732213246768<jupyter_text>Identify the distribution of labels<jupyter_code>rowsums = data.iloc[:,2:].sum(axis=1)
temp_data = data.copy()
temp_data['clean'] = (rowsums == 0)
labels = temp_data.columns[2:]

def calculate_label_proportions(label):
    proportion = temp_data[temp_data[label] == 1].shape[0] / temp_data.shape[0] 
    return "Percentage of {} comments: {:.2%}".format(label, proportion)

label_distribution = reduce(lambda res, cur: "\n".join([res, calculate_label_proportions(cur)]), labels, "")

print(label_distribution)<jupyter_output>Percentage of toxic comments: 9.58%
Percentage of severe_toxic comments: 1.00%
Percentage of obscene comments: 5.29%
Percentage of threat comments: 0.30%
Percentage of insult comments: 4.94%
Percentage of identity_hate comments: 0.88%
Percentage of clean comments: 89.83%<jupyter_text>We can see in the above code block that our data set is very unbalanced. With none of the labels coming near 10% of the data set let alone anywhere near 50%. When we go to train our network we want to be sure that we take steps to ensure our models give a true output of it's "accuracy". Therefore, we can initailly rule out the use of arithmatic mean (A.K.A accuracy) as our evaluation method. Furthermore, given that we're trying to perform to build a classifier that can output multiple classification the ROC AUC evaluation metric would be a good choice.<jupyter_code>x = temp_data[labels].sum()
plt.figure(figsize=(12,4))
ax= sns.barplot(x.index, x.values, alpha=0.8)
plt.title("Number of Occurances Per Class")
plt.ylabel('Number of Occurrences', fontsize=12)
plt.xlabel('Comment Type ', fontsize=12)

x<jupyter_output><empty_output><jupyter_text>We see that the `toxic` comment is the most common of the toxicity comment types. From looking at the head of the comments we see that a comment can be classed as one or more of the toxicity comment types. Therefore, we should look to identify any correlations one comment type would have with others.<jupyter_code># TODO: Confusion matrix<jupyter_output><empty_output><jupyter_text>It is also worth exlporing the frequency of certain words that appear in each of toxicity columns as well as the clean column. To visualise the word frequency we'll use word cloud diagrams for each category.<jupyter_code>def concat_strings(label):
    string_io = StringIO() 
    comments = temp_data[temp_data[label] == 1].comment_text
    comments.apply(lambda x: string_io.write(str(x)))
    text = string_io.getvalue()
    string_io.close()
    return text

fig = plt.figure(figsize=(20,30))

for i, label in enumerate(labels):
    text = concat_strings(label)
    wordcloud = WordCloud().generate(text)
    wordcloud = WordCloud(max_words=len(text), relative_scaling=.5).generate(text)
    fig.add_subplot(7, 1, i + 1)
    plt.imshow(wordcloud)
    plt.xlabel("{} comments".format(label))


plt.show()
    
# str.isupper(temp_data["comment_text"].iloc[6])

# TODO violin graph comment length
sns.set(color_codes=True)

# Load the example tips dataset
temp_data['comment_word_count'] = temp_data['comment_text'].apply(lambda x: len(str.split(x)))
temp_data['comment_sentance_count'] = temp_data['comment_text'].apply(lambda x: len(re.split('[\.|!|\n]', x)))
fig = plt.figure(figsize=(30,60))


fig.add_subplot(6, 2, 1)
sns.violinplot(x='toxic', y='comment_word_count', data=temp_data, split=True, inner="quart", pallette={0:'b', 1:'r'})
fig.add_subplot(6, 2, 2)
sns.violinplot(x='toxic', y='comment_sentance_count', data=temp_data, split=True, inner="quart", pallette={0:'b', 1:'r'})

fig.add_subplot(6, 2, 3)
sns.violinplot(x='severe_toxic', y='comment_word_count', data=temp_data, split=True, inner="quart", pallette={0:'b', 1:'r'})
fig.add_subplot(6, 2, 4)
sns.violinplot(x='severe_toxic', y='comment_sentance_count', data=temp_data, split=True, inner="quart", pallette={0:'b', 1:'r'})

fig.add_subplot(6, 2, 5)
sns.violinplot(x='obscene', y='comment_word_count', data=temp_data, split=True, inner="quart", pallette={0:'b', 1:'r'})
fig.add_subplot(6, 2, 6)
sns.violinplot(x='obscene', y='comment_sentance_count', data=temp_data, split=True, inner="quart", pallette={0:'b', 1:'r'})

fig.add_subplot(6, 2, 7)
sns.violinplot(x='insult', y='comment_word_count', data=temp_data, split=True, inner="quart", pallette={0:'b', 1:'r'})
fig.add_subplot(6, 2, 8)
sns.violinplot(x='insult', y='comment_sentance_count', data=temp_data, split=True, inner="quart", pallette={0:'b', 1:'r'})

fig.add_subplot(6, 2, 9)
sns.violinplot(x='identity_hate', y='comment_word_count', data=temp_data, split=True, inner="quart", pallette={0:'b', 1:'r'})
fig.add_subplot(6, 2, 10)
sns.violinplot(x='identity_hate', y='comment_sentance_count', data=temp_data, split=True, inner="quart", pallette={0:'b', 1:'r'})

fig.add_subplot(6, 2, 11)
sns.violinplot(x='clean', y='comment_word_count', data=temp_data, split=True, inner="quart", pallette={0:'b', 1:'r'})
fig.add_subplot(6, 2, 12)
sns.violinplot(x='clean', y='comment_sentance_count', data=temp_data, split=True, inner="quart", pallette={0:'b', 1:'r'})


# TODO violin graph comment frequency of uppercase words
temp_data.describe()
# temp_data[]
temp_data[temp_data.columns[-2:]].describe()<jupyter_output><empty_output><jupyter_text>In the above the table we see that the longest comment has approximately  1411 words.<jupyter_code>fig = plt.figure(figsize=(30,30))

def x_score(z, m, o):
#     z = (x - m) / o
    return (z * o) + m

word_count_mean = temp_data['comment_word_count'].mean()
word_count_std = temp_data['comment_word_count'].std()
word_count_outlier_score = x_score(2.0, word_count_mean, word_count_std) 
print(word_count_outlier_score)
temp_data_1 = temp_data[(temp_data['comment_word_count'] < word_count_outlier_score)]
print("Data count with outliers {}".format(len(temp_data)))
print("Data without word_count outliers".format(len(temp_data_1)))

sentance_count_mean = temp_data['comment_sentance_count'].mean()
sentance_count_std = temp_data['comment_sentance_count'].std()
sentance_count_outlier_score = x_score(2.0, sentance_count_mean, sentance_count_std) 
print(sentance_count_outlier_score)

temp_data_2 = temp_data[(temp_data['comment_sentance_count'] < sentance_count_outlier_score)]

print("Data without sentance_count outliers".format(len(temp_data_2)))

fig.add_subplot(2, 2, 1)
sns.violinplot(x='clean', y='comment_word_count', data=temp_data_1, split=True, inner="quart", pallette={0:'b', 1:'r'})
fig.add_subplot(2, 2, 2)
sns.violinplot(x='clean', y='comment_sentance_count', data=temp_data_2, split=True, inner="quart", pallette={0:'b', 1:'r'})<jupyter_output>265.73493153046707
Data count with outliers 159571
Data without word_count outliers
65.69847377402671
Data without sentance_count outliers<jupyter_text>Data Preprcessing Now that we've performed our data analysis we'll begin our work on preporcessing the data. We want to apply several methods for preprocessing our data. The first method is to use the bag of words technique and (word frequency). Then we want to use Word2Vec and GloVe. Below we will define several functions that will take in the raw comments and preprocess them.  Pre Processing StepsThe first preprocessing step we'll take is to clean the text before processing it into vectors.<jupyter_code>data.loc[50, 'comment_text'][:500]<jupyter_output><empty_output><jupyter_text>As we can see above, some of the comments contain escapped characters, punctuation, and other non-alphanumeric characters. Our inital preprocessing step is to remove all non alphabetical characters. By removing the escaped characters we'll be able to make the text cleaner. However, by removing some of the punctuation we could loose some semantic information. Particularly if we have a user quoting another user's comment or wikipedia that has unsavory text in it.<jupyter_code>def text_preprocessor(text : str)-> str:
    regex = r"(\n|\t|\d|[^\s\w])+"
    partially_processed_text = re.sub(regex, ' ', text.lower(), 0, re.MULTILINE)
    return re.sub(r" +",' ', partially_processed_text).strip().split()

text_preprocessor(data['comment_text'][0])<jupyter_output><empty_output><jupyter_text>As we can now see in the table above we've removed punctuations and escaped characters. We also converted the text to lower case.<jupyter_code>X = data['comment_text']
y = data[data.columns[2:]]
processed_data_set = train_test_split(X, y, test_size=0.33, random_state=42)
X_train_raw, X_test_raw, y_train, y_test = processed_data_set
# import os 
# processed_data_set_filename = "./pickles/data-set.p"
# os.makedirs(os.path.dirname(processed_data_set_filename), exist_ok=True)
# pickle.dump(processed_data_set, open(processed_data_set_filename, 'wb'))<jupyter_output><empty_output><jupyter_text>Bags of WordsNow that we've cleaned up the text up our text we're going to begin covnerting our text into numerical data. Our initial method for doing this will be using the **bag of words** model. This will involve us to create a vocabulary of unique tokens. Then we'll create a feature vector from each comment contianing the counts of how often each word occurs within it.<jupyter_code>vec = TfidfVectorizer(ngram_range=(1,2), tokenizer=text_preprocessor,
               min_df=3, max_df=0.9, strip_accents='unicode', use_idf=1,
               smooth_idf=1, sublinear_tf=1 )
X_train = vec.fit_transform(X_train_raw)
X_test = vec.transform(X_test_raw)
X_train.shape[0] / 2<jupyter_output><empty_output><jupyter_text>Naive BenchmarkingOur initial experiment is to first assert that we can classify comments with an accuracy of greater than 50%, which indicates random they we're able to make better predictions than merely guessing the toxicity of a comment. We therefore decidied to first use Multinominal Naive Bayes.<jupyter_code>from sklearn.naive_bayes import MultinomialNB
def create_classifiers(classifier, X, y, comment_type):
    clfs = classifier 
    clfs.fit(X, y[comment_type])
    return clfs
clfs = MultinomialNB(alpha=.01)
classifiers = [create_classifiers(clfs, X_train, y_train, comment_type) for comment_type in y_train.columns]
def predict_proba(clfs, x):
    return [clf.predict_proba(x)[0][1] for clf in clfs]

def predict(clfs, x):
    return [clf.predict(x)[0] for clf in clfs]
print(repr(data[(data['threat'] == 1)].iloc[0].comment_text))
print()
print(repr(data[(data['threat'] == 1)].iloc[2].comment_text))
threat_sub = data[(data['threat'] == 1)].iloc[0].comment_text #"I’m going to teach you to speak English with this fucking knife!"
threat_not_sub = data[(data['threat'] == 1)].iloc[2].comment_text # "Just die!!"
vec_threat_sub, vec_threat_not_sub = vec.transform([threat_sub, threat_not_sub])
print(predict_proba(classifiers, vec_threat_sub))
print(predict_proba(classifiers, vec_threat_not_sub))
from sklearn.metrics import roc_auc_score
y_pred = [predict_proba(classifiers, x) for x in X_test]
roc_auc_score(y_test, y_pred)<jupyter_output><empty_output><jupyter_text>As we can see above achieved an roc_auc_score of `~0.94`.  SVM NB bench marking From reading (this paper)[https://nlp.stanford.edu/pubs/sidaw12_simple_sentiment.pdf]  we may be able to build a more robust classifier befre develing into the use of CNN and RNN for sentimental analysis. Unfortuantely this model is not available out of the box from sklearn. Therefore we'll have to create our own implementation. It is also worth mentioning that from reading this paper using Logistic Regression instead of SVM will yield similar results. Therefore, instead of using SVM, mainly due to it's perfromance issues, we'll instead user Sklearn's Logistic Regression model.<jupyter_code>from sklearn.base import BaseEstimator, ClassifierMixin
from sklearn.utils.validation import check_X_y, check_is_fitted
from sklearn.linear_model import LogisticRegression
from scipy import sparse
import numpy as np


class NbSvmClassifier(BaseEstimator, ClassifierMixin):
    def __init__(self, C=1.0, dual=False, n_jobs=1):
        self.C = C
        self.dual = dual
        self.n_jobs = n_jobs
    
    def predict(self, x):
        # Verify that model has been fit
        check_is_fitted(self, ['_r', '_clf'])
        return self._clf.predict(x.multiply(self._r))

    def predict_proba(self, x):
        # Verify that model has been fit
        check_is_fitted(self, ['_r', '_clf'])
        return self._clf.predict_proba(x.multiply(self._r))
    
    def fit(self, x, y):
        y = y.values
        x, y = check_X_y(x, y, accept_sparse=True)
        self._r = sparse.csr_matrix(np.log(self._pr(x, 1, y) / self._pr(x, 0, y)))
        x_nb = x.multiply(self._r)
        self._clf = LogisticRegression(C=self.C, dual=self.dual, n_jobs=self.n_jobs).fit(x_nb, y)
        return self
    
    
    def _pr(self, x, y_i, y):
        p = x[y == y_i].sum(0)
        return (p + 1) / ((y == y_i).sum() + 1)

classifiers_svm = [create_classifiers(NbSvmClassifier(1.0), X_train, y_train, comment_type) for comment_type in y_train.columns]
print(predict_proba(classifiers_svm, vec_threat_sub))
print(predict_proba(classifiers_svm, vec_threat_not_sub))
y_pred = [predict_proba(classifiers, x) for x in X_test]
roc_auc_score(y_test, y_pred)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup