<jupyter_start><jupyter_text>Testing Sentiment Analysis-----In this notebook, we test sentiment analysis techniques for analysing natural language for which there are three experiments.As representations of conflict narratives, we curated a dataset comprising Hitler’s “<NAME>” along with political speeches from <NAME> and Osama bin Laden during the “War on Terror”. In how he advocated for non-violence, <NAME> provides control data. We do not suggest any moral equivalence between each of these orators. Instead, we are exploring functional equivalence in the use of language to legitimise violence. Each has successfully brought about dramatic change through force, or in the case of Luther King, no-violent means.Against this dataset, We tested three sentiment analysis API for this experiment. The first is TextBlob, a general-purpose and open-source NLP python library. As state-of-the-art technologies, and therefore more technically sophisticated, the next two are APIs from Google and Watson.Having such extremes in the dataset means we can assess experimental results through observation since the moral colour each text is accepted, and we know who each were, or were not, seeking to legitimise violence against.We will now see how quantitative representations of language can distort meaning. Where stark results would be expected from a dataset of extremes, our tests show that regardless of technical sophistication, these technologies are unable to distinguish abusive from non-abusive texts.In the first experiment, we compare different document scores for each text.In the second experiment, we compare scores for named entities identified in each document.More experiments results to follow.  Load the Sentiment Analysis Dataset and Show Metadata<jupyter_code>import os
from cndobjects import SentimentData
apis = ["textblob", "watson", "google"]
filename = "sentiment_analytics"
filepath = os.getcwd()
            
sentiment_analytics = SentimentData(apis = apis,
                                   filename = filename,
                                   filepath = filepath)

sentiment_analytics.fromDisk()
sentiment_analytics.df<jupyter_output>loading: sentiment_analytics from:
C:\Users\Steve\OneDrive - University of Southampton\CNDPipeline\Quantitative Analysis<jupyter_text>Test 1 - Document Sentiment ScoresIn this first test we look at how each API scores each text of the dataset overall.In this test we colate and display document sentiment scores for TextBlob, Watson and Google.Note - <NAME> is too large for the Google API limits, therefore, its score is derived from the mean sentence level sentiment score<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
import importlib
import cndplots
importlib.reload(cndplots)

fig, ax = plt.subplots(figsize = (30, 15))

scores = np.array([sentiment_analytics.df[api] for api in sentiment_analytics.apis])

im, cbar = cndplots.heatmap(scores,
                    sentiment_analytics.apis,
                    [f'{title} ({ref})' for title, ref in zip(sentiment_analytics.df["title"], sentiment_analytics.df["ref"])], 
                    ax=ax,
                    figtitle = "A Heatmap of Document-level Sentiment Scores",
                    cmap="Blues", cbarlabel="Sentiment Score")

texts = cndplots.annotate_heatmap(im, valfmt="{x:.2f}")

fig.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Test 1 - DiscussionFirstly, as a genocidal text, Textblob scores “<NAME>” at joint 15th for positivity and 5th highest for Watson. For negativity ranking, Google scores “<NAME>” as the sixth most negative. In comparison with the most positive of these texts, TextBlob scores “I Have a Dream” at 0.04 more positive, Watson scores “<NAME>” as 0.01 more positive and Google scores “<NAME>” at 0.13 more negative. A reasonable expectation would be for “<NAME>” to generate scores dramatically more negatively than “I Have a Dream”, which is not the case in these results.The highest score for TextBlob is Bush’s “The World Will Always Remember 9/11”. The highest score for Watson is bin Laden’s “Letter Calling for Global Islamic State”. The highest score for Google is also bin Laden’s “Letter Calling for Global Islamic State”. In how Luther King advocated for non-violent change, contrary to these results, his texts would be expected to score more highly.The lowest negativity score for TextBlob is -0.08 for Bush’s “Address at Islamic Center of Washington, D.C.”. The lowest negativity score for Watson is bin Laden’s “Al Jazeera Speech”. The lowest negativity score for Google is bin Laden’s “letter to America”. As a genocidal text, “Mein Kampf” would reasonably be expected to generate the most negative scores.Across the range of texts, and excluding scores of 0, the only text where all three APIs consistently score for positivity or negativity is bin Laden’s “Letter Calling for Global Islamic State”. For all other texts, there is no agreement between all three APIs for positivity of negativity.Google, scores nine of Bush’s 14 texts at 0, which suggests neutral sentiment for speeches in which he sought to legitimise the “War on Terror”.With each API generating contrasting scores, there is simply no way to assess the efficacy of these results. The APIs certainly do not produce results that might be expected. We can only conclude that quantitative expressions of sentiment generate meaningless outputs.  Test 2 - Testing Sentiment Scores at the Sentence LevelIn this next experiment we show the sentiment scores for each sentence of selected documents.<jupyter_code>import importlib
import cndplots
importlib.reload(cndplots)

# setup document references
# orator ref // document reference // smoothing window
orators_list = [("bush", 4, 10), ("laden", 0, 10), ("hitler", 0, 70), ("king", 1, 10)]

fig = cndplots.sentiment_plot(orators_list, sentiment_analytics, smooth = True, \
                    figtitle = 'Sentiment Analysis Comparison for Overall Document Scores and Sentence-Level Profiles',
                    xlabel = 'progress through the document (%)',
                    ylabel = 'rolling average sentiment')
fig.savefig(fname = "sentiment_plots")<jupyter_output><empty_output><jupyter_text>Test 2 - DiscussionThese four graphs show a sentence level sentiment profiles against each API for significant texts from the dataset.Firstly consider the profile of both Bush’s and bin Laden’s texts in which they declare war upon each other. Both have similar sentiment profiles relative to positivity and negativity; however, the sentiment relative to how they were received was very different. Their supporters would receive each positively while their opponents would view them negatively. Moreover, transcripts of Bush’s speech show how people applauded statements these APIs scored negativity.Secondly, in a dataset of extremes, we can see the profile for <NAME> is indistinguishable from the rest. We might expect such an extremely harmful body of words to produce more stark results. Notably, if you look at the overall document scores, where <NAME> and I Have a Dream are extreme opposites in terms of sentiment, Watson and TextBlob determine both as positive, and Google determines them to be negative. In a face-to-face discussion, we might now discuss what these results mean, and inevitably there will be disagreement, yet what is the assessment framework for analysing sentiment that moderates our differences in agreement? Missing from teh y axis of each graph is a unit of measurement for sentiment, so how can we assess what does not have a defined metric? What we might agree upon for now is that for a dataset comprising extremes of sentiment, and APIs of extremes in technical sophistication, these results distort th intended meaning of each text. And perhaps in how these APIs are unable to distinguish Mein Kampf from the others, these results might also be regarded as somewhat troublesome.  Review of Sentence ScoresFollowing the findings from the sentence-level sentiment profiles, this next section is a more detailed review of sentence-level scores.First is a reivew of agreement between each API for sentence scores for which there are three assessments. The first assessment reviews agreement between each of the APIs. If all three APIs score either positively or negatively they are in agreement, if one differs from the others they are in disagrement. The second assessment reviews agreement between general purpose and state-of-the-art. If Watson and Google agree, then there is agreement, otherwise there is disagrement. The third assessment reviews Watson's emotion scores. If an positive emotion correlates with a positive sentiment score there is agreement, otherwise there is disagreements.Second is a review of the most positive and negative scores from each document. In this review scores of +1 or -1 for any API are extracted from each document, then the highest and lowest score not +1 or -1 are extracted. Through observation, these sentences are then reviewed.Third is a comparison of selected sentences from "Mein Kampf" and "I Have a Dream". Sentences from Mein Kampf are selected for obvious assertions of racism. The iconic, 'I Have a Dream" statements are selected from "I Have a Dream"  Agreement between APIs<jupyter_code>import pandas as pd

agreement = {1 : {"assessment" : "assess the agreement between APIs", "score" : 0}, 
             2: {"assessment" : "assess the agreement between state-of-the-art APIs", "score" : 0},
             3 : {"assessment" : "assess agreement between Watson emotion and sentiment", "score" : 0}
            }

for orator in sentiment_analytics.values():
    
    for text in orator:
        
        for sentence in text["sentences"]:
            scores = sentence["scores"]
            
            # assessment 1: agreement between each
            # if all the scores are positive or all the scores are negative or all the scores are neutral there is agreement
            if all([i > 0 for i in scores.values()]) or all([i < 0 for i in scores.values()]) or all([i == 0 for i in scores.values()]):
                agreement[1]["score"] +=1
                
            # assessment 2: agreement between each state of the art
            sota = ["watson", "google"]
            # if all the scores are positive or all the scores are negative or all the scores are neutral there is agreement
            if all([i > 0 for k, i in scores.items() if k in sota]) or all([i < 0 for k, i in scores.items() if k in sota]) or all([i == 0 for k, i in scores.items() if k in sota]):
                agreement[2]["score"] +=1
            
            # sadness, joy, fear, disgust, anger
            api = "watson"
            pos_emotion = ["joy"]
            neg_emotion = ["sadness", "fear", "disgust", "anger"]
            watson_score = sentence["scores"]["watson"]
            emotions = sentence["emotion"]
            
            # if sentiment score is positive and the highest scoring emotion is a positive emotion, or if sentiment score is negative and highest scoring emotion is a negative emotion
            if all([watson_score > 0, max(emotions, key=emotions.get) in pos_emotion]) or all([watson_score < 0, max(emotions, key=emotions.get) in neg_emotion]):
                agreement[3]["score"] +=1
            
df = pd.DataFrame(agreement)       
for i, v in enumerate(df.values[1]):
    df.values[1][i] = f'{round(v*100/len(sentiment_analytics))}%'
df.T<jupyter_output><empty_output><jupyter_text>These results show that agreement between all three APIs is 42%, while agreement between state-of-the-art APIs is 67%. There is a reasonable amount of disagreement between these APIs, yet what is the assessment framework for sentiment analysis to determine which API is correct?Within the Watson API, agreement between sentiment and emotion assertion is 52%, this means for 48% of sentences the sentiment and emotion scores are contradictory.   Review of highest scoring sentences from each document<jupyter_code>import pandas as pd

orator = "hitler"
text = 0
title = f'{orator} ({sentiment_analytics[orator][text]["datestamp"]}) {sentiment_analytics[orator][text]["title"]}'
print(title)
print('-'*len(title))

pd.set_option('max_colwidth', 1000)
for sentdata in sentiment_analytics.minmax[orator][text].values():
    
    if sentdata["sentences"]:
        print(sentdata["explain"])
        
        if isinstance(sentdata["sentences"], list):
            display(pd.DataFrame(sentdata["sentences"]))
        if isinstance(sentdata["sentences"], dict):
            display(pd.DataFrame(sentdata["sentences"]).T)<jupyter_output>hitler (2020-06-30) Mein Kampf
------------------------------
List of sentences an API has scored at +1<jupyter_text>Comparison of selected sentences from Mein Kampf and I Have a DreamSeveral sentences from Meing Kampf were selected for comparison with Luther King's "I Have a Dream" statements. Sentence from <NAME> were selected for their obvious expression and promotion of hate and any contradiction with either sentiment or emotion scores.<jupyter_code>import pandas as pd
from collections import Counter
import seaborn as sns

racism_texts = [(238, "A change would be possible only by a mixture of blood, but in this case the quality of the superior race would be debased."),
                (240, "It is especially the cultural creativeness which disappears when a superior race intermixes with an inferior one, even though the resultant mongrel race should excel a thousandfold in speaking the language of the race that once had been superior."),
                (326, "Thus, conversely, a State may be called bad if, in spite of the existence of a high cultural level, it dooms to destruction the bearers of that culture by breaking up their racial uniformity."),
                (389, "Nature generally takes certain measures to correct the effect which racial mixture produces in life."),
                (394, "At all critical moments in which a person of pure racial blood makes correct decisions, that is to say, decisions that are coherent and uniform, the person of mixed blood will become confused and take measures that are incoherent."),
                (837, "No boy or girl must leave school without having attained a clear insight into the meaning of racial purity and the importance of maintaining the racial blood unadulterated."),
                (1133, "The constructive principle of Aryan humanity is thus displaced by the destructive principle of the Jews, They become the 'ferment of decomposition' among nations and races and, in a broad sense, the wreckers of human civilization."),
                (1855, "And the swastika signified the mission allotted to us – the struggle for the victory of Aryan mankind and at the same time the triumph of the ideal of creative work which is in itself and always will be anti-Semitic."),
                (4525, "A state which in this age of racial poisoning dedicates itself to the care of its best racial ele­ments must some day become lord of the earth."),
               ]

dream_texts = [(61, "I have a dream that one day this nation will rise up and live out the true meaning of its creed:"),
               (62, "We hold these truths to be self-evident, that all men are created equal."),
               (63, "I have a dream that one day on the red hills of Georgia, the sons of former slaves and the sons of former slave owners will be able to sit down together at the table of brotherhood."),
               (64, "I have a dream that one day even the state of Mississippi, a state sweltering with the heat of injustice, sweltering with the heat of oppression, will be transformed into an oasis of freedom and justice."),
               (65, "I have a dream that my four little children will one day live in a nation where they will not be judged by the color of their skin but by the content of their character."),
               (66, "I have a dream today!"),
               (67, "I have a dream that one day, down in Alabama, with its vicious racists, with its governor having his lips dripping with the words of \"interposition\" and \"nullification\" -- one day right there in Alabama little black boys and black girls will be able to join hands with little white boys and white girls as sisters and brothers."),
               (68, "have a dream today!"),
               (69, "I have a dream that one day every valley shall be exalted, and every hill and mountain shall be made low, the rough places will be made plain, and the crooked places will be made straight; \"and the glory of the Lord shall be revealed and all flesh shall see it together.\"")
              ]

# for orator in sentiment_analytics.values():
#     for text in orator:
#         for i, sentence in enumerate(text["sentences"]):
#             if "race" in sentence["text"].split(" "):
#                 print(i, '=>', sentence["text"])

# function to colour sentiments based on positivity or negativity
def color_red_or_green(val):
    if isinstance(val, (int, float)):
        color = 'red' if val < 0 else 'green'
        return 'color: %s' % color
    return val
    
# function to get the formatted sentiment and emotion scores
def get_sents(orator, text, sents):
    for n in sents:
        sent = sentiment_analytics[orator][text]["sentences"][n[0]]
        sent_scores = dict()
        sent_scores["text"] = sent["text"]
        sent_scores.update(sent["scores"])
        sent_scores.update(sent["emotion"])
        yield sent_scores

def sentiment_rank(dataframe):

    """
    review sentiment scores to determine prevalence of positivity/negativity
    """
    for col in dataframe[sentiment_analytics.apis]:
        pos = [i > 0 for i in dataframe[col] if i != ''].count(True)
        neg = [i < 0 for i in dataframe[col] if i != ''].count(True)
        neu = [i == 0 for i in dataframe[col] if i != ''].count(True)

        dataframe[col][dataframe.index[-1]] = (f'positive = {pos} negative = {neg} neutral = {neu}')
        
    return dataframe

def emotion_rank(dataframe):
    
    """
    review emotion scores to determine ranking of expressed emotions in the sentences.
    """
    
    ranking  = {"first" : "", "second" : "", "third" : ""}
    emotions = ["sadness", "joy", "fear", "disgust", "anger"]
    
    for rank in ranking:

        cache = list()

        # iterate through the column slice containing emotions
        for index, row in dataframe[emotions].iterrows():

            # exclude just added new_row row
            if not any(row.values):
                continue

            # remove the previous ranked emotions from consideration
            temp = row
            for v in list(ranking.values()):
                if v in temp: temp = temp.drop(v)

            # sort the emotion scores in descending order to get 
            # top emotion minus previously ranked emotion
            sorted_row = temp.sort_values(ascending = False)
            cache.append(temp.loc[lambda x: x == sorted_row[0]].keys()[0])

        # add top ranking score to ranking dictionary
        # add ranking to dataframe
        emotion = Counter(cache).most_common()[0][0]
        ranking[rank] = emotion
        df[emotion][len(dataframe) -1] = rank
        
    return dataframe

hitler = ("hitler", 0, racism_texts)
king = ("king", 1, dream_texts)

# create the dataframe of selected sentences with sentiment and emotion scores
pd.set_option('max_colwidth', 1000)        
df = pd.DataFrame(get_sents(*hitler))

# add an empty row for assessment data
df = df.append(pd.Series(dtype=str), ignore_index=True).fillna('')

# add assessment data
df = sentiment_rank(df)
df = emotion_rank(df)

# display dataframe
u = df.index.get_level_values(0) # used to capture subset slice of df for gradient for rows with mixture on string and index
cm = sns.light_palette("green", as_cmap=True) # set colour scheme
display(df.style.applymap(color_red_or_green, subset = sentiment_analytics.apis).\
                  background_gradient(cmap=cm, subset = pd.IndexSlice[u[:-1], emotions]
                                     ))

pd.set_option('max_colwidth', 1000)        
df = pd.DataFrame(get_sents(*king))

# add an empty row for assessment data
df = df.append(pd.Series(dtype=str), ignore_index=True).fillna('')

# add assessment data
df = sentiment_rank(df)
df = emotion_rank(df)

# display dataframe
u = df.index.get_level_values(0) # used to capture subset slice of df for gradient for rows with mixture on string and index
cm = sns.light_palette("green", as_cmap=True) # set colour scheme
display(df.style.applymap(color_red_or_green, subset = sentiment_analytics.apis).\
                  background_gradient(cmap=cm, subset = pd.IndexSlice[u[:-1], emotions]
                                     ))<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup