# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

#
# [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/kasparvonbeelen/ghi_python/blob/main/7%20-%20Trends%20over%20time.ipynb)
#
#
#
# ## 7: Studying trends over time in Political Manifestos (1964-2020)
#
#
# ## Text Mining for Historians (with Python)
# ## A Gentle Introduction to Working with Textual Data in Python
#
# ### Created by <NAME> and <NAME>
#
# ### For the German Historical Institute, London
#
# <img align="left" src="https://www.ghil.ac.uk/typo3conf/ext/wacon_ghil/Resources/Public/Images/institute_icon_small.png">
#
#
#
#

#
# This notebook combines the insights and techniques from previous sessions and explores a more realistic research example that focuses on studying trends over time in the UK Political Manifestos between 1964 and 2020. By creatively putting the stuff together you've seen so far, you can start to interrogate these corpora from a longitudinal perspective.
#
# This notebook will focus on plotting timelines showing (relative) frequencies for selected (or target) tokens.

# **[Important]** Please run the following cells, which download and extract the data needed in the remainder of the Notebook. If you are using Colab and you need to restart the Kernel/Runtime (or it restarted by itself), please run these cells again.

# !mkdir working_data
# !mkdir working_data/manifestos

# !wget -O working_data/manifestos.zip https://github.com/kasparvonbeelen/ghi_python/blob/main/data/manifestos/python.zip?raw=true
# !unzip -o working_data/manifestos.zip -d working_data/manifestos

# First, we start with collecting paths to our manifesto files. These are stored as separate `.txt` files one for each manifesto. As seen previously, we rely on an external library that we need to import first.
#
# We first define where our data is stored and create a `Path()` object to which we apply `Path.glob()`, with `"*.txt"` as the argument. The asterisk `*` serves as a wild card. Basically what are saying here is: in the folder `'working_data/Manifestos/Processed/'` return as files ending with `.txt` (we don't care what precedes the `.txt` extension).
#
# There is one last technicality we can ignore for now: we convert the result of `.glob()` to a `list`. 

from pathlib import Path
folder = Path('working_data/manifestos/Processed/')
files = folder.glob('*.txt')
file_paths = list(files)

# Now we can count the number of documents in our corpus by applying the `len()` function to the variable `file_paths`

len(file_paths)

# And, using index notation, we print the path to the first file in our corpus (not the content itself, this is for later!)

file_paths[0]

# So far we've used several tools such as `word_tokenize()` and `Counter()` to count tokens. In this Notebook, we want to repeat this procedure for all documents in our collection. We can write a **function** that takes a path as an argument and returns a mapping between word types and their frequencies. 
#
# The code cell below takes care of this we first import the tools we need and create a function to open a document and compute word frequencies. To create this function, we use the `def` keyword followed by a name, parentheses and a colon. The word between the parenthesis defines the arguments this function accepts.
#
# All the code inside this function should look familiar. For each given path, the function executes the sequence of statements (note the indentation) and returns the variable `word_counts`.
#
# It is important to run the cell below (but only once). Python will then store the function `count_words` into memory so you can reuse it later.

# +
from collections import Counter # import Counter for computing word frequencies
import nltk # import nltk
nltk.download('punkt') # we need this download to use the tokenizer
from nltk.tokenize import word_tokenize # impor the tool for tokenizing strings

def count_words(path): # define the function and the arguments it accepts
    text = open(path,'r').read() # open the document
    text_lowercase = text.lower() # lowercase the text and store in text_lowercase
    tokens = word_tokenize(text_lowercase) # get the tokens as a list
    word_counts = Counter(tokens) # compute word frequencies with Counter()
    return word_counts # return the word frequencies



# -

# To check if the function we created is ready, print`count_words`. You'll see that it is now stored as a `function` object.

print(count_words)

# We can apply this function to the first document by passing `file_paths[0]` as an argument. The second line prints the ten most frequent words.

word_counts = count_words(file_paths[0])
word_counts.most_common(10)

# We know how to count words, but the goal is to plot word counts over time. When you look more closely at the path and file name (i.e. the part after the last `/` you'll notice that it encodes a time-stamp.
#
# The filenames are structured as follows `{party identifier}_{year}{month}.{extension}`. Such a setup—where the filename encodes metadata is rather common when working with heritage collections. We can now start writing a function that extracts the time-stamp from the filename.
#
# First, we print get the file name from the full path (we use the `.stem` attribute here).

first_file = file_paths[0]
print(first_file)
file_name = first_file.stem
print(file_name)

# Now, we only need to extract the date information from the file name. We slice the first four characters from the string and convert it to an integer. 

file_name[-6:-2]

int(file_name[-6:-2])

# Slightly changing the indices we can capture both year and month.

year = int(file_name[-6:-2])
month = int(file_name[-2:])
print(year, month)

# In the very last step, we use a library `datetime` to convert these numbers (year and month are just integer values) to a proper timestamp. Since we don't have access to the exact day we just use the first day of the month.

import datetime
time_stamp = datetime.datetime(year,month,1)
time_stamp


# That were many steps! Since we like to repear it for each file, it is again useful to package them in one function.

def path2timestamp(path):
    file_name = path.stem
    year = int(file_name[-6:-2])
    month = int(file_name[-2:])
    time_stamp = datetime.datetime(year,month,1)
    return time_stamp


# We are almost ready to explore our corpus! We iterate over our corpus with a `for` and apply both our functions to each document. To keep track of our yearly counts we store the results in a dictionary to map time-stamps to word counts.
#
# The code below reminds you how iteration works. We loop over each element in `file_paths`, and `p` (the loop variable) takes each value (path) in turn. 

for p in file_paths[:10]:
    print(p.name)

# You are free to choose any name for `p` as long as you use it consistently!

for fp in file_paths[:10]:
    print(fp.name)

# The `for` loop below raises a `NameError` as we did not define `pp`.

for fp in file_paths[:10]:
    print(pp)

# We need one more element to make our word-counting program complete: we need to keep track of word counts for each year. For this, we can use a dictionary and increment the count while looping over the text files.
#
# To see how incremental counting works, first create a new variable `i` with the value zero.

i = 0

# Each time we run the cell below, we add `1` to `i` using the `+=` operator. Run the cell multiple time until you understand how this works!
#
# Note the difference between `=` and `+=`, former is used to declare variable, the latter to incrementally **add** a value to `i`.

i+=1
print(i)

# We can apply incremental counting also the values of dictionaries. This will be necessary as we traverse each file in our collection, look up the year it was written and how often our query appears, and add this value to the dictionary, meaning we increment the count for that specific year. 
#
# We demonstrate this with a practical example. We have a dictionary, with keys referring to years (1950 and 1951) and values to word counts (1 and 10), meaning that in 1950 our query appeared once and ten times one year later.

year2counts = {1950:1,1951:10}
print(year2counts)

# Imagine we open a new document in which our query appears twice. We can now add two for 1950.

year2counts[1950]+=1
print(year2counts)

# Now the value for 1950 is 3. We can, of course, do the same for 1951.

year2counts[1951]+=5
year2counts[1951]

# The incremental counting of word frequencies happens in line 8 in the code cell below. Instead of dictionary with use a `Counter()` but you can ignore the difference for now, `Counter()` objects are just a bit more flexible but behave similar to dictionaries.

# +
target_term = 'people' # define query as a string
results = Counter() # here we keep track of our word counts by mapping years to word frequencies

for fp in file_paths: # iterate over each path in our collection
    time_stamp = path2timestamp(fp) # returns a time stamp given a path
    word_counts = count_words(fp) # count the words in our document
    count_for_target = word_counts[target_term] # get frequency of target term
    results[time_stamp]+= count_for_target  # incrementally add the word count for our target term to results
# -

# This small program returns a mapping between time-stamps and word frequencies.

results

# We use an external library `pandas` to plot the timeline. We'll see more of `pandas` in part II.
#
# The first line in the cell below (starting with `%`) is necessary to plot the figure in the Notebook.

# %matplotlib inline 
import pandas as pd # import pandas
pd.Series(results).plot() # convert Counter to a pandas.Series and apply the `.plot()` method

# ## Extensions
#
# We create a few more examples that extend the functionalities of our word-counting program.
#
# ### Multiple words over time
#
# Instead of plotting the evolution of just one word, we track the evolution of a group of words over time. The main differences with the previous program are:
# - `target_terms` refers to a list and not a string (line 1)
# - the code cell now contains a double `for` loop: we iterate over all files (line 4) and within this block (i.e. for each file) we iterate over each item in `target_terms` (we collect the counts for each query term). Please note the use of indentation (the second for loop is followed by extra white space). Again, this is not just to make the code more readable, it is part of the Python syntax.

# +
target_terms = ['poverty','poor'] # define query terms as a list
results = Counter() # create Counter to keep track of word counts

for fp in file_paths: # iterate over paths
    ts = path2timestamp(fp) # get time stamp from path
    word_counts = count_words(fp) # get word counts for document
    for target_term in target_terms: # iterate over the query terms
        count_for_target = word_counts[target_term] # get word count for a query
        results[ts]+= count_for_target # increment count for the specific time stamp

pd.Series(results).plot() # plot results


# -

# ### Relative frequencies
#
# The upward trend in the previous could be misleading: is poverty really becoming a more important issue? Or are the manifestos just getting longer and thus containing more words? To control for the size of the corpus, we can compute the **relative** frequency of terms: we count how often a word appears and divide this by the total number of tokens).
#
# In the example below, we slightly change the function for counting words. Instead of only returning dictionary (or Counter) with word type frequencies, it also returns the total number of tokens. We name this extended function `count_words2` not to overwrite the function we created earlier.

def count_words2(path):
    text = open(path,'r').read() # open and read the document 
    text_lowercase = text.lower() # lowercase the text file
    tokens = word_tokenize(text_lowercase) # get tokens
    word_counts = Counter(tokens) # count tokens
    return word_counts,len(tokens) # return word frequencies and the total number of tokens


# We also add one more `Counter()` to keep track of the total number of tokens for each year

# +
target_terms = ['austerity','cuts'] # formulate queries
word_counts = Counter() # create dictionary for storing the frequency of our query for earch time stamp
total_counts = Counter() #  create dictionary for storing total word counts for earch time stamp


for fp in file_paths: # iterate over paths
    year = path2timestamp(fp) # get timestamp from path
    word_count,total_count = count_words2(fp) # get the word counts and the total number of tokens
    for target_term in target_terms: # 
        count_for_target = word_count[target_term]
        word_counts[year]+= count_for_target
        total_counts[year]+= total_count

# -

# Before we plot the results, we need to divide the frequency of our query by the total number of words. Please have conulst the Breakout on dictionaries for my information about the how to work with this data type.

results = {}  # empty dictionary where we save relative frequency
for year in word_counts.keys(): # iterate over all time stamps in the word counts dictionary
    results[year] = word_counts[year] / total_counts[year] # divide the frequency of query by the total number of word for 

pd.Series(results).plot()

# ## Fin.

StarcoderdataJupyterScriptsDedupFiltered