Do you think it's possible to determine what a text document is about? Read on to see how we can use R to programmatically identify the main topics of several South Park seasons and episodes!
In the second article of the series, I showed you how to use R to analyze the differences between South Park characters. I assumed that Eric Cartman is the naughtiest character in the show and tested this hypothesis with real data–check it out to see my conclusion if you haven't already.
In this last article of the series, you'll learn how to identify episode and season topics programmatically. I'll show you a very powerful tool that can help us describe what's going on across different documents. We'll be using data from my southparkr package and the wonderful tidytext package. I'll include most of the code used to create all the tables and plots so you can follow what's going on more easily.
If you haven't watched the show, beware–spoilers ahead!
Now let me introduce the method we'll be using: TF-IDF, which stands for term frequency-inverse document frequency. Wikipedia offers a nice explanation–it's a numerical statistic that's used to determine the importance of a given word to the context of a document that's part of a larger collection of documents.
Very common English words like the, a, and, etc. (known as stop words) are found in nearly every text. If we were to tally the words in a random document with just raw counts, these words would certainly be at the top.
See? We were right! There are no words with real meaning here that would tell us what the show is actually about. Rather, these words appear frequently simply because they're popular in the English language.
Here's another example to help make this more concrete: Imagine an author who's written several books, perhaps spanning multiple genres. Each book will contain certain words that are unique to its plot and context, but these words likely won't appear in the others. TF-IDF analysis will help us identify these unique words so we can get a better sense for what a document is about.
You'll see this later on in action with South Park–each episode contains unique lines of dialog with different words; TF-IDF will reveal the most important words for each episode. Because I know South Park very well, I'll personally evaluate the results of this analysis to see if it succeeded.
But How Does TF-IDF Work?
Simply put, term frequency is how often a given word is used in a document (as a percentage). We'll use the raw count of a word divided by the total number of words in a document to determine this metric. Easy enough, right?
Note that the TF-IDF will always be 0 for words that appear in every document, since ln(1) is always 0.
Let's do a concrete example with the word alien. South Park episodes will be our documents; we'll calculate the term frequency for each episode in which the word 'alien' appears. Its inverse document frequency will be a single number. The following table captures all of these results for each episode where the word alien appears.
I'd like to dive into analyzing these three specific seasons, since each one has a cumulative plot that evolves throughout the episodes. This is something the creators have rarely done in the show-most other episodes stand alone and don't require too much knowledge of the previous ones.
This is perfect for us because we should be able to use TF-IDF to reveal what each season is about! You can imagine that this wouldn't be too meaningful for seasons where each episode has a different plot, right?
Let's examine the most significant words of these seasons in the following bar plot. The TF-IDF x-axis values are multiplied by 1000 so they're more readable.
# is needed in order to properly display the following bar plot.
x = "Top 10 most significant season topics"
Let's now see how well the analysis turned out. The storyline of Season 18 focuses on Randy Marsh having a musical career as Lorde-and that happens to be the number-one word on our list! They also make fun of gluten-free diets throughout the season. A secondary but nonetheless big focus in this season is on virtual reality.
Season 19 has even stronger messages. It begins with an announcement that there will be a new principal: PC Principal. He starts promoting a PC (politically correct) culture because South Park is obviously full of racists and intolerant people. Mr. Garrison doesn’t take it anymore and starts running for President of the United States alongside his running mate Caitlyn Jenner—a former decathlon Olympic champion, Bruce Jenner, who had a sex change operation. The last important topic of the series is ads that negatively influence everything and even start being indistinguishable from regular people.
In Season 20, it was all about SkankHunt42, Kyle’s dad. He became a troll who enjoyed harassing all sorts of people online. It ended with the fictional Danish Olympic champion Freja Ollegard killing herself. Denmark then created a worldwide service called trolltrace.com that was able to identify any troll online. Eric Cartman also started dating Heidi Turner and tried to get to Mars with the help of Elon Musk’s SpaceX company.
If you compare the bold words with the ones in the bar plot, you’ll see that our TF–IDF analysis did a great job identifying the topics of these seasons! There were also a few words that were picked up that I didn’t mention (like drones or handicar); those were mainly significant in a few episodes, but they made it to the top 10 anyway. But overall, the method did great!
What Are the 3 Most Popular Episodes About?
I’ll do the same plot I did for the three seasons but now for the three most popular episodes (according to IMDB). Because episodes generally have a single, central theme, the results should be more accurate. Let’s find out!
# created the top 10 tf-idf words for the three seasons.
# popular South Park episodes.
x = "Top 10 most significant episode topics"
I’ll try to describe each of the three episodes using a few sentences with their true storyline.
Scott Tenorman Must Die: Scott Tenorman, a ninth grader, sells his pubes to Eric Cartman. Eric later realized that he got tricked and wants to get back at Scott, so he trains Mr. Denkins’s pony to bite off Scott’s wiener. Eric ends up having Scott’s parents killed and serves them to Scott in a chilli con carne at a chilli festival. The episode ends with Scott’s favorite band, Radiohead, coming to the festival and mocking Scott.
The Return of the Fellowship of the Ring to the Two Towers: Stan’s parents rent a porno video tape called Back Door Sl**s 9 and The Lord of the Rings. The boys are given a quest to deliver the LOTR movie to Butters. The two tapes get mixed up, though, and Butters is given the porno tape instead.
Make Love, Not Warcraft: The boys play World of Warcraft but encounter a player who’s even stronger than the admins and who starts killing innocent players. Their only way to level up to fight the bully character is to start killing computer-generated boars. Once they level their characters sufficiently, their only chance is to use the sword of a thousand truths to win the fight!
If you know these episodes, you must agree with these amazing results!
# word for each episode.
Characteristic word: hella","Episode name: Merry Christmas Charlie Manson!
Characteristic word: fingerbang","Episode name: Do the Handicapped Go to Hell?
Characteristic word: sars","Episode name: South Park is Gay!
Characteristic word: marjorine","Episode name: Follow That Egg!
Characteristic word: ichi","Episode name: The Return of Chef!
Characteristic word: chef","Episode name: Smug Alert!
Characteristic word: rabbit","Episode name: D-Yikes!
Characteristic word: canada","Episode name: Eek, A Penis!
Characteristic word: pimp","Episode name: W.T.F.
Characteristic word: decay","Episode name: T.M.I.
Characteristic word: bracelet","Episode name: Obama Wins!
Characteristic word: pc","Episode name: Where My Country Gone?
I showed you where to get all the South Park dialog.
You learned how to create a dataset of words from spoken lines. This tidy data format helps you perform powerful data analysis more easily.
You used ggplot2 to create very informative plots, some of them even interactive thanks to the plotly package.
You even used a statistical proportion test to compare characters to see who is the naughtiest one, for example.
You also saw most of the code alongside the text. There are a few bits that I intentionally omitted. To see all of the code, visit my Github page as usual.
If you need a bit more practice with R to understand any of the code we worked with, be sure to check out the Data Visualization 101 course on Vertabelo Academy; you’ll learn how to use the ggplot2 package that I used to produce my plots.
As always, reach out to me if you need any help with data analysis!