Thousand Roads

A Pokémon fansite dedicated to the creative side of the Pokémon fandom, especially fanfiction.

Lies, Damn Lies, and Fanfiction: Introduction

Last Updated July 20 2015

I'm a bit of a sucker for attempting to combine things that I'm working on with fandom, and recently I've been floundering around in the world of statistics and machine learning. Naturally, all I could think about was how I could link those topics with Pokémon. It seemed to me like Fanfiction.net's database contained plenty of information I could sink my teeth into, and asking questions of it might produce results other people would be interested in seeing as well. That in mind, I set about writing a scraper to collect information on Pokémon fanfiction on FFN, then preparing a few articles about what I was doing with it.

To start off with, the dataset. It was collected on July 31st, 2012, and saved as a SQLite database, which you can download (warning: 20MB) if you would like to play along or try some analyses of your own. It contains data on 44,346 stories contributed by 20,249 authors, including anything you can see about a 'fic from its summary on the just-in page: title, summary, number of reviews, genres, wordcount, etc. Note that FFN removed the "General" genre option shortly before I downloaded the dataset, so that entry in the "genres" table is currently unused.

Throughout, I'll primarily be using R to explore the data and create plots. If you want to try it out yourself, you'll need your own copy of R (it's free and runs on all major operating systems), as well as the RSQLite package if you want to interface directly with the provided database. Although the plots I'll be using can be generated using R's built-in functions, I'm going to opt to use the ggplot2 package, which makes getting prettier results far easier, instead. R is lousy with packages, and some others may be used for specific analyses (which will be noted at the time), but those two are the main ones.

And, finally, because I'm doing this in part as an exercise for myself and other people might benefit from being able to follow along, I'll be including the code necessary to reproduce the analyses in each article, as well as a little discussion of the math involved in some cases. Under the assumption that "most" people who will be looking at the articles will be more interested in the bottom line--the results--I'm going to hide this content under "read more" tags. Simply click the "read more" flag to expand the hidden content, then "collapse" to hide it again. Like so:

Read More...

Got all that? So much for housekeeping, then. Let's get started with a look at some of Pokémon fanfiction's "top tens."

Read More...

Fics Long and Short

I had been planning to start off with some of the numbers that are inherently of the most interest to a review-obsessed fanfic writer, checking out the most popular stories in the category for various criteria, but FFN just went and added the ability to sort by reviews, favs, and so on, so that's easy to find out without whipping out any SQL. The search system still isn't very sophisticated about combining search criteria, though, so it can't yet tell us about the most-reviewd one-shots, for example:

StoryAuthorReviews
A Hot Summer DayKristina Chang191
The Best Lemon in the WorldI-PWN-AT-BASS145
PokemonPerdy Toes121
NaturalBexmar111
Undeniable FactsTsukiaa90
Why I Hate YouXenolicious87
My ValentinexxWNxxLove82
I'm an IdiotCar80
The Theory of TriosAngelicFairy78
Happy AccidentMaia's Pen73

Read More...

Not surprisingly, these are almost all shipping 'fics--the popularity of romance in general is attested to by the genre stats later in this article. Number three is the odd one out, being categorized as pure humor. The most highly-reviewed 'fic has an order of magnitude fewer responses than the most-reviewed chapterfic, but the fact that the same person can respond to multiple reviews of a chapterfic, whereas they only get a single shot at a one-off, no doubt plays a large role in this. It would be interesting to see whether the number of reviewers is similarly far higher for chapterfics, or whether you get around the same number of people responding to chaptered 'fics and one-shots--but that question can't be addressed by the data I have right now.

It's also possible to look at the longest (in terms of wordcount) 'fics around:

StoryAuthorWords
Brave New WorldRi22,188,441
Latias' JourneyRi21,273,300
Guardians of PokemonMr Panda764,874
The Darkness Withintracefan669,265
Blessed DefianceRocketlord6485662,729
Tales of FlameBuwaro654,714
Pokemon: The Final JourneyPaulXion638,341
AAAML: Diamond an Pearl versionEchidnaPower638,341
The Longinus DirectiveBlue Bongo581,025
PedestalDigital Skitty545,006

Read More...

If the word counts produced by FFN are based on criteria anything at all like those used on Wikipedia, these are some truly impressive fanfics: if published, they would all appear on Wikipedia's list of longest novels. The top seven beat out War and Peace, and both 'fics by Ri2 are longer than the longest novel. In fact, all seven books of the Harry Potter series combined add up to 1,084,170 words, meaning that if you read both of Ri2's stories up to their current point of completion, you've done more than the equivalent of reading Harry Potter three times over--and Ri2 has written many more than just those two 'fics. Clearly, Ri2 just needs to file the serial numbers off, publish one of those suckers, and collect that Guinness World Record.

And, on the other end of the spectrum, the masters of brevity:

StoryAuthorWords
No Matter Whatanimechic211
New championskaterman1
Thy's daredevil storyRead it if you dareYou're So Left1
Wreck the Hall of LegendariesWolffez1
Malin Kokimiyoi's Big Adventure!Chibi Kero2
Shortest stories ever!JLeashDaBeast2
Abandoned to AcceptedEclipse-Chan3
Hellgar Book 2: The PastApkitsune3
CharmanderLina Inverse8
Pokemon Mystery Dungeon sky Eve's JourneyWriter of Wolf Fang9

Read More...

"No Matter What" definitively uses the fewest pixels of the lot, although one of the reviews for "Wreck the Hall of Legendaries" suggests that it, too, once housed a single period. At this point, though, its contents are a question mark--the plot thickens!

Props then to "Charmander" for being the shortest apparently-legitimate attempt at writing in the Pokémon section--though "Shortest stories ever!" is at least almost a complete sentence.

Fics Living and Dead

Here's an issue that's altogether too close to my heart: fanfics that take years to finish. What are the longest-running tales out there?

StoryAuthorPublishedLast Updated
EvolutionJaxxon1/29/20013/31/2012
The Dark Side of InnocenceAnna Sartin7/28/20004/28/2011
Of Moonlight Shadows and Echoes PastZeoViolet2/23/20025/8/2012
Hidden JourneyPikajenn9/21/200110/29/2011
Species vs SpeciesCSakuraS1/10/200211/11/2011
A Tale of an Eevee & Two Trainers Re:ReMakeSissy8/1/20021/24/2012
GS Distress II: RevelationsMirai Gensao3/14/20027/22/2011
Midsummer NightsTrish12/8/200110/29/2010
The Sweetest FeelingSatsumi9/18/20024/9/2011
Pilot LightTashojiri1/9/20036/19/2011

Read More...

I actually grabbed eleven records here rather than ten, since the original number nine (Ash & Mistys House Party) only updated recently with an author's note saying that someone else was taking over the story, which I don't think counts.

The longest-running 'fic's been going on for over eleven years now, and almost half have a decade under their belts. Perhaps even more remarkable is the fact that most of them are either completed or still alive; Species vs Species is the only one explicitly labeled dead, and four more haven't updated in a year. A full half, though, have authors dedicated enough to keep producing for something like a decade, and it's certainly possible that some of the dormant ones aren't goners yet.

Also not quite appearing on the list are two stories I know well: The Quest for the Legends clocking in at fourteenth (6/25/04 - 6/8/2012) and Clouded Sky at twenty-first (11/25/04 - 6/10/2012). Neither of them are completed--but I have no doubt that they both will be in due time.

Of course, by far the majority of 'fics don't click along for years--they simply die. Using a fairly lax definition of "dead"--"Is not marked complete, and hasn't been updated for at least one year"--we can see about what portion of all chapterfics do just that:

Read More...

It's actually not nearly so bad as I thought. Of the 44,000 stories in the database, around 18,000 are complete. Of course, 14,000 or so of them are one-shots--so of all the chapterfics started in FFN's Pokémon section, only about 4,000 have been finished thus far. This is despite the fact that chapterfics are clearly more popular to write than one-shots. Over double that number are "dead" by my criteria. This isn't much in comparison to the number currently in progress, but it doesn't inspire confidence in the chances of those stories being finished. All other things being equal, any new chapterfic you spot being started today probably has around a 33% chance of reaching completion--and as we've seen above, that could easily take years.

There's a little wrinkle to this, though: back in the day, there was no way to indicate whether a work was in progress or completed. The option wasn't added until some time after I joined the site, which means that there are at least five or six years of fanfics that are "not complete" by default. Without knowing exactly when the "completed" option became available, I can't distinguish between 'fics that were simply left hanging and those that were either one-shots or chapterfics that were completed before the option to indicate completedness was added. This shouldn't be a big deal if the distribution of one-shots, dead, and completed 'fics from that era is about equal to one seen for 'fics today, but that's a dangerous assumption to make, and perhaps one that I'll look at more in a later article.

My definition of dead is perhaps a little strict as well: I know of several fanfics that have gone over a year without an update, only to rise from the grave and, on occasion, stagger on into completion. I've also discounted those deadfics where the author decides to leave the story on their profile but also mark it complete to indicate that there will be no further updates. So, in the end: it's hard for a fanfic aficionado to tell whether a story is definitively dead or not, and even harder for someone looking only at this data. One thing and another, though, I'd say that expecting two chapterfics to die for every one that gets finished sounds like a reasonable estimate--if a more optimistic one than I'd expected, given my experience in the fandom. One of the things I'm most itching to try with this data set is seeing whether I can predict how likely a chapterfic just starting out is to eventually reach completion, so I'll surely be coming back to these sorts of issues in later posts.

Fics Common and Rare

Moving on from looking at the characteristics of individual 'fics, I'd like to spend the last section of this article considering kinds of 'fics instead. How prevalent is one type of 'fic versus another? One of the easiest categorizations to look at is genre, since it's right there on the fanfic's summary, chosen from a set list of options.

If you were going to pick a story at random out of FFN's Pokémon 'fic database, what genre would it likely belong to? It's easy to figure out the distribution of 'fics by genre:

Read More...

So, if the pokémon category has ever seemed like a morass of endless shipping to you--it more or less is! Nearly half of all stories are categorized as "romance." (Though of course this number doesn't say anything about shipping specifically. Adventure comes in at a healthy second, which suggests to me that the most-written kinds of stories are shipfic and journeyfic, or perhaps a combination of the two.

Unfortunately, it's difficult to see in this plot that there actually are a few Westerns in the fandom--twenty, to be precise. However, this is clearly the least popular genre, with nearly an order of magnitude fewer stories than the next-least-common genre, "spiritual," which has 191 constituents. These both strike me as particularly rare genres in general fiction as well; spiritual's lack of popularity may be a result of ill definition--just what is a spiritual 'fic, anyway--but while Pokémon has its own wild, wild west (Orre!) for writers to play in, I imagine most people writing stories set there don't conceive of them as "westerns."

Those are the kind of stories people like to write--but how about what people like to read? We can look at the number of reviews given to all fics in a genre in a very similar manner.

Read More...

By and large, the results aren't a surprise. Romance and action/adventure are still the clear winners, with humor in third. A close look at some of the less popular genres shows a few mismatches, though: while there are more tragedies written than either suspense or supernatural stories, for example, they tend to receive fewer reviews than stories in either of the other two categories. More "friendship" stories are written than those termed "drama," but drama 'fics are reviewed substantially better. Angst 'fics also receive less than their share of reviews in proportion to how many of them are written.

For the most part, though, the two graphs correspond pretty well. It seems that what people write and what people want to read are more or less in sync--at least at the coarse level of genre.

Of course, it's also possible to look at popular combinations of genres as well. Plotting all of these on one graph would be getting a little out of hand, but it's easy enough to pull up another top ten:

First GenreSecond GenreFics
RomanceNone4,763
NoneNone3,511
AdventureNone3,435
RomanceHumor2,531
AdventureRomance2,198
HumorNone1,718
RomanceAdventure1,651
RomanceDrama1,419
AdventureFriendship1,356

Read More...

The most popular genre mix isn't a mix at all--it's "romance" with nothing else. The fact that romance, or something mixed with it, dominates the top ten, isn't all that surprising, given how far ahead it is of the other genres in terms of popularity. What is rather interesting is that "no category" takes the second place spot. This possibly has something to do with the fact that all the 'fics categorized as "general" have recently been considered uncategorized, but there's no way to know for sure. It was possible to submit stories without genres in the past, and information on how popular that option was has now been muddied by the removal of the "general" category.

One final item of interest on this topic: A close examination of the data reveals that FFN does not enforce a "stories can only have a genre once" policy, as can be seen by the fact that there are seven stories with the label "Adventure/Adventure":

TitleAuthor
Pokemon 4: Devastationzootswoman
Pikachu SpyMewGal
The Pokemony JourneyTodd Maxwell
Pokemon RevengeLaw BreakersTorrence Tang
To only be a MasterPhenoix of Fire
PokeQuestMenno Simons
A Boy and His Pokemon The Legacy of ToddVersatz

Read More...

This means that our count data is a little skewed--the odd 'fic is getting counted twice. However, the number of times that this happens is small enough to have no real impact on the overall results, so I'm going to ignore it here and in the future.

And with that, I'm going to call this first look at the data a wrap. This first installment was a bit rambling and more an introduction to the format and data than anything else; in future, there will be a bit more of a coherent theme to what gets investigated, as well as, hopefully, some actual statistics. Next up: looking at trends over time. How is fanfiction today different from the fanfiction of yesteryear? How might it change in the future?

eevee

reppad98

This was very interesting to read, thank you!

rayquaza-mega

Negrek Admin

Website: Thousand Roads

Thanks! Glad you enjoyed it. I seriously need to update this one (and expand on it...). It's super out of date.

croagunk

Jenn

This is a really neat page! Hidden Journey isn’t fully dead, I promise. I’ve just been writer’s blocked, then I lost what I had written so I had to redo it, plus real life sucks. 😭 I’ve also written a couple of other stories for different fandoms, but under different author names.

rayquaza-mega

Negrek Admin

Website: Thousand Roads

Wow, thanks for stopping by to leave a comment! It's always fun to hear from people whose stories showed up in this article. I'm glad yours isn't quite dead yet, and it's great to hear you're still writing. Best of luck with all your stories!

koffing

RisingDawn

Wow really interesting. I knew Romance fics were popular but damn. I thought Action/Adventure would be more popular.

Leave a Comment

Please keep comments PG and related to the content of the page. For more general chat, visit the guestbook. Spam and other inappropriate posts will be deleted. Posts use a Markdown dialect for formatting. These are common formatting commands:

  • **bold** = bold
  • *italic* = italic
  • [link text](http://www.example.com) = link text
  • ---strikethrough--- = strikethrough
  • &slugma; = slugma

You can find a full list of formatting options here, but note that images are disabled.

Required fields are marked with an asterisk.

677
https://thousandroads.net/fanfic/fanfic-intro
thousandroads.net
crawl=CC-MAIN-2021-04
41011
application/http; msgtype=response
sha1:ESWXBMDGT5SOIP7BRQ7ECJQ7TBR4QZQR
<urn:uuid:bb64173f-ee56-408e-a5c3-82f3f9d90155>
2021-01-24T18:27:19Z
104.236.47.68
text/html
sha1:4Z4HMOB5CWOPQPGBT2SZUPERJFQDR6BR
<urn:uuid:0485ab0e-7ae0-43e1-b7fd-57e54fca4ea8>
https://thousandroads.net/fanfic/fanfic-intro
response
<urn:uuid:054c0165-3181-4107-a088-e142d8ce172b>
isPartOf: CC-MAIN-2021-04
publisher: Common Crawl
description: Wide crawl of the web for January 2021
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-90.ec2.internal
software: Apache Nutch 1.17 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.2-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/