STORY TRACKER

Culturomics and the new Google tool for tracking cultural trends

Two hundred years of history in the form of 5,195,769 digitised books can now be probed for cultural trends using Google's new culturomics tool

Email updates and links to science@guardian.co.uk. We'd like to hear about your own research using the new tool. What trends have you unearthed? The paper's authors have agreed to analyse some of the best ones for us

Read the research in Science (register to view it in full)
  • theguardian.com,
  • Jump to comments ()
Google Ngram for 'pagan, christian' View larger picture
Google Ngram for 'pagan, christian'. Click to enlarge. Photograph: Google

Ever wonder which is more popular, "math" or "maths"? At Travels in a Mathematical World, the authors look at the rise and fall of these two ways of shortening mathematics.

The graph shows that the American "math" has always been more popular than British "maths" but the use of the American English word rocketed from 1940 onwards.

From these data it seems math has been much used through history, earlier and to a greater extent than maths. I have no preconceptions about which came first; what seemed surprising to me was how early either was used.

Some very interesting graphs of ngrams from Information is Beautiful, including the occurrence of different types of recreational drugs (cocaine is suprisingly common in Victorian times), ketchup v salsa, gay v homosexual and religion vs science.

Tom Beesley got an "intriguing result" when he compared the usage of the words "pagan" and "christian" between 1800 and 2000 (see above). After a peak in around 1810, it seems the dominance of Christians in the literary corpus went into steep decline in the 1840s and they have been bumping along the bottom ever since. Meanwhile, the popularity of Pagans has remained remarkably stable over the same period.

That the word "pagan" is now in much wider usage than "christian" is remarkable indeed.

But as Christopher Collins comments in an email: "It seems that this search engine does not really answer any questions, but rather generates hypotheses that could direct research into culture." Collins had fun comparing "present" and "future" in English and other languages. It's certainly intriguing, but what it actually means is hard to say.

Our own Martin Robbins has used the tool to identify a marked cultural trend in favour of a certain liberal-leaning newspaper.

The Guardian kicks sand in the face of the Daily Mail #ngrams - http://bit.ly/dEfZhQless than a minute ago via TweetDeck

A vast collection of Google ngrams is already being amassed at #ngrams on Twitter.

A bona fide linguistics researcher has weighed in with a blopost at the Language Log. Geoff Nunberg of the University of California Berkeley welcomes the research, and the new Google tool, but looks forward to more bells and whistles:

The big news is that Google has set up a site called the Google Books Ngram Viewer where the public can enter words or n-grams (to 5) for any period and corpus and see the resulting graph. They've also announced that the entire dataset of n-grams will be made available for download. Some reports have interpreted this as meaning that Google is making the entire corpus available. It isn't, alas, nor even the pre-1923 portion of the corpus that's in public domain. One can hope…

At present, that's all you can with this. You can't do many of the things that you can do with other corpora: you can't ask for a list of the words that follow traditional for each decade from 1900 to 2000 in order of descending frequency, or restrict a search for bronzino to paragraphs that contain fish and don't contain painting, etc. And while Lieberman Aiden and Michel made an impressive effort to purge the subcorpus of the metadata errors that have plagued Google Books, you can't sort books by genre or topic. The researchers do plan to make available a more robust search interface for the corpus, though it's unlikely that users will be able to replicate a lot of the computationally heavy-duty exercises that the researchers report in the paper. But my sense is that even this limited functionality will be interesting and useful to a lot of humanists and historians, even if linguists won't be really happy until they have the whole data set to play with.

The Wall Street Journal reminds readers how controversial Google's project to digitise every book on the planet has been:


The online library project has been hobbled by lawsuits, copyright disputes and fears over the potential for the company to have an information monopoly. "There have been computational hurdles, scientific hurdles, organizational and legal hurdles," said mathematician Erez Lieberman Aiden at the Harvard Society of Fellows, who helped create the database.

To avoid copyright violations, the scientists are making available the vast catalog of frequency patterns of words and phrases, not the raw text of books. Google Labs posted freely downloadable data sets and a special viewer at http://ngrams.googlelabs.com Thursday. These data sets consist of short phrases—up to five words —with counts of how often they occurred in each year.

The tool suggests that the relative literary importance of countries has been converging over the past 200 years (see above) – from the runaway dominance of "England" in the 19th century, through a steep decline toward mediocrity in 1960, and a much tighter, lower-frequency range of country names in 2000, including India, China, Germany and France. Surprisingly, perhaps, by a small margin England still gets the most mentions in the most recent texts in the database.

Thanks to @EdYong for tweeting this to our attention.

Under the somewhat combative headline "Reading by numbers: Science invades the humanities", The Economist neatly summarises how the Harvard researchers conducted their study with the help of Google's burgeoning books database. It notes that the search giant has digitised 15m of the estimated 130m titles printed since Johannes Gutenberg perfected his press in the 15th century.

Culturomics is a work in progress, as the researchers freely admit. And as The Economist says:

Clearly, books do not exhaust the whole of human culture. In recent decades their relative importance has waned. Nor are the books Google has already chosen to scan necessarily a representative sample of literature across the ages. This means that any findings based on them ought to be treated with caution.

Still, Dr Michel and his team hope that their approach will spur a more rigorous, quantitative approach to the study of human culture. In fact, their paper doubles as a manifesto for a new discipline. They dub it "culturomics", making them the first clutch of culturomists. More are sure to follow—whether or not this particular, clunking neologism survives.

Science writer Philip Ball has posted a longer, unedited version of the news piece he wrote for Nature at the blog Homunculus. In his introduction he adds a proviso about the tool:

There's the danger that in the wrong hands this kind of thing could end up supplanting textual and historical analysis with lexical statistics. But there's clearly a wealth of interesting stuff to be gleaned this way. And I thoroughly approve of a paper that is not afraid to show a sense of humour.

Ed Yong at Not Exactly Rocket Science believes the grand title "culturomics" given to this new field of cultural research is well deserved:

As the team says, the corpus "will furnish a great cache of bones from which to reconstruct the skeleton of a new science." There are strong parallels to the completion of the human genome. Just as that provided an invaluable resource for biologists, Google's corpus will allow social scientists and humanities scholars to study human culture in a rigorous way. There's a good reason that the team are calling this field "culturomics".


Yong has written a great summary of the paper, but concludes wistfully:

A note on language: This was one of the most well written papers I've ever had the pleasure to read, full of wit and flair. I've highlighted papers from these researchers before for exactly the same reason and they haven't disappointed this time round. It's vexing for a science writer, really – when you could just as well edit a paper down for length rather than translating it, it makes you question the future of your profession!

Email updates and links for this story tracker to science@guardian.co.uk and add your comments about the new discipline of "culturomics" below.

We want to hear about your own research using the new tool. What linguistic and cultural trends have you managed to winkle out of Google's searchable database? The paper's authors have agreed to analyse some of the best ones for us in a forthcoming article.

Story summary

Google and scientists at Harvard University have launched a searchable database of digitised books that they say will be an essential tool in a new discipline they're calling "culturomics".

The researchers at Harvard have published a paper in today's issue of Science describing some of the research they have already conducted using the database.

It comprises more than 5m books – both fiction and non-fiction – published between 1800 and 2000, representing around 4% of all the books ever printed. Dr Jean-Baptiste Michel and Dr Erez Lieberman Aiden of Harvard University have developed the search tool, which they say will give researchers the ability to quantify a huge range of cultural trends in history.

Today's best video

Today in pictures

In the hours and days after an important science story breaks, the Guardian science desk provides updates and links to external sources of information and comment

Google Ngram for 'pagan, christian'

Google Ngram for 'pagan, christian' Photograph: Google

;
http://www.theguardian.com/science/2010/dec/16/culturomics-google-tool-cultural-trends
www.theguardian.com
crawl=CC-MAIN-2014-35
149669
application/http; msgtype=response
sha1:DNRFOS4OMLMB4TJOLRGUYQRWEK2LUVSI
<urn:uuid:07ed27ab-8234-4e91-a438-f8e93d3946c9>
2014-08-28T02:59:23Z
23.235.46.207
sha1:J462FAKGU7X2KWZGU4TZB3LOENI6W2BX
<urn:uuid:1b71a7fa-0f6e-4394-91f4-bb4972727174>
http://www.theguardian.com/science/2010/dec/16/culturomics-google-tool-cultural-trends
response
<urn:uuid:9e55d657-d05e-4e31-af46-e36ee9bfbb99>
robots: classic
hostname: ip-10-180-136-8.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2014-35
operator: CommonCrawl Admin
description: Wide crawl of the web with URLs provided by Blekko for August 2014
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf