Mining for meanings

Yes, I have a suit. On 8 May at the National Library of Australia I gave my suit an outing as I delivered my Harold White Fellowship presentation. Thanks to everyone who came along.

If you missed it or want to relive the fun, the NLA has made a podcast available. My slides are also online, so you can follow along for the full audio-visual-not-quite-3D experience.

Use your arrow keys to navigate through the slides, and yes the first page is intentionally left blank. If you linger for a bit on slide two or three, you’ll see the Trove API in action. The presentation itself was constructed using deck.js.

The slides also include links to lots of different examples and demos, and introduce my new favourite plaything. I don’t really know what to call it yet, or what it’s actually for, but it makes me happy, and it makes me think. TF-IDF FTW. I’ll write up some more details shortly.

Tagged with: , , ,
Posted in digital humanities, speeches

The new QueryPic (or what a difference an API makes)

It seems a bit late to be introducing the newest version of QueryPic. Folks are already using it to explore the contents of digitised newspapers made available through Trove and Papers Past. Some, like the National Library of New Zealand, Andrew S. Bowman and the Carnamah Historical Society are already blogging about it. But I suppose I’d better document a few things…

As I noted in my post about QueryPicNZ (yes I now have a rather confusing proliferation of QueryPics), I was waiting for the Trove API to become public. Last week I noticed a little ‘API’ link pop up in the Trove footer and so I set to work…

"The past" versus "the future" in the new QueryPic

My original version of QueryPic (recently reviewed in the Journal of the Digital Humanities) used a series of Python scripts to harvest and scrape content from the Trove web pages. This meant that you had to download the scripts and be code-confident enough to run them in a terminal. It’s still a useful tool and I’ll be updating it as well, but I wanted to create something quicker and simpler that encouraged people to explore and play.

The latest version of QueryPic (QueryPic+, QueryPic Web, QueryPic 2.0?) simply runs in your browser. It uses JQuery to grab data on the fly from the Trove and DigitalNZ APIs. Like previous versions, it uses the HighCharts library to turn the data into pretty graphs.

What does it do? It’s really pretty basic. QueryPic just displays the number of articles matching your search query over time. By default, these are displayed as a proportion of the total articles available for that year, but a dropdown field lets you switch to view the raw numbers. It’s simple, but it’s also remarkably evocative, suggestive and fun. Just try it!

Why stop at just one query? To compare frequency patterns you can add as many as you like. Just keep entering new words or phrases.

If you notice an interesting peak or trough you can just click on it and another API request will be fired off to retrieve the first 20 matching articles. So it’s also a new way of exploring the newspaper databases themselves.

There are plenty of limitations — not all newspapers are digitised, for example, and the quality of the OCR is patchy. The National Library of New Zealand’s post does a great job summing up a number of issues relating to Papers Past. It’s not magic, it’s not perfect, but is it useful? I think so.

Tasks for the future:

  • Create some sort of backend that makes it easy to save , share and cite your query data. The ‘share’ link just regenerates the graph which, of course, might change as new articles are added to the databases.
  • Make it possible to add more complex queries — I want to keep the interface simple, so I’ll probably create a bookmarklet to take any Trove or Papers Past query and display it using QueryPic.
  • As I mentioned over at the WraggeLabs Emporium, I intend to rewrite my various Trove tools to work with the new API. This will include the classic Python version of QueryPic. I still think it’s useful for harvesting your own data.
The code is on my GitHub site and you can also follow updates at the QueryPic page in the WraggeLabs Emporium.

 

Tagged with: , , , ,
Posted in digital humanities, experiments

QueryPicNZ

You may have noticed I have a bit on an interest in exploring ways of using digitised historical newspapers. In the last year or so I’ve spent a lot of time scraping, mining, processing and visualising content from the Trove collection of digitised Australian newspapers. But what about other countries?

Recently I was invited to a digital history workshop organised by Sydney Shep (@nzsydney) at the Victoria University of Wellington. In between sessions I started to play with the DigitalNZ API guided by Chris McDowall (@fogonwater). In anticipation of the forthcoming Trove API I’d already done a bit of work converting QueryPic to run in the browser. It didn’t take long to adapt this to work with New Zealand newspapers available through Papers Past.

So presenting for your enjoyment and education… QueryPicNZ.

Wind, rain and snow in QueryPicNZ

Like QueryPic, the New Zealand version graphs newspaper search results over time. But thanks to the DigitalNZ API it has a number of advantages:

  • it runs in your browser — no need to download or run any scripts
  • results appear almost instantly
  • easy to combine queries — just search on a new word or phrase
  • easy to remove queries — just use the ‘Clear last’ button
  • easy to share — just copy the provided link or use the Tweet button

It’s limited to simple word or phrase searches at the moment, but eventually I’ll add the ability to process more sophisticated queries. I also want to add a way of saving, sharing and citing graphs. For now the ‘share’ link simply regenerates the graph, so if the content has changed the result could well be different.

The code is available on GitHub.

Ultimately, I want to combine Trove and Papers Past so that you can query and combine content from either Australia or New Zealand… perhaps even other countries?

Tagged with: , , , , ,
Posted in digital humanities, experiments

Mining the treasures of Trove

In February I made a quick dash to Melbourne to talk at VALA2012.

The paper I originally submitted, ‘Mining the treasures of Trove: New approaches and new tools’, provided a general introduction to the use of digitised historical newspapers and the possibilities of digital history. You can download the pdf from the VALA2012 proceedings, or view online at Scribd.

I ended up presenting something a little different, focusing on my recent work around 1913 and extracting editorials from the Trove newspaper database. You can view the slides on Slideshare or watch a video of the whole presentation on the VALA2012 site.

Tagged with: , , , , ,
Posted in conference presentations, digital humanities

Extracting editorials #3

By my own criteria I’ve already failed… I started this series of posts with the intention of documenting the process of finding and extracting editorials as I was actually doing the work. But here I am about to describe some work I finished a few weeks back. Oh well…

In my previous instalments (here and here), I focused on the Sydney Morning Herald. Having continued the hunt for missing editorials I started in the last post, I’ve now got a CSV file with the urls of the first editorial published in every edition of the SMH from 1913. Good-o, I thought, I can now start harvesting and analysing some content.

But then ensued a crisis of faith. The whole point of this exercise was to be able to build up some comparisons  – between newspapers, between states, between the city and the bush. But the process of actually finding the editorials seemed beset with difficulties. Could the rules I developed for the SMH be applied elsewhere? Could I ever assemble a useful set of editorials without large amounts of human intervention? I decided to try a few quick experiments to see whether the whole project was worth pursuing.

I started with a few assumptions:

  1. The first (and only the first) editorial in any issue is headed with the name of the newspaper.
  2. Editorials are published on even numbered pages.
  3. Editorials vary in length between about 100 and 1500 words.

These assumptions were based on my own experience as a long-time newspaper researcher and on some preliminary poking around. For example, when I looked at The Argus I noticed that editorials were typically followed by news summaries. Unfortunately, these are treated as a single article in Trove, resulting in large blocks of text that are only part editorial. By specifying an upper word limit I hoped to filter these sorts of articles out. Similarly, there are sometimes brief announcements or publication details headed with the name of the newspaper. The lower word limit was intended to exclude these.

The next step was to harvest every article from 1913 that was headed with the name of its publication. I created a script to generate a list of all the newspapers that published issues in 1913. Then I called my existing harvester to download all the matching articles and save the details to a series of CSV files — one CSV file per newspaper.

In the previous instalment of this series I created a script to check the CSV output of my harvester for missing or duplicate dates. I extended this to perform a series of tests on each article based on the assumptions above. First, I filtered out articles on odd-numbered pages, then articles that were too short or too long. Finally I checked the remainder for missing or duplicate issue dates.

The details of the articles in each category were written out to JSON files. Using these files and a bit of JQuery magic I could quickly build a simple web interface that allowed me to explore the results.

Summary details of each newspaper

You can browse the summary results for the full list of newspapers, or you can drill down to view the actual articles assigned to each category.

Full details

I’ll save the full analysis for the next post, but if you play around with the results you quickly notice a few things. First, letters to the editor often include the name of the newspaper! If you look at The Mercury, for example, you’ll notice I’ve identified 1057 potential editorials — most of which are letters. Fortunately they should be fairly easy to filter out. In most cases the ‘even numbers only’ assumption worked pretty well, and the word length filters did remove quite a lot of false positives. There are still plenty of problems, but I’m encouraged enough to continue. Yes, there will be a Part #4!

 

Tagged with: , ,
Posted in digital humanities, experiments

2011 — the year of little sleep

2011 was a busy year. It’s hard to believe that it was only February when I first posted about my experiments mining the contents of the Trove newspaper database. Since then I’ve developed a set of digital tools, organised THATCamp Canberra, given a series of presentations on the possibilities of digital history, pushed ahead with Invisible Australians, and tried to develop my own digital research program. Oh yes, and endeavoured to earn enough money to feed the kids and pay the mortgage…

It looks like 2012 could be even busier, so before I lose track completely, I thought I’d pull together some of the past year’s exploits for handy reference. So here’s (most of) my presentations for 2011…

8 June 2011 — ‘Confessions of an impatient historian’
Scholars’ Lab, University of Virginia

18 August — ‘Digital history: new tools and techniques’
National Museum of Australia

24 August — ‘Hacking the archives’
Archival description in an online world, Recordkeeping Roundtable, Sydney

5 September 2011 — Digital research methods
Cultural heritage students, University of Canberra

14 September 2011 — ‘Every story has a beginning’
Keynote presentation at the Indexing See Change Conference (Australian and New Zealand Society of Editors)

13 November 2011 — ‘Digital history: new tools and techniques’
Dragontails 2011: 2nd Australasian conference on overseas Chinese history & heritage, Museum of Chinese Australian History, Melbourne

30 November 2011 — ‘It’s all about the stuff’
National Digital Forum, Wellington, New Zealand

7 December 2011 — ‘An introduction to digital history’
Digital December, State Library of NSW

Posted in conference presentations, digital humanities

It’s all about the stuff — the movie

Videos from NDF2011 are now available online. Here’s the movie version of my talk It’s all about the stuff. I seem to spend a lot of time in the shadows…

Tagged with: , ,
Posted in conference presentations, digital humanities

QueryPic

Back when I was looking at ‘When did the Great War become the First World War?‘ I promised a detailed post on how I constructed the graphs. But of course I got distracted. Then I started adding new features to the script and redesigning the graphs, so…

Anyway, the result is a rather neat little gizmo henceforth named QueryPic (I got a bit sick of ‘search summariser’ and ‘graph-maker thing’). The first version just harvested data and left all the graph-making to you. But QueryPic does it all! It harvests the data and makes the graph. Woohoo.

Here’s an example showing ‘drought’ versus ‘flood’:

QueryPic features

  • Explore your Trove newspaper query over time in the form of a simple line graph.
  • Interactive — click on a point to retrieve sample articles from that date.
  • Combine data sources to compare queries.
  • Choose your interval — plot by year or month.
  • Switch views between total results and the proportion of all articles.

Running QueryPic

Yes, it’s a Python script and yes it runs on the command line. Let’s get that out of the way now. I don’t think I have the time and energy to develop cross-platform gui versions of all my tools. I’d rather spend the time adding new features or exploring new possibilities. Sorry, but until I have a wealthy benefactor or a technical support team, I think that’s the way it has to be. In any case, the code is all there – so build your own gui!

Actually, if I did have the time and energy I don’t think I’d build a standalone gui anyway. What would be much cooler would be a web service, where people could run, share and combine their queries. Social graph-making! A celebration of serendipity! A historical playground! Hmmm…

But for now there’s this python script. It’s dead easy to use. Starting from the beginning…

  1. Do you have Python installed? If you have a Mac or Linux the answer is yes. Fire up a terminal and type ‘python -V’ — see, I told you. If you have Windows you can get a handy installer. Do it.
  2. Get the source code. Just download this zip file and open it into a new folder.
  3. Open a terminal and cd into the new folder.
  4. Run ‘python do_totals.py [your Trove query]‘.
  5. Watch in excitement as the script chugs away retrieving data from Trove.
  6. Once the script is finished, go to the ‘graphs’ directory, where you’ll find your newly-created html page complete with fancy interactive graph.
  7. Open the html page in the web browser of your choice.
  8. Enjoy! Celebrate! Drink a toast in my honour!

Customising QueryPic

There are a number of optional arguments that you add to the command line to customise your results:

-n (or –name) [a query name]
Give a name to your query. The name is used to create filenames for the html and data files, it is also used in the legend of the graph. The default is to use the search keywords as the name.

-d (or –directory) [a directory path]
The full pathname of the directory/folder for your results. The default is a ‘graphs’ sub-directory in the current directory.

-g (or –graph) [a graph name]
Specify the name of the html file that’s created. This is useful for displaying multiple queries on a single graph. Just run QueryPic for each query, using the same graph name each time. The default is either the value specified by the -n parameter or a name derived from the search keywords.

-m (or –monthly)
Plot the query at monthly intervals. The default interval is a year.

What QueryPic actually does

QueryPic builds a simple visualisation of your search query in the Trove newspaper database. A list of search results is difficult to interpret and offers little context. QueryPic shows you the number of articles matching your query over time, enabling you reframe your questions, pursue hunches, or simply play around.

QueryPic takes your Trove newspaper query and looks for a date range. If it doesn’t find one, it assumes you want your graph to go from 1803 to 1954 (the complete contents of the newspaper database — except for the Women’s Weekly). QueryPic then strips out any date parameters from the query, so it can fire off the query within the start and end dates, at the specified date interval.

Date interval? In the previous version of this script you could only plot points at yearly intervals, so it was impossible to zoom in an see what might be happening over the span of a single year or two. But amazing advances in QueryPic technology mean you can now plot changes by month. Here for example is a new version of my Great War/First World War graph, focused on 1938–1946 and plotted at monthly intervals.

So for each interval within the date range QueryPic fires off a request to Trove. From the response it scrapes out the total number of results for that date. If the total is greater than zero, it then fires off a second request to find the total number of newspaper articles for that year. Your query results divided by the total number of articles gives the proportion of articles for that date matching your search query.

The number of results and the proportion are written to a javascript file, together with some other important information including the original query and the date the harvest was performed. Remember, the Trove newspapers database is always changing! QueryPic then grabs a copy of it’s own special html template and inserts a reference to this javascript file. For good measure, it also inserts a link to your original query. The file is saved under a new name, ready for you to open and explore.

The html file contains everything necessary to take your data and turn it into a graph. It does this using the HighCharts javascript library. Please note, that while licence conditions allow HighCharts to be redistributed as part of a non-commercial package, it is not free for commercial use. Check the HighCharts website for details.

Some examples

Plot ‘cat’ against ‘dog’ in a graph called ‘animals’:

python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat" -g "animals"
python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat" -g "animals"

Specify a directory for your results:

python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat" -d "/User/bill/Documents/graphs"

Plot results at monthly intervals:

python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat&fromyyyy=1920&toyyyy=1921" -m

Specify a name:

python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat" -n "Felines"
Tagged with: , ,
Posted in digital humanities, hacks

Extracting editorials #2

As I explained in the first of this series, I’m documenting my efforts to extract every editorial published in the Sydney Morning Herald in 1913 from the Trove newspaper database. It’s an experiment both in text mining and historical writing — an attempt to put the method up front.

While I didn’t think there was anything very thrilling in the first instalment, recording my thoughts and assumptions in this way has already proved useful. In a comment, Owen Stephens noted that his attempt to reproduce my search query produced fewer results. After a little bit of poking around I realised that the fulltext modifier, which I often use to switch off fuzzy matching, counteracts the ‘search headings only’ flag. So my query was returning results that had the string ‘The Sydney Morning Herald’ anywhere in the article.

Try it for yourself.

Here’s my original query — searching for fulltext:”The Sydney Morning Herald” in headings only (supposedly). You’ll notice that it returns 335 results and it’s clear from a quick scan that a number are false positives (they don’t follow the pattern for editorials).

Here’s Owen’s query — searching for “The Sydney Morning Herald” in headings only. It returns 294 results, without any obvious false positives.

So my attempt to disable fuzzy matching actually produced a less accurate result! Weird.

Actually, I think one important benefit of this sort of text mining is that it helps you understand how the search engines you’re using actually work. Once you start poking and prodding, the idiosyncrasies start to emerge.

Anyway, I harvested Owen’s cleaner result set and opened up the resulting csv file. As it seemed in Trove, there we’re very few false positives. Indeed there were only two articles that didn’t seem to follow the standard editorial format, and these were notes added to the editorial page. On the other hand, there were obviously about 20 editorials missing. I could have manually worked through the csv file to identify the missing dates, but I thought I’d try to create some tools that would do the work for me.

What I wanted was the details of the first editorial in every edition of the newspaper in 1913 — so there should be one, and only one, article for each day on which the newspaper was published. I needed a tool that would analyse the csv file and do two things:

  • identify dates that occur multiple times (false positive alert!)
  • identify dates that are absent from the result set (missing in action!)

The resulting code is all on GitHub if you want follow along. I wrote a Python script that opens up the csv file, extracts all the date strings, converts them to datetime objects and then saves them to a list. Once that’s done it’s pretty easy to loop through and find duplicates:

def find_duplicates(list):
    '''
    Check a list for suplicate values.
    Returns a list of the duplicates.
    '''
    seen = set()
    duplicates = []
    for item in list:
        if item in seen:
            duplicates.append(item)
        seen.add(item)
    return duplicates

Finding missing dates was a little more complicated, but Google came to the rescue with some handy code samples. All I had to do was set a start and end date (in this case 1 January 1913 and 31 December 1913) and create a timedelta object equal to a day. Then it’s just a matter of adding the timedelta to the start date, comparing the new date to the dates extracted from the csv file, and continuing on until you hit the end. If the new date isn’t in the csv file, then it gets added to the missing list.

if year:
        start_date = datetime.date(year, 1, 1)
        end_date = datetime.date(year, 12, 31)
    else:
        start_date = article_dates[0]
        end_date = article_dates[-1]
    one_day = datetime.timedelta(days=1)
    this_day = start_date
    # Loop through each day in specified period to see if there's an article
    # If not, add to the missing_dates list.
    while this_day <= end_date:
        if this_day.weekday() not in exclude: #exclude Sunday
            if this_day not in article_dates:
                missing_dates.append(this_day)
        this_day += one_day

I’ve tried to make the code as reusable as possible, so you can either supply a year, or the script will read start and end dates from the csv file itself.

All that left me with two more lists of dates: ‘duplicates’ and ‘missing’. At first I just wrote these out to a text file, but then I decided it would be useful to write the results to an html page. That way I could add links that would take me to the actual issue within Trove, helping me to quickly find the missing editorial.

Unfortunately there’s no direct way to go from a date to an issue — you first need to find the issue identifier. How do you do this? If you dig around in the code beneath the page for each newspaper title, you’ll find that the ajax interface pulls in a json file with issue information. You can access this through a url like: http://trove.nla.gov.au/ndp/del/titlesOverDates/[year]/[month]. Here’s an example for January 1913.

The json includes all issues for all titles in the specified month. So you then have to loop through to find a specific title and day. Once you have the issue identifier you can just attach it to a url:

def get_issue_url(date, title_id):
    '''
    Gets the issue url given a title and date.
    '''
    year, month, day = date.timetuple()[:3]
    url = 'http://trove.nla.gov.au/ndp/del/titlesOverDates/%s/%02d' % (year, month)
    issues = json.load(urllib2.urlopen(url))
    for issue in issues:
        if issue['t'] == title_id and int(issue['p']) == day:
            issue_id = issue['iss']
    return 'http://trove.nla.gov.au/ndp/del/issue/%s' % issue_id

My results file with links to Trove

Finally, to save myself having to cut and paste the missing dates back into the csv file, I added a few lines to write them in automatically.

So now I have a handy little html page, complete with dates and links, that I’m working through to find all the missing editorials. All I need for the next stage are the urls for the editorial and the page on which it’s published. I’m just cutting and pasting these from the citation box in Trove into the csv file. Once this is done I can start trying to find all the editorials.

PS: I noted in my first post that one benefit in finding the editorials was that the main news articles usually appeared on the page after the editorials. I’ve been thinking some more about ways to identify ‘major’ news stories. Word length perhaps? But not always. Hmmm, but major stories do seem to be published at the top of the page. After a bit more poking around in the code I found that there’s a ‘y value’ assigned to each article that indicates its position on the page. So if I harvest all the articles on the page after the editorials and then rank them by their y values? Interesting…

Tagged with: , ,
Posted in digital humanities, experiments, hacks

It’s all about the stuff: collections, interfaces, power and people

This is the full version of a paper I presented at the National Digital Forum, 30 November 2011.

In 1901, one of the first acts of the Commonwealth of Australia was to create a system of exclusion and control designed to keep the newly-formed nation ‘white’. But White Australia was always a myth. As well as the Indigenous population, there were already many thousands of people classified as ‘non-white‘ living in Australia — most were Chinese, but there were also Japanese, Indians, Syrians and Indonesians.

Here are some of them…

The real face of White Australia

The administration of what became known as the White Australia Policy created a huge volume of records, much of which is still preserved within the National Archives of Australia. These photographs are attached to certificates that non-white residents needed to get back into the country if they decided to travel overseas. There are thousands upon thousands of these certificates in the Archives. Thousands of certificates representing thousands of lives — all monitored and controlled.

But is is too easy to see these people as the powerless victims of a repressive system. There were many acts of resistance. Some argued against the need to be identified ‘just like a criminal’. Others exercised control over their representation, submitting formal studio portraits instead of mug shots.

Most commonly and most powerfully, people resisted the policy simply by going ahead and living rich and productive lives.

My partner, Kate Bagnall, is helping to rewrite Australian-Chinese history by overthrowing the stereotype of the culturally isolated Chinese man living a lonely, meagre existence surrounded by gambling and opium dens. By mining the available records, by reading against the grain of contemporary reports and by working with family historians, Kate is documenting their intimate lives — their wives, their lovers, their families and descendants — the sorts of relationships that sent a shudder through the edifice of White Australia. Power can be reclaimed in many subtle and subversive ways.

‘The real face of White Australia’ is an experiment. It uses facial detection to technology to find and extract the photographs from digital copies of the original certificates made available through the National Archives of Australia’s collection database. The photographs you see here come from just one series, ST84/1. There’s no API to the collection so I reverse-engineered the web interface to create a script that would harvest the item metadata and download copies of all the digitised images. There are 2,756 files in this series. On the day I harvested the metadata, 347 of those files had been digitised, comprising 12,502 images. It took a few hours, but I just ran my script and soon I had a copy of all of this in my local database.

Then came the exciting part. Using a facial detection script I found through Google and an open source computer vision library, I started experimenting with ways of extracting the photos. After a few tweaks I had something that worked pretty well, so I pointed my aging laptop at the 12,502 images and watched anxiously as the CPU temperature rose and rose. It took a few emergency cooling measures, but the laptop survived and I had a folder containing 11,170 cropped images. About a third of these weren’t actually faces, but it was easy to manually remove the false positives, leaving 7,247 photos.

These photos. These people.

With my database fully primed and loaded it was just a matter of creating a simple web interface using Django for the backend and Isotope (a jQuery plugin) at the front. Both are open source projects. All together, from idea to interface, it took a bit more than a weekend to create, and most of that was waiting for the harvesting and facial detection scripts to complete. It would be silly to say it was easy, but I would say that it wasn’t hard.

What we ended up with was a new way of seeing and understanding the records — not as the remnants of bureaucratic processes, but as windows onto the lives of people. All the faces are linked to copies of the original certificates and back to the collection database of the National Archives. So this is also a finding aid. A finding aid that brings the people to the front.

According to Margaret Hedstrom the archival interface ‘is a site where power is negotiated and exercised’. Whether in a reading room or online, finding aids or collection databases are ‘neither neutral nor transparent’, but the product of ‘conscious design decisions’. We would like to think that this interface gives some power back to the people within the records. Their photographs challenge us to do something, to think something, to feel something. We cannot escape their discomfiting gaze.

But this interface represents another subtle shift in power. We could create it without any explicit assistance or involvement by the National Archives itself. Simply by putting part of the collection online, they provided us with the opportunity to develop a resource that both extends and critiques the existing collection database. Interfaces to cultural heritage collections are no longer controlled solely by cultural heritage institutions.

It’s these two aspects of the power of interfaces that I want to focus on today.

There are a growing number of examples where the records created by repressive or discriminatory regimes have, in Eric Ketelaar’s words, ‘become instruments of empowerment and liberation, salvation and freedom’. Nazi records of assets confiscated during the Holocaust have been used to inform processes of restitution and reparation. Government records have helped members of Australia’s Stolen Generations trace family members. Descendants of inmates incarcerated by American colonial authorities in what was the world’s largest leprosy colony in the Philippines, have embraced the administrative record as an affirmation of their own heritage and survival. Records can find new meanings. Power can be reclaimed.

Technology can help. Tim Hitchcock has described how something as simple as keyword searching can turn archives on their heads. Recordkeeping systems tend to reflect the structures and power relations of the organisations that create them. The ‘hierarchical and institutional nature of most archives’, Hitchcock argues, ‘contains an ideological component which is sucked in with every dust-filled breath’. But digitisation and keyword searching free us from having to follow the well-worn paths of institutional power. We can find people and follow their lives against the flow of bureaucratic convenience. We can gain a wholly new perspective on the workings of society. ‘What changes’, Hitchcock asks, ‘when we examine the world through the collected fragments of knowledge that we can recover about a single person, reorganised as a biographical narrative, rather than as part of an archival system?’

Projects such as Unknown no longer may help us answer that question.

Unknown no longer

It’s aiming to extract the names and biographical details of slaves from the 8 million manuscript documents held by the Virginia Historical Society. The documents include court records, receipts, wills and inventories. Here is a page from the ‘Inventory of Negroes at Berry Plain Plantation, King George County, Virginia’ for 1855, listing names, occupations and valuations.

Tim Hitchcock is one of the directors of London Lives a project that similarly seeks to find the people in 240,000 manuscript pages documenting the lives of plebeian Londoners in the 17th century.

London Lives

More than three million names have already been extracted from the records of courts, workhouses, hospitals and other institutions. Work is continuing to link these names together, to merge these various shards of identity and piece together the experiences of London’s poorest inhabitants.

Remember me from the US Holocaust Memorial Museum is working with photographs taken by relief agencies in the aftermath of World War Two. The photographs are of displaced children who survived the Holocaust but were separated from families. What happened to them? The project is seeking public help to identify and trace the children.

Remember me

These are all projects about finding people. Finding the oppressed, the vulnerable, the displaced, the marginalized and the poor and giving them their place in history. This is what Kate and I hope to do with Invisible Australians, the broader project of which our faces experiment is part.

Invisible Australians

‘Invisible Australians’ aims to extract more than just photographs. We want to record and aggregate the biographical data contained within the records of the White Australia Policy — to extract the data and rebuild identities.

But we want to do more, we want to link these identities up with with other records, with the research of family and local historians, with cemetery registers and family trees, with newspaper articles and databases we don’t even know about yet. We want to find people, families and communities.

It’s ridiculously ambitious and totally unfunded. But it is possible.

The most exciting part of online technology is the power it gives to people to pursue their passions. As with the faces, we don’t need the help of the National Archives. We need the records to be digitized, but that’s happening anyway and we can afford to be patient. Most of the tools we need already exist, and are free. In the past 12 months, for example, there have been a number of open source tools released for crowd-sourced transcription of manuscript records.

People with passions, people with dreams, people who are just annoyed and impatient, don’t have to wait for cultural institutions to create exactly what they need. They can take what’s on offer and change it.

Interfaces can be modified. It is amazingly easy to write a script that will change the way a web page looks and behaves in your browser. I was frustrated by the standard interface to digitized files in the National Archives of Australia’s Recordsearch database — so I changed it.

Before and after

Not only did make it look a bit nicer, I added new functions. My script lets you print a whole file or a range of pages and display the entire contents of the file on a pretty cool 3d wall.

I’ve shared this script, and a few other Recordsearch enhancements. Anyone can install them with a click and use them.

Wragge Labs Emporium

Interfaces are sites of power and we can claim some of that power for ourselves. Online technologies not only free us from the having to brave the physical intimidation of the reading room, they free us up to engage with the records in new ways. The archivist-on-duty would probably not be pleased if I pulled out some scissors and started snipping photos out of certificates. Or if I pulled a file apart and pasted it’s contents on the wall. But online we are free to experiment.

The power of cultural heritage organisations is perhaps expressed most forcefully in their ability to control the arrangement and description of their collections. ‘Every representation, every model of description, is biased’, note Verne Harris and Wendy Duff, ‘because it reflects a particular world-view and is constructed to meet specific purposes’. Archives, libraries and museums are already starting to share this power, by allowing tagging, or seeking public assistance with description through crowd sourcing projects. But most of the these activities still happen within spaces created and curated by the institutions themselves. Our cathedrals of culture might be opening their doors and inviting the public to participate in their ceremonies, but that doesn’t make them bazaars. The architecture stills speaks of authority.

In any case, people already have a space where they can explore and enrich collections — it’s called the internet.

It would be great to see cultural institutions doing more to watch, understand and support what people are doing with collections in their own spaces — following them as they pursue their passions, rather than thinking of ways to motivate them.

A quick example… You might have heard of Zotero, it’s an open source project that lets you capture, annotate and organize your research materials.

Zotero

One cool thing about Zotero is that you can build and contribute little screen scrapers, called translators, that let Zotero extract structured data from any old collection database. You might not be surprised to learn that I’ve created a translator for Recordsearch. Another cool thing about Zotero is that you can share the stuff that you collect in public groups.

Invisible Australians Zotero group

Put those two cool things together and what do you have? Well to me they spell out user generated finding aids — parallel collection databases created by researchers simply pursuing their own passions.

Linked Open Data greatly increases opportunities for collection description to leak into the wider web. If objects and documents are identified with a unique URL, then anyone can can make and publish statements about them in machine-readable form. These statements can then be aggregated and explored. Initiatives such as the Open Annotation Collaboration will hasten the development of these shared descriptive and interpretative layers around our cultural collections.

And of course all this descriptive and interpretative work can be harvested back to enhance existing collection databases. We could start doing it now — though I will spare you today my rant about the possibilities of mining footnotes.

As well as exploring the possibilities of user-generated content, cultural institutions are starting to open up their collection data for re-use. APIs are great (though Linked Open Data is better), and New Zealand is lucky to have an organisation like DigitalNZ which just gets it. People can and will make cool things with your stuff.

But again, we don’t have to wait for everything to be delivered in a convenient, machine-readable form. If it’s on the web anybody can scrape, harvest and experiment.

You probably all know about the National Library of Australia’s newspaper digitisation project — it’s building a magnificent resource. But I wanted to do more than just find articles. I wanted to explore and analyze their content on a large scale. So I built a screen scraper to extract structured data from search results, and then used the scraper to  power a series of tools. I have a harvester that lets you download an entire results set — hundreds or thousands of articles — with metadata neatly packaged for further analysis.

Harvester

Or what about a script that graphs the occurrence of search terms over time, and allows you to ask questions like When did the Great War become the First World War?.

When did the Great War become the First World War?

In the end I got a bit carried away and built my own public API to the Trove newspaper database.

Unofficial Trove newspapers API

I think it’s important to note that the tools I developed were guided by the types of questions I wanted to ask. While we should welcome APIs and celebrate their possibilities, we should also remain critical. APIs are interfaces, they too embed power relations. Every API has an argument. What questions do they let us ask? What questions do they prevent us from asking?

Even as we move from the age of lumbering, slow-witted data silos into the rapidly-evolving realms of Linked Open Data, we have to constantly question the models we make of the world. Ontologies and vocabularies are culturally determined and historically specific. Yes, they too are interfaces, complete with their own distributions of power and authority. But we can revisit and change them. And we can relate our new models to our old models, capturing complex, long-term shifts in the way we think about the world. That’s incredibly exciting.

All of this hacking, harvesting, questioning, enriching and meaning-making makes me think about the possibilities of grassroots leadership. Online technologies enable people to take cultural institutions into unexpected realms. They can build their own interfaces, ask their own questions, determine their own needs — they can point the way instead of simply waiting to be served.

You might wonder what the National Library of Australia thinks of my various scrapers and harvesters. I can’t speak for them, but I can say that they’ve awarded me a fellowship to explore further the possibilities of text-mining in their newspaper database.

The idea of grassroots leadership brings me back to the title of this talk — ‘It’s all about the stuff’. It seems to me that we tend to model the interactions between cultural institutions and the public as transactions. The public are ‘clients’, ‘patrons’, ‘users’ or ‘visitors’. But the sorts of things I’ve been talking about today give us a chance to put the collections themselves squarely at the centre of our thoughts and actions. Instead of concentrating on the relationship between the institution and the public, we can can focus on the relationship we both have with the collections.

It’s all about the stuff.

It’s all about the respect and responsibility we both have for our collections.

It’s all about the respect and responsibility we both have for people like this.

 

 

Tagged with: , , , , , ,
Posted in archives, conference presentations, digital humanities

Extracting editorials #1

In their chapter in Writing History in the Digital Age, Trevor Owens and Fred Gibbs encourage historians to write about the ways they work with data — to document their methods, their working assumptions, their dead ends and their discoveries. It’s an important argument and one that makes me wonder again about forms of publication that might integrate narrative, methods and sources.

In the meantime though we have blogs. My problem is that I’m easily bored so by the time I get to the end of a project or experiment I’m already thinking about the next one. Going back and trying to write things up seems a bit of a chore (which is why I’m always way behind in my blog writing). Also leaving the writing to the end means that I tend to take shortcuts — leaving out some of the ‘boring’ procedural stuff or the ‘stupid’ ideas that just didn’t work.

But Trevor and Fred’s chapter has made me think I should be a bit more diligent, so as I start a new series of text-mining experiments I’ve decided to write things up as I’m doing them. So be warned, this could get messy…

So what do I want to do? You might not be surprised to learn that it’s another Trove newspaper database experiment. I want to see if I can harvest newspaper editorials over a certain period and then analyse these to build up a picture of what issues, events or ideas were perceived as important. As I’m currently looking at ways of harvesting digital sources relating to 1913 for an exhibition being developed by the National Museum of Australia, I’m going to start by focusing on 1913.

But editorials are opinion pieces, wouldn’t it be better to harvest ‘news’ articles?

First of all, I’m thinking that editorials will be fairly easy to identify and extract — there’s no real way in Trove to separate out current news from other sorts of articles. Secondly, I’m assuming that the issues that make it into editorials have some importance attached to them. Attached by whom, you may well ask — whose voice is being represented in the editorial? This is an important question and I’m thinking that it could be explored in interesting ways by harvesting editorials from a range of papers and regions. Thirdly, finding the editorials might actually help me find the major news articles, simply because in this period the main news stories were often on the page after the editorials.

So how do I find them? Looking at the Sydney Morning Herald for 1913, you can see that the editorials follow a regular pattern:

  • the first editorial is always headed with the name of the paper and the date, followed by the title
  • subsequent editorials that follow have a title but no subtitle (most other types of articles have a subtitle)
  • editorials are published on an even-numbered page, usually about half way through the newspaper

To check this I conducted a search for articles including ‘The Sydney Morning Herald’ in their title. The search returns 335 results. Of course we’d expect there to be 312 (6 x 52), but it looks like there’s quite a few false positives and some days missing altogether (presumably due to OCR errors). You can see there’s a fair bit of consistency in the pages that editorials appear on, but it doesn’t quite seem consistent enough to rely on. So I’ve decided that as a first step I’ll harvest all the articles from this query. I’ll then do some manual cleaning to remove the articles that aren’t editorials and try and identify and retrieve the missing days.

Remember, this won’t give me all the editorials, only the first editorial from each day. To get all the editorials, I’ll have to write a new script that will take this first result set, retrieve all the articles from the editorial page and then try to work out which of the articles are editorials — they should be the ones that come after the first editorial and have no subtitle. Or that’s the theory.

I’ve harvested the query. You can view the spreadsheet on Google Docs if you feel so moved.

[After I wrote the sentence above I checked the CSV file properly and realised I'd stuffed up. There's a bit of a bug in my harvester that means if the query string you use includes a start value, the harvester wil retrieve the same page of results over and over again... I really need to fix that. :-) I'm now running it again. You wanted warts and all, right?]

[After I wrote the paragraph above I checked my new harvest and realised I'd stuffed up again. There were only half as many results as there should have been! So I poked around and realised some recent changes I'd made to the harvest script meant I was only getting odd numbered results (I was incrementing the row value twice). A lesson in what happens when you do this stuff late at night... Trying again. ]

I’m not sure when I’ll have time to do the cleaning. But hey folks this is what research is like for people like me who have to try and fit it in around the edges of their lives. You can expect posts to come in sudden bursts and then dry up altogether for long periods as other priorities intrude.

 

Tagged with: , , ,
Posted in digital humanities

An infrastructure wishlist

I have problems with the idea of infrastructure, particularly that of the e-research variety. It seems like we always end up talking about huge amounts of money and multi-institutional partnerships. It just doesn’t seem like a great model for innovation. As I’ve previously argued, I’d like to see something more like the funding schemes offered by the NEH Office for Digital Humanities. Encourage people with ideas, don’t just reward the good networkers. Build tools and apis, not portals and platforms.

Of course I’d still like to see the digital humanities well represented in the list of Virtual Laboratories and eResearch Tools currently under consideration by NeCTAR. It’s time the digital research needs of the humanities were properly recognised. There are lots of possibilities, most of which we can’t yet envisage, but as I was asked what I would like to see as part of a Virtual Laboratory I had a go at setting down a few brief ideas. For what it’s worth, here’s my e-research infrastructure wishlist…

Grappling with abundance

Traditional historical research is often based on a presumed scarcity of resources — the skill is in tracking down the sources. But large digital collections, like the Trove newspapers database, change this — you now have to make sense of the sheer volume of material. Digital history, through techniques such as text-mining and visualisation, offer a way of using these new riches effectively. We need to ensure that investments in digitisation are accompanied by evolutions in scholarly practice.

Understanding what’s not online

At the same time, it must be recognised that large quantities of our cultural heritage are not available in digital form. For example, only about 10% of the holdings of the National Archives of Australia are described in their collection database, and only a small proportion of these are digitised. Easy online access could foster a certain circularity in historical research where only ‘known’ resources are consulted. We need to develop tools and visualisations that reveal the valleys as well as the mountaintops — identifying the holes in our research fabric.

Critical engagement

More generally, we need to foster critical engagement with the tools and assumptions of digital research. Federated searching sounds great, but as scholars we need to expose the assumptions implicit in any such tool. What is being federated, from where, how is relevance being determined etc? Humanities e-research infrastructure should have built-in levels of reflexivity that enable scholars to understand the limits and assumptions of their digital research. Every algorithm contains an argument.

Documenting change

The resources we build are arguments with are subject to change. The Trove newspapers database, for example, is constantly adding new titles and articles, while users are improving the text transcriptions. Any analysis based on the holdings of this database needs to explicitly recognise this. At the very least the tools we have need to be able to generate time-stamped citations. It would be even better if we could capture a snapshot of the data to accompany our analyses. Perhaps there are possibilities for using something like the Memento project to ensure that the temporal context of humanities research is adequately documented.

Show your working out

Scholarly publication in history, and the humanities generally, tends to present a finished product. But as we delve further into digital research the research processes themselves will be equally important both for fostering critical engagement with tools and methods and for enabling others to reproduce or extend the research. We need easy ways for researchers to expose their working out (subject to whatever access controls they think appropriate). It should be possible to save a series of steps – search, analysis, visualisation etc as modules for sharing and re-use.

Follow your nose

Search needs to be complemented by rich, exploratory environments that encourage browsing, enable you to follow relationships, and foster serendipitous discovery. The problem with many collections is knowing enough about what’s in them to frame a useful search. Browsing, though a variety of interfaces — people, maps, events, record types, physical proximity — overcomes this problem. As more cultural institutions make use of Linked Open Data and shared identifiers — such as People Australia, Geonames or the Powerhouse Object Thesaurus — the possibilities for navigating this rich contextual space will increase.

Citation

We need to develop better models for embedding rich citations within scholarly research — citations that describe not only the resource in structured, machine-readable forms, but also relevant relationships. This will link research directly to resources, making scholarly outputs a means of resource discovery, and enabling resource databases to re-use the scholarly research to enhance their own descriptions and finding aids.

Constructing narratives

Moving beyond simple citation, we need better ways of exposing the structures of people, events, places and things that are referenced in our narratives. Linked Open Data provides a model, but we need tools to make it simple and examples to make it obvious.

Posted in digital humanities

Every story has a beginning

Entering the web of data

[view the presentation...] [view the triples...]

Keynote delivered at the annual conference of the Australia and New Zealand Society of Indexers, 14 September 2011.


This is me.

Today, Wednesday, 14 September 2011, I’m honoured to be able to join you here in the luxurious surrounds of the Brighton Savoy Hotel for the ‘Indexing See Change‘ conference. This is an event, a moment in history; we can pinpoint ourselves, this gathering, both in time and in space.

If we do that, if we move outside the moment and position ourselves on a timeline or a map, interesting things start to happen. Connections emerge.

Here we are at number 150, The Esplanade, in Brighton. A bit over a kilometre away is the stately villa, Kamesburgh. For many years Kamesburgh was also known as the Anzac Hostel — a refuge for permanently-incapacitated World War One veterans.

The Anzac Hostel opened on 5 July 1919. Here it is draped in its patriotic finery, from the collections of the Australian War Memorial. According to the caption, the Anzac Hostel was ‘a home, not an institute’.

Also amongst the War Memorial’s holdings is a wheeled bed that was used at the hostel. This particular bed was apparently occupied by one man, Albert Ward, for forty-three years.

Death notice for Alexander Kelley. Argus, 29 January 1944.

It was probably in a bed just like this that Alexander Dewar Kelley passed away on 27 January 1944. Alexander Kelley was cremated, and his remains interred amongst the roses at what is now called the Springvale Botanical Cemetery. Not far from my own grandparents.

Alexander Kelley spent close to half his life in the Anzac Hostel. Like many young men, he bravely answered his nation’s call to arms, but returned from war much changed. We can follow Alex’s war through his service record, easily-accessible through the website ‘Mapping Our Anzacs‘.

Alex was a coach painter who enlisted in the AIF in January 1916. Within a year he was in France. In May 1917 he suffered a gunshot wound to the head, but was able to rejoin his unit in August. Less than a month later though, he was wounded again, this time more severely. For Alex the war was over, and he was shipped back to Australia in May 1918.

‘Mapping Our Anzacs’ includes a scrapbook feature through which visitors to the site can attach notes or photographs to a service record. Amongst the the many thousands of postings is a fragment from a diary, found tucked inside the bible of Alexander Kelley’s mother. The diary entry reads simply: ‘Alex arrived from Front. Wet day. Saw him at “Caulfield”.’

Alex had survived and had returned to his family. This was a day to remember. But there was sadness too, for Alex was not the same young man who had left for the battlefields of Europe. In the diary fragment, ‘Caulfield’ is enclosed in inverted commas, indicating perhaps that the reunion took place, not in the suburb, but in the Caulfield rehabilitation hospital. Alexander Kelley was wounded in the face, hands and legs. He was left blind in both eyes and his right leg was amputated. He would live the remainder of his life a little over a kilometre away from here at the Anzac Hostel.

This is just one story. There are over 375,000 World War One service records held by the National Archives of Australia. How can we hope to understand a number like that? How can we hope to imagine the war’s impact on families, on communities?

‘Mapping Our Anzacs’ uses familiar Google maps to display the places of birth and enlistment recorded in many of those service records. But technical limitations make it impossible to display all the places at once. You can, however, take the same data and open it in Google Earth. If you then zoom in on Victoria, you see something like this.

Mapping Our Anzacs data viewed in Google Earth.

Each marker represents a place where a service person was born or enlisted. It’s impossible to read, of course, but that’s the point. There is so little blank space. As you zoom further, more markers appear, more place names resolve. It’s simple, but it’s powerful. They came from everywhere. From the smallest village to the biggest city; nowhere was untouched.

The ‘Mapping Our Anzacs’ scrapbook offers another perspective. It’s possible to extract the images posted to the scrapbook and present them on a 3D wall. Amidst an assortment of memorabilia, there are faces. Not places, or records — this is a wall of people.

Mapping Our Anzacs Scrapbook photos viewed through CoolIris

It’s worth noting too that like the markers on the maps, these faces link back to the actual service records. So they’re not just a new way of seeing the collection, they’re a new way of exploring it.

But the records don’t stand in isolation, they themselves have a context. A couple of years ago, Mitchell Whitelaw from the University of Canberra, undertook a project called ‘The Visible Archive‘ to investigate ways of visualising the holdings of the National Archives of Australia. Have you ever wondered what 360km worth of records looks like?

The collections of the NAA visualised by Mitchell's Series Browser.

This represents the holdings of the National Archives. Files within the archives are organised into series, and each square in this image represents a single series — there are about 60,000 of them. Naturally the size of the square gives an indication of the size of the series itself. It’s a fascinating and strangely beautiful picture.

It’s easy enough to pick out the World War One service records — Series B2455. In the interactive version of Mitchell’s series browser you can click on a box and display links between series, as well as other series created by the same government agency. Again, it’s not just a way of seeing the collection, but a means of exploring and interpreting it. As Mitchell says:

Visualisation enables us to literally show everything, to display large volumes of data in a way that reveals patterns and communicates context, but also provides access to the fine grain of individual elements.

But we can also employ such techniques to ask new kinds of questions. Can you imagine how Alexander Kelley and the other inhabitants of the Anzac Hostel must have felt in 1939? They had lost so much in the Great War, the ‘war to end all wars’, and yet within their own lifetime it was all happening again. More young men were answering the call, more lives were going to be destroyed.

There must have been a dreadful, disheartening moment when Australians realised that the Great War was not an end, but a beginning — the first in a series of devastating global conflicts. At some point the ‘Great War’ became the ‘First World War’, but when?

When did the 'Great War' become the 'First World War'?


This is one possible answer. This graph draws its data from the 50 million or so digitised newspaper articles in Trove, the National Library of Australia’s discovery service. It shows the proportion of newspaper articles that included the phrase ‘the great war’ compared to the proportion containing ‘the first world war’ (and variations thereof). The lines cross late in 1941. With German victories in Europe and Africa, the opening of the Eastern Front and the Japanese attack on Pearl Harbour, 1941 makes sense.

What is perhaps more intriguing is the dramatic peak in the occurrence of ‘the great war’ in 1939. It’s no surprise that the looming threat of a new conflict would provoke comment and comparisons, but it does make you wonder about the context of those discussions and how they might have changed as the reality of war edged closer.

To start exploring this I’ve harvested the content of the 6,600 articles from 1939 that included the phrase ‘the great war’. Using an online text analysis service called VoyeurTools I can quickly generate a picture of their contents.

This simple visualisation shows us the relative frequencies of words within the articles. It doesn’t reveal any great mysteries, but it does suggest some possibilities for further prodding. The prevalence of ‘time’ and ‘new’, for example — might these help us understand the shift in perspective from one war to the next? We can follow this up by browsing the different contexts in which the words were used.

But what actually is it that we’re actually searching? We know that Trove includes newspapers from 1803 to 1954, but if we’re really going to analyse shifting words and ideas it’s important to have a clear picture of the sources of those words.

Something like this perhaps. This graph shows the holdings of the Trove newspaper database on 4 August 2011, organised by state. You can see, for example, that if you’re searching on a topic between the 1920s and 1940s you’re probably likely to get more results from Queensland than anywhere else.

So starting from our location here, today, we can make connections across time and space. We can pull back and look at the big picture, or dive in and examine the fabric of a single life. Through the web we can build and explore a rich and complex contextual network.


It’s an exciting time to be a cultural data hacker. We now have a growing range of tools and technologies available for extracting interesting data from a wide variety of sources, both structured and unstructured.

The ‘Visible Archive’ project started with well-structured data, courtesy of Peter Scott, the developer of the Series System — the descriptive framework used by many Australian archives. But we’re rarely so lucky.

Even when the data starts off in nicely-organised fields in a database there’s no guarantee that that’s how it’s going to be delivered to our web browser. In order to extract the data from my Trove graphs, for example, I had to write a little program called a ‘screen scraper‘ to identify and save the important metadata elements from the raw web page itself.

Where there are no subject keywords we can infer them using techniques such as topic modelling. Where there are no access points we can identify people, organisations, places and events using special tools developed for named entity extraction. Where there are no common identifiers across datasets we can employ record linkage technologies to find possible connections.

We can count words, we can identify parts of speech, we can formulate a measure of the similarity of any two pieces of text. Once we have some useful data we can manipulate and enrich it. Place names can be geolocated — you simply send your place name off to a web service and get back its latitude and longitude.

Increasingly these sorts of tools are becoming accessible to anyone. For historians they offer a means of wrestling with rapidly-growing bulk of source material that is becoming available in digital form. How do you make use of 5 million digitised books, 50 million newspaper articles or the complete archive of every public message ever sent on Twitter?

The digital historian Dan Cohen has noted:

These computational methods which allow us to find patterns, determine relationships, categorize documents, and extract information from massive corpuses, will form the basis for new tools for research in the humanities and other disciplines in the coming decade.

Dan is involved in a number of interesting projects investigating the possibilities of these techniques — often grouped together under the heading ‘text mining’. One of these projects, ‘With Criminal Intent‘, is looking to see what patterns can be drawn out of the digitised proceedings of criminal trials held at the Old Bailey from 1645 to 1913. That’s 197,745 trials, in case you were wondering.

Here’s one of their visualisations showing how the length of trials varies over time. Much to the surprise of the research team, this graph suggests a dramatic shift in legal practice around 1825 — defendants started pleading guilty!

A visualisation by the With Criminal Intent project showing changing trial lengths.

Rather than falter under the growing weight of digital sources, these technologies can actually thrive. The more raw material available, the more chance there is to observe and track new patterns. As digitisation continues apace will we ever reach the point when history can simply be read from a graph?

There are some researchers at Harvard who seem to think that’s where we’re heading. Borrowing liberally from the store of scientific metaphors they have staked out the new field of ‘culturomics‘. By mining massive digital resources, like Google’s scanned books, they hope to map the ‘cultural genome’ that would enable us to follow the evolution of language and culture.

But there’s something quite barren in this ambition. I prefer the vision of digital humanist Stephen Ramsay, who commented in regard to the ‘With Criminal Intent’ project:

The Old Bailey, like the Naked City, has eight million stories. Accessing those stories involves understanding trial length, numbers of instances of poisoning, and rates of bigamy. But being stories, they find their more salient expression in the weightier motifs of the human condition: justice, revenge, dishonor, loss, trial. This is what the humanities are about. This is the only reason for an historian to fire up Mathematica or for a student trained in French literature to get into Java.

Ultimately it’s the stories that nourish, anger, inspire and depress us. The closely-packed map of places recorded in World War I service records is so powerful because we know that under each marker are men, women, families, communities — each with their own story. These new technologies offer new perspectives, they raise new questions, and they challenge us with new contexts to explore and understand. But there is still space for stories and perhaps we can use them to give our stories new life and depth.


This is another World War One service record. It belongs to Charlie Allen. Charlie enlisted three times in the AIF and was discharged on medical grounds each time. It seems he had a problem with his ankle.

Charlie’s service record notes a tattoo, proclaiming his love for ‘Maud Gordon’. He married Maud in Sydney in 1917 and had two daughters soon after.

Charlie survived the war without further injury, but was not so lucky in peace. On 11 March 1938, Charlie was crushed to death between two railway cars. The accident happened at the Bunnerong Power Station, only a short distance from his home in Matraville. He was buried nearby in the Botany Cemetery.

We also know quite a bit about Charlie’s early life. Why? Because Charlie’s father was Chinese and he was therefore categorised as a ‘half-caste’, as someone who was not white, and therefore fell under the restrictions imposed by the White Australia Policy.

Charlie was born in Sydney in 1896. His mother was Frances Allen (sometime sweet shop owner and brothel keeper), his father Charlie Gum (a buyer for Wing On company). Charlie was raised by his mother, but in 1909, at the age of 13, he was taken to China by his father.

NAA: ST84/1, 1909/22/41-50

This certificate granted Charlie an exemption to the Dictation Test. Without it, he may not have been allowed back into the country.

Every time one of many thousands of non-Europeans resident in Australia sought to travel overseas and return home again they needed one of these certificates.

Charlie’s father returned to Sydney, leaving him in China. He lived with relatives in the town of Shekki (inland from Hong Kong). Charlie was naturally homesick, but had no means of getting back to Australia. He wrote to his mother in 1910:

Do try and bring me home every minute I think of you and long for a piece of bread and butter this tucker is not doing me well.

His mother wrote to the Prime Minister Billy Hughes in an attempt to enlist government help but to no avail. Charlie finally returned to Australia in 1915.

Despite this experience, Charlie visited China again in 1922 for 7 months. Once again carrying papers to grant him re-entry to the country of his birth.

These fragments of Charlie’s life have been assembled by my partner, Kate Bagnall, a historian of Chinese-Australia. They are remarkable, and yet not so, because there are many thousands of stories like Charlie’s contained within the voluminous records generated by the administration of the White Australia Policy.

We’re all of course familiar with the general outlines of the White Australia Policy, and the way it underpinned conceptions of Australia as a nation in the first half of the 20th century.

But what we sometimes forget is that it was also a massive bureaucratic exercise.

Forms and certificates were printed, issued, used and filed. Regulations were modified, guidelines were distributed and administering officers were managed and advised. Individual cases were reviewed, policy was changed and new forms and certificates were printed, issued, used and filed…

Much of this system is now preserved in the National Archives.

You can get a idea of the range of material available from a case study Kate has prepared focusing on the efforts of Poon Gooey, a successful businessman in Horsham, to keep his wife and family in Australia.

If we look again at Charlie’s certificate from 1909 we can see that it contains a lot of interesting structured data:

  • name
  • place of birth
  • age
  • height
  • destination
  • date of departure
  • name of ship

We estimate that there are probably about 50,000 of these forms remaining in the Archives, and then there’s case files and a variety of other government documents.

Wouldn’t it be great if we could extract this structured data. If we could piece together the slivers of identity that remain within the Archives and give people back their lives.

This is the dream of Invisible Australians, a project Kate and I are trying to turn into a reality. Our aim is to build systems that will enable this data to be extracted, aggregated, shared and connected — whether to a family tree, a cemetery record, or another document in another archive.

Imagine being able to navigate the network of lives, families and relationships. To follow their journeys, to share their tragedies, to celebrate their small victories against a repressive system.

Imagine being able to watch them age.


We tend to assume that new technologies require us to change, to adapt. But sometimes they can take advantage of our strengths. Mitchell Whitelaw is interested in finding out what happens when you take large cultural datasets and try to ‘show everything’. Such an approach, he suggests, takes advantage of the raw processing power of computers, while giving us space to do what we’re good at — finding patterns, making connections, crafting meanings.

The History Wall tries to create a similar sort of space. The History Wall brings together material from a range of different sources — newspaper articles from Trove, biographies from the Australian Dictionary of Biography, records from a database of NSW convicts, population statistics, collection items from the National Museum of Australia — you can pretty much plug anything in as long as it has a date attached to it.

Irish History Wall

For a particular year, the Wall retrieves a random sample from the available sources, jumbles everything up and then throws it onto the screen. As a result, no two views of the Wall are ever quite the same. This is not a traditional exhibition. There is no curator controlling the content or designing the structure. It’s ephemeral, it’s serendipitous — instead of relying on an authorial voice to smooth over the gaps and transitions, it leaves open the cracks and allows new contexts to seep in and around each item.

As the pioneering digital historian Edward Ayers noted:

even isolated and inert pieces of evidence — a list, a letter, a map, a picture — can assume new and unimagined meanings when placed in juxtaposition with other fragments.

This is not an absence of narrative, but an opportunity for narration. Edward Ayers suggests that we’re actually quite comfortable filling in blanks and untwisting timelines:

Humans, presented with pieces of information about people, put things into the form of a story. They need not be simple stories, for we know how to deal with unexplained lapses of time, flashbacks, and overlapping narratives. We know how to imagine, infer, things happening at the same time in different places. Film and television train all of us at early ages to weave strands of narrative out of intentional (if carefully constructed) confusion and to take pleasure in that weaving.

And so I can show you a death notice, or a certificate and you will take those fragments, those isolated data points and you will construct a story — you will see the person behind them, you will imagine their life. It’s what we do. We’re good at it.

Computers on the other hand will just see data.

In her ode in praise of humanities data, digital humanist Amanda French wonders whether we always need to crunch our data into abstract, pliable forms:

What I wonder is whether instead we can begin with the data, or with a datum, and simply watch for what it may tell us, even if what it tells us is simply a story.

Yes we can. And we should teach computers how to do it as well. Not because we want them to take over. Not because they can necessarily do it faster or better. But because they can help us share, preserve and connect those stories.

Let’s think again about the array of documents that Kate has assembled to piece together the story of Charles Allen. How can you share this sort of material? Typically you’d ‘write it up’. You’d capture the story behind the data and commit it to words. The documents would then become evidence — points of connection between your text and the historical record.

So in order to share the meanings of these documents we remove them from the context of the person’s life and marshal them as allies to proclaim the authenticity of our rendering. Wouldn’t it be better if we could tell the story, but maintain within our texts the direct connections between sources and subject?

What we need is a data framework that sits beneath the text, identifying people, dates and places, and defining relationships between them and our documentary sources. A framework that computers could understand and interpret, so that if they saw something they knew was a placename they could head off and look for other people associated with that place. Instead of just presenting our research we’d be creating a whole series of points of connection, discovery and aggregation.

Sounds a bit far-fetched? Well it’s not. We have it already — it’s called the Semantic Web.

The Semantic Web exposes the structures that are implicit in our web pages and our texts in ways that computers can understand. The Linked Data movement takes the basic ideas of the Semantic Web and turns them into a collaborative activity. You share vocabularies, so that other people (and computers) know when you’re talking about the same sorts of things. You share identifiers, so that other people (and computers) know that you’re talking about a specific person, place, object or whatever.

Linked Data is Storytelling 101 for computers. It doesn’t have the full richness, complexity and nuance that we invest in our narratives, but it does at least help computers to fit all the bits together in meaningful ways. And if we talk nice to them, then they can apply their newly-acquired interpretative skills to the things that they’re already good at — like searching, aggregating, or generating the sorts of big pictures that enable us to explore the contexts of our stories.

This is why we’ve always imagined Invisible Australians to be something more than an online database. We want to provide points of connection that other people can build into their own stories. But to do that we have to pay attention to things like vocabulary management and authority control, we have to construct web addresses that are not going to break every time we upgrade our software. We have to think about the sorts of things we’re talking about — not just people, but government agencies, legislation, certificates, and correspondence. How do we describe these entities and what sorts of relationships do they have?

And of course we need to expose all these structures so that we can say, these things are people, these are events, these are places and these are documents.

Or perhaps, to introduce Alexander Kelley.

Or remember Charles Allen.


You might be wondering why we don’t just leave it all to the computers themselves. Didn’t I just talk about all the exciting new tools and techniques that enable us to analyse the structures of texts? Perhaps we should just wait for the Culturomics guys to solve all the problems.

But who defines the problems?

Our postmodern sensibilities encourage a suspicion of neutrality. Labels like ‘the new museology’ or Archives 2.0 reflect an awareness that the way we describe and arrange our collections is itself culturally-determined. It’s not just a matter of what our descriptive systems show, but what they hide.

Tim Hitchcock, another member of the ‘With Criminal Intent’ team, has described how online technologies can change the way we access archives. Instead of being forced to navigate the hierarchical structures that archives impose on records, which in turn tend to reflect the workings of the institutions that created the records, we can directly find the people whose lives were regulated, influenced, shaped or controlled by the policies of those institutions.

Instead of merely hearing ‘the institutional voice… in all its stentorian splendour’, he says, we can listen in to ‘the quieter tones uttered by the individual’.

This reminds us that search boxes, along with other digital tools, themselves embody arguments. There are assumptions built into their code about what is relevant, what is significant, what is necessary.

We can build our own tools of course, and we can critique other people’s algorithms. But what if we just want to collect and share stories?

Linked Data gives us a way to present an alternative to Google’s version of the world. We can argue back against the search engines, defining our own criteria for relevance, and building our own discovery networks.

Changing the way we access resources changes the sorts of stories we can tell. Tim Hitchcock asks:

What happens when institutions and archives are ‘decentred’ in favour of the individual? What changes when we examine the world through the collected fragments of knowledge that we can recover about a single person, reorganised as a biographical narrative, rather than as part of an archival system?

Perhaps the invisible become visible.

Tagged with: , , , , ,
Posted in shoebox, web

the real face of white australia

In many of the presentations I’ve given in recent times I’ve managed to include a question raised by Tim Hitchcock in his chapter in The Virtual Representation of the Past. Tim asks:

What changes when we examine the world through the collected fragments of knowledge that we can recover about a single person, reorganised as a biographical narrative, rather than as part of an archival system?

The idea of turning archival systems on their head to expose the people rather than the bureaucracy is what motivates Kate Bagnall and I in our attempts to make the Invisible Australians project into a reality.

Invisible Australians aims to liberate the lives of those who suffered under the restrictions of the White Australia Policy from the rich archival holdings of the National Archives of Australia and elsewhere.

We always knew that the portrait photographs, included on a range of government documents, would provide a compelling perspective on these lives, but we weren’t quite sure how we were going to extract them. Up until last weekend, I’d assumed that we’d develop a crowdsourcing tool that contributors would use to mark-up the photos.

Now I’m not so sure.

In the space of a couple of days I’ve extracted over 7,000 photographs and built an application to browse them — here is the real face of White Australia

How did I do it? Paul Hagon, at the National Library of Australia, gave a presentation last year in which he explored the possibilities of facial detection in developing access to photographic collections. The idea lodged in my brain somewhere and a few days ago I started to poke around looking to see how practical it might be for Invisible Australians.

It didn’t take long to find a python script that used the OpenCV library to detect faces in photographs. I tried the script on a few of the NAA documents and was impressed — there were a few false positives, but the faces were being found!

So then the excitement kicked in. I modified the script so that instead of just finding the coordinates of faces it would enlarge the selected area by 50px on each side and then crop the image. This did a great job of extracting the portraits. I tweaked a few of the settings as well to try and reduce the number of false positives. Eventually, I developed a two-pass system that repeated the detection process after the image had been cropped and it’s contrast adjusted. This seemed to weed out a few more errors. You can find the code on GitHub.

Once the script was working I had to assemble the documents. I already had a basic harvester that would retrieve both the file metadata and digitised images for any series in the NAA database. Acting on Kate’s advice, I pointed it at series ST84/1 and downloaded 12,502 page images.

All I then had to do was loop the facial detection script over the images. Simple! The only problem was that my 3-year-old laptop wasn’t quite up to the task. As it’s CPU temperature rose and rose, I was forced to employ a special high-tech cooling system.

Keeping my laptop alive...

But after running for several hours, my faithful old laptop finally worked it’s way through all the documents. The result was a directory full of 11,170 cropped images.

The results

There were still quite a lot of false positives and so I simply worked my way through the files, manually deleting the errors. I ended up with 7,247 photos of people. That’s a strike rate of nearly 65% which seems pretty good. The classifier, which does the actual facial detection, was probably trained on conventional photographs rather than on the mixed-format documents I was feeding it.

Then it was just a matter of building a web app to display the portraits. I used Django for the backend work of managing the metadata and delivering the content, while the interface was built using a combination or Isotope, Infinite Scroll and FancyBox.

It’s important to note that the portraits provide a way of exploring the records themselves. If you click on a face you see a copy of the document from which the photo was extracted. A link is provided to examine the full context of the image in RecordSearch. This is not just an exhibition, it’s a finding aid.

What next? There are many more of these documents to be harvested and processed (and many more still yet to be digitised). I will be adding more series as I can (though I might have to wait until I can afford a new computer!). I’d also like to explore the possibilities of facial or object detection a bit more. Could I train my own classifier? Could I detect handprints, or even classify the type of form?

In the meantime, I think our experimental browser helps us to understand why the Invisible Australians project is so important — you look at their faces and you simply want to know more. Who are they? What were their lives like?

UPDATE: For more on the photos and the issues they raise, see Kate Bagnall’s posts over at the Tiger’s Mouth.

Tagged with: , ,
Posted in archives, experiments

When did the ‘Great War’ become the ‘First World War’?

Townsville Daily Bulletin, 9 December 1939

I’m interested in time — in the way we imagine, manipulate, experience and describe time, particularly in the service of ideas such as ‘progress’.

This was one of the themes of Atomic Wonderland, but beyond constructing a few case studies it’s not all that easy to study. Or at least it wasn’t. Now projects such as Victorian Books are showing how we can explore the changing weights of ideas across times and cultures by analysing the contents of large textual collections.

Returning visitors will be probably be aware of my own experiments mining the contents of the National Library of Australia’s digitised newspapers database, available through Trove. So far I’ve focused on the development of generic tools and techniques, but I thought it would be interesting to apply these to my study of ‘progress’. Happily the NLA agreed and have awarded me a Harold White Fellowship for 2012 to do just that. Yippee!

I’ll be taking up the fellowship in February, but in preparation I’ve started to develop a few little sketches that prod at our fondness for periodisation. Labels such as ‘the Roaring Twenties’, ‘the Great Depression’ or even ‘the First World War’ are so familiar that we sometimes forget that they themselves have a history.

To begin with I decided to examine the question of when ‘the Great War’ became ‘the First World War’. At some point we realised that the Great War was not the final act in a centuries-long drama of European jealousy and jostling, but the first in a series of global conflicts. Can newspapers tell us when?

I already had a script that would generate a basic time series from a Trove query string. It simply takes the query, fires off a separate search for each year and grabs the number of matching articles. If the number of matches is more than zero, it also retrieves the total number of articles for that year and calculates the proportion matching the query. The results are saved in a json file which can be easily visualised using something like HighCharts. The original script needed a few tweaks to streamline the process, but I’ll describe these in detail in my next post.

For this experiment I constructed two queries. The first simply searched for the phrase ‘the great war‘ between 1900 and 1954. The second was a bit more complicated — it searched for any of the phrases ‘first world war’, ‘world war one’, ‘world war 1′ or ‘world war i’ across the same period. I fed the queries to my script and after a bit of ker-chugging, whirring and clunking I ended up with a graph.

Click to view the full interactive graph.

The result is not really surprising. As you can see on the full graph, the two lines cross late in 1941. With German victories across Europe and North Africa, the opening of the Eastern Front and, finally, the Japanese attack on Pearl Harbour, 1941 seems to make sense. But it’s interesting to see this reflected so clearly in such a rough and ready analysis.

What is perhaps more intriguing is the huge spike in 1939. Of course it makes sense that people would be referring back to the Great War as the prospect of a new conflict loomed, but it does make you wonder about the context of these discussions and how they might have developed as war edged closer.

Notable too are the earlier blips in the First World War count — the first centred on 1916 and the second on 1935. The peak in 1916 is actually due to the tags and comments added by Trove users. The standard ‘search everything’ option in Trove includes these as well as the text of the articles themselves. By using other search options you can choose to exclude the tags that match your query, but that seems rather messy. It would be nicer if Trove gave you the option of ignoring these matches from the start.

The West Australian, 24 May 1935

The second blip is a bit more interesting. By clicking on the graph and exploring the results from Trove, you can see that it’s due to the screening of a documentary film called ‘The First World War‘. The film used archival footage drawn from a number of nations and was based on Laurence Stalling’s book The First World War: A Photographic History. As one newspaper article noted: ‘this picture presents war, stripped of its gaudy trappings, and fearful in its grim reality’.

By way of comparison I tried a similar query using the Google Books Ngram viewer. The crossover point seems a little later, but of course books take longer to publish than newspapers. There is, however, no peak in 1939 for ‘the Great War’ — at least not if you use the combined ‘English’ corpus. If you examine the British-English and American-English corpora separately it’s a rather different story. Querying the British-English corpus produces something much closer to our Trove graph, complete with a spike around 1939. Again, this is only as we’d expect given the lesser significance of the First World War in American history.

This is, of course, only a sketch — something to prompt new questions or suggest avenues for attack. It’s made me want to find out a bit more about the nature of discussions in 1939, so I’ve fired up my Trove Newspaper Harvester and downloaded the text of all 6,582 articles from 1939 that include the phrase ‘the Great War’. More about that soon…

Tagged with: ,
Posted in digital humanities, experiments

Some exhibition magic with Zotero and Omeka

Letter from CKR Kilby (‘Parkwood’, Hall FCT)

One of my most exciting archival discoveries was a cache of letters written by farmers from across NSW in 1938. Seeking to marshal support for the long-range weather forecaster Inigo Jones, The Land newspaper asked it's readers to send their opinions of the 'weather prophet'. And they did. One hundred and two letters were received, and duly forwarded to the Minister for the Interior. These 102 letters now reside in the National Archives of Australia.

I tell the full story of the newspaper’s campaign in Inigo Jones: The Weather Prophet. but I’ve always wanted to do something more with the letters. Inspiration finally arrived this year when a conference on rural media was organised to mark the The Land‘s centenary. I decided to create a little exhibition using the letters and, in doing so, provide myself with a platform for further research.

Step one: assembling the letters with Zotero

I’d already asked for the file containing the letters to be digitised, so images of all of the letters were available through RecordSearch. Zotero ships with a RecordSearch translator that I wrote some years ago, so 102 clicks later, I’d captured both the metadata and all of the images into my own database.

The Zotero translator saves individual pages with a generic title, so I then had to do some manual editing. I also had partial transcripts of the letters which I’d created during my original research. I simply cut and pasted the transcripts into a note field in Zotero and tidied them up. Pretty soon my collection was complete.

Step two: creating an exhibition with Omeka

The Weather Prophets

I actually started building the exhibition during an Omeka Bootcamp session I ran at THATCamp Melbourne. Starting with nothing but a LAMP server and my Zotero library, I had the basics of site up within the hour.

It was simply a matter of installing Omeka, adding the Zotero import plugin, creating an API key in my Zotero profile, and then pointing the Zotero import plugin to my collection. (The API key is necessary if you want to download files, such as my images.)  With a press of a button the internets started chugging away and pretty soon all the letters were in Omeka. Hey presto — instant exhibition!

With the framework built I could start to play a bit. First I installed Omeka’s geolocation plugin and started mapping where the letters had come from. Google’s geocoder did a pretty good job of finding many of the small country towns, but I also made use of Geoscience Australia’s gazetteer. In the end, I managed to geolocate all but one letter. Hey presto — a map!

I thought the pre-installed ‘Summer’ theme suited my exhibition, but I did make a few minor tweaks here and there. This included adding some RDFa into the list of references (which were also imported from Zotero).

Step three: going deeper with Voyeur Tools

Word cloud of the letters created by Voyeur Tools

The transcripts of the letters were all available on the site, but I thought it would be interesting to analyse their content a bit more systematically. To do this, I exported the collection from Zotero and wrote a few lines of Python code to save each transcript in a separate text file, named after the letter’s author. I could then zip up the transcripts and feed them to Voyeur Tools.

Most of the letters are pretty short, so there’s a limited amount to be gleaned from them in isolation. Voyeur Tools did, however, make it easy for me to explore further the prevalence of the phrase ‘on the right track‘.

What will be more interesting will be to compare the letters to other bodies of text about Inigo Jones, weather forecasting and rural life. I’m hoping to start mining some of this sort of material from the Trove newspapers database.

Next steps?

Using Zotero , Omeka and Voyeur Tools it was quick and easy to build my own little research hub. It’s still a work-in-progress of course — that’s the point! Now I can continue to assemble and analyse my sources, while giving my work a public face. Magic!

Tagged with: , , , , ,
Posted in digital humanities, weather

Confessions of an impatient historian

Here’s the slides from the talk I gave recently in the Scholars’ Lab at the University of Virginia. Thanks to everyone who came and to the Scholars’ Lab for their hospitality.

A podcast version is now available.

Tagged with: , , ,
Posted in digital humanities, speeches

Mining the treasures of Trove (part 2)

One of the advantages of building something yourself is that if you’re not happy with it you can tweak, change, modify and adapt until you are. But one of the disadvantages is that sometimes you get so caught up in all the tweaking, changing and adapting that you overlook a much simpler solution.

So I had a harvester that could save the publication details and content of all the newspaper articles in a search on Trove. But the warm glow of self-satisfaction quickly began to fade as I started to think about how I wanted to use the content I was harvesting.

The harvester saved the text of articles organised in directories by newspaper title. This seemed to make sense. It meant that you could easily analyse and compare the content of different newspapers. But what if you wanted to examine changes over time? In that case it’d be much easier if the articles were organised by year — then I could just pull out the a folder from a particular year, feed it to VoyeurTools, and start tracking the trends.

There ensued some minor tinkering. As a result, you can now you can pass an additional option to the harvest script, telling it whether to save the article texts and pdfs in directories by year or newspaper. Simply set the ‘zip-directory-structure’ option in harvest.ini to either ‘title’ or ‘year’. If you’re using the command-line you can use the ‘-d’ flag to set your preference. Easy.

But that set me wondering whether it might be possible to generate an overview, showing the number of articles matching a search over time. So I started on a modification of my harvest script that did just that — cycling through the search results, adding up the numbers. It wasn’t until I ran the new script for the first time that I realised there was a much simpler alternative.

All I needed to do was repeat the search for each year in the search span and grab the total results value from the page. D’uh…

So instead of sending hundreds or perhaps thousands of requests to Trove, all I needed was one for each year. From there it was easy and soon I had my first graph.

My first graph: Chinese in Australia (The Chinese Australian expert in my house predicted the 1888 peak.)

I was pretty pleased with that, but of course the raw numbers of articles on their own are rather misleading. The more interesting question was what proportion of the total number of articles for that year the search represents. Another quick tweak and I was grabbing the overall totals and calculating the proportions.

Total numbers versus proportions -- Chinese in Australia #2

At this point I invited my Twitter followers to suggest some possible topics — you can see the results on Flickr.

But what do the peaks and troughs represent? I wanted to use the graphs as a way of exploring the content itself. This was possible as I’d saved the data as JSON and used jqPlot to create the graphs in an ordinary HTML page. Courtesy of some clever hooks in the backend of jqPlot I could capture the value of any point as it was clicked. That gave me the year, so all I had to do was combine this with the search keyword values and send off a request to Trove.

So now instead of just looking at the graphs, you could explore them.

Explore -- Chinese in Australia #3

Perhaps you’re wondering how I managed to pull the Trove results into the page? Just a bit of simple AJAX magic combined with my own unofficial Trove API. (More about that in the next exciting installment!)

I’ve created a little gallery of graphs to explore. I’m still open to suggestions!

The code for gathering the data is all on Bitbucket, so start building your own. Just run the ‘do_totals.py’ script in the bin directory from the command line. The script takes two flags:

  • -q (–query) the url of your Trove search (compulsory)
  • -f (–filename) the path and filename for your data file (don’t include an extension)

The script will create a javascript file containing two JSON objects, ‘totals’ and ‘ratios’. These can then be fed to jqPlot. View the source of one of my interactive graphs to see how.

Of course it would be really nice to create a web service where people could create, share, compare and combine their graphs — but that might have to await a generous benefactor…

Tagged with: , , ,
Posted in digital humanities, experiments

Inigo Jones: The Weather Prophet

I’m currently working on a html version of this with added links and content, but in the meantime…

Prologue

Canberra was in the grip of a heatwave — the longest in its recorded history. After two weeks of hot weather, the temperature topped the century once more, as 800 visitors swarmed into town for the 1939 meeting of the Australian and New Zealand Association for the Advancement of Science (otherwise known as ANZAAS). All accommodation was booked; delegates were billeted to homes in Canberra and Queanbeyan, while some of the more adventurous took to camping, creating ‘a miniature scientists’ settlement’ on the banks of the Molonglo River. As well as the heat, visitors grappled with the city’s unusual layout. The Canberra Times reported, ‘even members of the geography and astronomical sections lost their bearings’.

The following day, 11 January 1939, delegates gathered at Telopea Park School for the opening of the congress. As the temperature soared again to a record 108.5º, the Canberra Times observed that ‘most interest of a scientific character centred [on] a courageous prophecy by Mr Inigo Jones the famous Queensland weather forecaster’. Jones predicted an early end to the broiling conditions. ‘The heat wave’, he explained, ‘was cyclical, occurring at 35 year intervals’. There had been similar spells in the 1867-68 season and again, 35 years later, in 1902-3. Therefore the current heat wave, Jones claimed confidently, ‘was following expected lines’. As the death toll mounted and the threat of bushfire loomed, everyone hoped that he was right.

The ANZAAS meeting brought together the nation’s scientific elite, as well as a number of eminent visitors — including HG Wells. But amidst this jostle of intellectual worthies, Inigo Jones was, according to the Canberra Times, ‘one of the outstanding figures’. Jones was a determined battler whose ‘fight for recognition as a long range forecaster’ had begun in the early 1920s. Although he had received some support from the Queensland government, the newspaper noted that commonwealth authorities had been ‘stubbornly turning deaf ears to his claims’. However, it seemed that this attitude might finally be changing, for the federal government had recently announced the formation of a special committee to investigate Jones’s methods.

With the details of this committee still to be finalised, the ANZAAS meeting offered Jones a timely platform from which to espouse the benefits of his system. ‘I am getting along with the paper for the Congress and trust to make a good job of it’, Jones wrote to David Rivett in December 1938, ‘perhaps some of the committee of enquiry may hear it read’. His paper, entitled ‘Meteorology as a branch of astronomy’, surveyed international research into the use of astronomical cycles for long range weather forecasting. As Jones explained, the idea that our weather might be determined by celestial events was ‘by no means new’.

The appearance of spots on the surface of the sun had long been the source of conjecture, particularly when it was recognised, around the middle of the nineteenth century, that the number of sunspots increases and decreases on a regular cycle of around 11 years. Given that the sun dominates our experience of weather, might not this sunspot cycle set in motion regular changes in the Earth’s climate? In the late nineteenth and early twentieth centuries, many scientists and enthusiastic amateurs embarked on the hunt for climatic cycles, believing that if such patterns could be found, then it might at last be possible to forecast the weather not just months, but perhaps even years ahead.

‘After fifty years’ study’ Inigo Jones was convinced that he had discovered the ‘key to the puzzle’. The sunspot cycle, he explained, was determined by the movements of the outer planets — Jupiter, Saturn, Neptune, and Uranus. This critical insight enabled him to derive a series of cycles, of varying length and importance, that could be used to develop long-range forecasts. ‘I am convinced… that the sunspot period is due to the action of Jupiter first and the other planets later’, he concluded his address, ‘and just as Pythagorus and Hipparchus, and later Copernicus, grasped the truth but not the whole truth, so may this work yet need its Kepler to place the coping stone of completion upon it’. On that modest note, Queensland’s Copernicus commended his paper to the further study of the assembled scientists.

Discussion followed amongst members of the ‘Astronomy, Mathematics, and Physics’ section of ANZAAS. ‘We have worked out all the cycles in England’, commented Sir George Simpson, the Director of the British Meteorological Bureau, ‘but they only give you an explanation of about 1 per cent of the weather variations’. Nonetheless, he advised Jones to continue his observations in the hope of finding some mathematical relation from which ‘reliable deductions’ could be made. Speaking ‘as one prophet to a brother prophet’, Professor VA Bailey similarly urged Jones to make predictions that were open to scientific verification.

The mood changed, however, when Edward Kidson, the New Zealand government meteorologist, took the floor. Detailed criticism of Jones’s paper ‘would be merely a waste of time’, he asserted. Indeed, he insisted that Jones himself had ‘no clear mental picture’ of the mechanisms he was describing. Kidson was in no mind to indulge the fancies of the elderly Queenslander, and moved that the section express an opinion that the paper ‘fell far below the standard which should be expected in a communication to such a gathering of scientists’. Discussion was quashed, and Jones withdrew, disappointed.

This ‘harsh and ill-mannered’ treatment outraged The Land newspaper, one of Jones’s most steadfast supporters. ‘It was a clear indication’, the newspaper thundered, ‘of just what Mr Jones can expect at the hands of those scientists who believe that because a system is new, or not universally accepted, it lacks merit or is not even worthy of investigation’. It warned the government to ensure that such ‘biased critics’ were not appointed to the committee that was to review Jones’s system. Country Life lambasted ‘so-called scientists’ whose intolerance made the newspaper ‘inclined to despair of “homo sapiens”’. But ‘the joke is on them’, the article concluded, for while Inigo Jones’s efforts at long-range forecasting had won him the admiration of ordinary farmers, ‘the official academicised meteorologists of the world cannot accurately forecast the weather a day ahead’.

The ANZAAS Congress marked a critical moment in Jones’s career, as he waited for the review committee to pass judgement on his system. But the Congress also highlighted the dramatic divergence in opinions surrounding the weather prophet himself. For some Inigo Jones was a neglected visionary, to others nothing more than a crank. While now he is generally cast as an amusing sidelight in the development of Australian meteorology, he is still remembered by many as a great Queensland scientist, and his forecasts continue to attract attention — particularly in times of drought.

As we grapple still with the unpredictability of our climate, with the difficulties of seasonal forecasting, it seems worthwhile to reconsider the life and work of a man who was believed to hold the answer to our uncertainties. This is not a complete biography of Inigo Jones. Instead it is an attempt to trace some of the events , influences, and relationships that culminated in the review of his system in 1939. The focus is on the way Jones and his quest were perceived — by meteorologists, by scientists, by supporters, and, of course, by himself.

> View complete publication on Scribd

Tagged with: , , , ,
Posted in articles and book chapters, weather

Mining the treasures of Trove (part 1)

Some time ago a well-meaning optometrist told me I had the eyes of a 60 year-old. I lay the blame for this premature ocular degeneration upon the many tiring hours I spent squinting at the screens of dodgy microfilm readers. Newspapers were a major source of my PhD research, and back then that meant learning a little too much about films, spools and lenses. Not to mention the unending struggle to capture and hold the best machines.

Now it’s different. Instead of spending some weeks, as I did, sampling the Australian Womens Weekly in the hope of finding relevant articles, I can go to the National Library’s Australian Newspapers database in Trove and do a keyword search. Easy. The eyesight of future historians is safe.

But ready access to millions of newspaper articles across 150 years brings new challenges. Used to coaxing evidence from a meager array of sources, historians now, as Dan Cohen notes, have to ‘grapple with abundance’. How do we use and understand our new documentary riches?

Fortunately there are a growing array of tools to help. Zotero helps us manage our sources. Voyeur Tools brings sophisticated text analysis techniques within the grasp of all. And where the tools we need do not exist, we can make them.

So that’s what I did.

I’m interested in the way we talk about the weather. Wouldn’t it be good, I was thinking a few weeks back, if I could harvest the content of newspaper articles about weather or climate and start to analyse it — looking for patterns and shifts, mapping correlations or divergences against the actual climatic record.

Well… why not?

There’s currently no API that allows you to access Trove in this way, though one is apparently under development. But being the impatient sod that I am, I’d already built most of the parts I needed. Some time ago I wrote a screen scraper to query the newspaper database and extract article details from the returned HTML. This scraper is used in the History Wall to display random newspaper articles. It’s also sitting behind my little Headline Roulette game.

I’ve been continuing to improve and refine the scraper and recently used it to create my own API to Trove hosted on Google’s AppEngine. I’ll be posting some more details about this soon. And yes, I also developed a Zotero translator for the newspapers site, which I promise to finish off!

So to make a harvester, all I needed to do was run my scraper over all the results in a search and save them in some useful form. It took me less than an hour to develop a working prototype. Since then I’ve been adding a few bells and whistles…

The other night I harvested about 1400 articles that included the phrase climate change. The harvester had saved the text content of the articles in a zip file, so I uploaded it to Voyeur Tools. Here’s a simple word cloud:

(corpus)

Or what about the ‘atomic age’ seen through major Australian newspapers in 1945-46?

(corpus)

Hmmm, this is fun…

It’s harvest time

But all you really want to know is how to do it, right? So after that overlong introduction, here’s everything you need to know.

What does the harvester do?

You feed the harvester the url of a search you’ve constructed in Trove. The harvester then loops through all the results pages, extracting the article details. These details are saved in a CSV (comma separated values) file that you should be able to open as a spreadsheet or import into a database. The fields in the CSV file currently are:

  • article id
  • article title
  • article url
  • newspaper
  • newspaper life dates and location
  • newspaper id
  • issue date
  • page reference
  • page url
  • number of user corrections to the OCR output
  • text of the article (including paragraph breaks)

In addition to the CSV file, the harvester can create two other data files for you. The first is a zip file that contains the text of all the articles, organised by newspaper and article. The internal structure of the zip is something like this:

[newspaper1 id]-[newspaper1 title]/
     [article1 id]-[article 1 issue date]-p[article1 page reference].txt
     [article2 id]-[article 2 issue date]-p[article2 page reference].txt
[newspaper2 id]-[newspaper2 title]/
     [article3 id]-[article 3 issue date]-p[article3 page reference].txt
     [article4 id]-[article 4 issue date]-p[article4 page reference].txt

For example:

35-The-Sydney-Morning-Herald/
     29765619-Friday-24-May-1946-p5.txt
     29763575-Friday-10-May-1946-p2.txt

Why is this useful? Once you have the texts organised in this format you can start feeding them to text-analysis programs. Voyeur Tools makes it easy, but there are other options like Mallet or NLTK. (Read about my first attempts at using NLTK over at NMA Labs.) As well as simple word frequencies and collocations you might want to investigate the possibilities of entity extraction, topic modelling or sentiment analysis.

The harvester also gives you the option of downloading a PDF version of every article. These are also saved in a zip file for convenience, with the same structure as above. Of course, if you’re harvesting a large number of articles this zip file might get very big.

Quick start (for the impatient)

If you just want to dive straight in here’s what you need to do:

  1. If you don’t have it already, install Python (v.2.7 is recommended, but other versions should work ok)
  2. Download my Trove Newspapers code
  3. Unzip the TroveNewspapers file and put the contents somewhere handy.
  4. Navigate to the trovenewspapers/bin directory.
  5. Open the file ‘harvest.ini’ with a text editor (Notepad will do).
  6. Change the default settings of ‘harvest.ini’ as instructed.
  7. Save ‘harvest.ini’.
  8. Find and run ‘do_harvest.py’.
  9. Sit back and watch as your harvest chugs away.

The boring details (for the cautious)

Install Python

My scraper and harvester are written in Python, so you’ll need to have it installed on your system. If you’re working in Linux you should already have it. Downloads are available for all other platforms. For example, if you’re in Windows just download and run the Windows x86 MSI Installer.

Download my code

Now download my Trove Newspapers code. It’s avaiable as a zip file, so unzip it and put the contents somewhere you can find it again. The contents look something like this:

trovenewspapers/
     data/
     harvests/
     __init.py__
     harvest.py
     retrieve.py
     utilities.py
     LICENSE.txt

We’ll talk about the bin directory shortly. The data directory contains information about the newspaper holdings available through Trove. This is used by the scraper. If you ever want to update this data, you can use the save_titles function in utilities.py, but it’s not important for the harvester.

The harvests directory is empty, but if you start a harvest without specifying an output location, this is where it’ll end up.

BeautifulSoup is an extremely useful Python library for screen scraping. The scraper relies heavily on it, so I’ve included a copy in the package for your convenience.

The two files that do all the work are harvest.py and retrieve.py. Open them up in an editor and have a look if you’re interested. The scraper logic is all in retrieve.py, while harvest.py builds and runs the harvester.

But if all you want to do is start harvesting you can ignore all this and head straight to the bin top-level directory. Here you’ll find two files, do_harvest.py and harvest.ini. You’ll also find a README file which contains another version of these instructions and some added documentation.

Set your harvest options

Open up harvest.ini in any old text editor. You’ll see it contains some instructions and a series of configuration options with default values. If you run a harvest with out changing the defaults you’ll generate a fascinating set of 28 articles that contain the phrase ‘Inclement Wragge’.

The options you can set are:

  1. query — the url of your Trove search
  2. filename — where you want to save the CSV file
  3. include-text — do you want to save the texts in a zip file (yes or no)?
  4. include-pdf — do you want to save pdfs of the articles in a zip file (yes or no)?
  5. start — the result number to start at (leave at 0 for a new harvest)

At this point you need to think — what do I actually want to harvest? Head over to the advanced search page for the newspapers database and start playing with the options until you get the results you want. Try to be as precise as possible — you don’t want to download lots of irrelevant articles.

Once you’re happy with your search, just copy the url in your browser’s location box. This url contains all the search parameters the harvester needs to find and process your results. Just paste the complete url into harvest.ini next to the ‘query’ option.

Set the filename option to tell the harvester where to save your CSV file. The harvester will use the filename you supply to build the filenames for the zip files (if you want them). If you don’t include a path the files will be saved in the bin directory. If you don’t set a filename, the harvester will create a default name — trove-newspapers-[timestamp].csv.

The ‘include-text’ and ‘include-pdf’ options should be pretty obvious. Set them to ‘yes’ if you want to save texts and pdfs, or ‘no’ if you don’t.

The ‘start’ option allows you to start your harvest at someplace other than the beginning of your results set. This is useful if your harvest is interrupted for any reason (more below). Just set it to the result number you want to start at.

Once your options are set, just save harvest.ini. It’s launch time!

Start your harvest

Remember the do_harvest.py file? It contains a little script that reads your configuration settings in harvest.ini and sends them off to the harvester. So to get things going all you need to do is run do_harvest.py.

How you actually do this depends a bit on your operating system and its settings. If you’re on Windows, then the python installation program should have told the OS to treat any file with a .py extension as a Python script. So you should just be able to double click it.

On Linux, the easiest way is to open up a terminal, cd to the bin directory and type ‘python do_harvest.py’.

That should be it. The script will let you know what’s going on, listing the articles as it processes them. Enjoy!

For lovers of the command line

The do_harvest script can also be run from the command line, with the various options supplied as arguments:

  • -q (or –query) [full url of Trove newspapers search].
  • -f (or –filename) [file and path name for the CSV output].
  • -t (or –text) Create a zip file containing the text of articles.
  • -p (or –pdf) Create a zip file containing pdfs of articles.
  • -s (or –start) The result number to start at.

For example:

python do_harvest.py -q http://trove.nla.gov.au/newspaper/result?exactPhrase=inclement+wragge -f /home/wragge/trove-output.csv -t -p

Command line arguments will override any of the settings in harvest.ini.

If you’re using Windows you’ll need to make sure that the location of your Python
installation is included in your Windows path variable.

If something goes wrong

If there are problems at the Trove end, the harvester will take a little 10 second nap before trying again. It’ll do this 10 times before it finally gives up. Just before it dies, the script will write some details out to an error file ([your filename]_error.txt), including some instructions on what to do next.

This error file will include the number of the last completed record. Simply insert this as the ‘start’ value in harvest.ini (or include on the command line with the -s flag) and run do_harvest.py again. The harvester will spring back into life.

You’ll probably have a duplicate row in your CSV file at the point where the harvest failed, but that’s easy to delete.

What’s next?

Please have a go and let me know how you fare. You can add comments here, or raise issues over at my Bitbucket repository.

I’m thinking about building a little GUI version if there’s enough interest, and I have a few other improvements in mind.

I’ll be posting more about my adventures hacking Trove, and also about my efforts to analyse the results of my harvests (hence the part 1).

Tagged with: , , ,
Posted in experiments, the shed
Elsewhere
Sections
Twitter
http://discontents.com.au/page/2/
discontents.com.au
crawl=CC-MAIN-2013-48
276207
application/http; msgtype=response
sha1:YHEWP4TAVKNGWDJPWKJ2IX2RL24UW3XN
<urn:uuid:d3f73915-7f2a-4cce-aff7-8c5b0f3d991a>
2013-12-05T17:05:13Z
74.86.13.168
sha1:IYKR4MWTYUJI6U5TDXZU47IKI4DIAWT4
<urn:uuid:719a0e2e-8aad-42d9-ad47-2974c10a1831>
http://discontents.com.au/page/2/
length
response
<urn:uuid:ba42b8cb-9eba-47ab-bd15-1933e41e2ef5>
robots: classic
hostname: ip-10-33-133-15.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2013-48
operator: CommonCrawl Admin
description: Wide crawl of the web with URLs provided by Blekko for Winter 2013
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf