Digital HASS Champion Simon Musgrave showcases content about digitised books etc.

Published by Katy McHugh on

Written by Dr Simon Musgrave.

Mahendra Mahey was one of the invited speakers at the Digital Humanities Australasia 2018 conference in Adelaide last year. One of the things he talked about was how the British Library came to be gifted a collection of digitised books by Microsoft. The rather prosaic description of this material is: “The datasets in this collection are comprised and derived from 49,455 digitised books (65,227 volumes) largely from the 19th Century.” Which made me start thinking about whether any of these books mentioned Australia. I may have been seeking a way to avoid some other task, but during the conference, I downloaded the metadata for the collection and I wrote a Python script to pull out any records which mentioned Australia or Australians:


(Don’t judge me please, I know I’m a crap programmer – and there’s a debugging artefact left in there.)

The metadata is in json format, but I didn’t try to parse the json beyond isolating records. So the results include mentions in any field of the records, including titles and places of publication. And because the search is a regular expression search, using ‘Australia’ as a keyword also found ‘Australian’ and ‘Australians’. It turns out that there are 276 records in the catalogue which were retrieved and this is what a sample looks like:


The complete set of records can be downloaded here.

I should add that getting to the books is a bit more difficult than accessing the metadata. You can download a zip archive containing all the text in a json format – that’s a 10GB zip archive which extracts to a single file. Alternatively, you can download a decade at a time of ALTO xml format files. I had hoped to use this material in a class I was teaching, but I didn’t manage to get useable versions yet – if anyone does manage, I’d love to hear about it!
https://tinker.edu.au/digital-hass-champion-simon-musgrave-showcases-content-about-digitised-books-etc/
tinker.edu.au
crawl=CC-MAIN-2019-26
40581
application/http; msgtype=response
sha1:3FLUG5AVCBAN3EC3QJELEKR4PELAXLIN
<urn:uuid:e216c18c-d050-4a57-b4c7-d4f5ef91139c>
2019-06-27T10:22:10Z
45.113.232.37
text/html
sha1:5GCGTVIYCETS2Q36FKTQHMABKLKEZAMX
<urn:uuid:a694f3e7-fd4f-4a76-b301-a7aa1f0dd495>
https://tinker.edu.au/digital-hass-champion-simon-musgrave-showcases-content-about-digitised-books-etc/
response
<urn:uuid:a277d9b0-b6d0-428f-8ae9-20ed88291fa8>
isPartOf: CC-MAIN-2019-26
publisher: Common Crawl
description: Wide crawl of the web for June 2019
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-146-47-206.ec2.internal
software: Apache Nutch 1.15 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.1-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/