An Open Data Approach to Revealing Indigenous Texts in Large-Scale Digital Repositories A Case-Study of Locating Pages of Māori Text in the HathiTrust

1. Abstract

In this case study we report on our experiences in locating pages of M?ori text in the HathiTrust Digital Library (HTDL). Using traditional biographic metadata, i.e., the language field, only 182 items were returned out of HTDL’s 17.1 million volumes. Our Open Data approach is based on the freely available HathiTrust Extracted Features Dataset. We establish a collection of high frequency terms in Te Reo Māori, which we iteratively use as search terms to identify a group of candidate texts. We then apply NLP analysis to verify those texts that contain substantial amounts of the Māori language. Using this approach we were able to increase the number of volume returned to 598. This positive result suggests that scholars who want to analyse other low-resourced languages should be able to adopt our workflow to reveal otherwise hidden texts in their desired languages.

David Bainbridge (davidb@waikato.ac.nz), University of Waikato, New Zealand, J Stephen Downie (jdownie@illinois.edu), University of Illinois, USA and Hemi Whaanga , University of Waikato, New Zealand

Theme: Lux by Bootswatch.
https://dh2020.adho.org/wp-content/uploads/2020/07/669_AnOpenDataApproachtoRevealingIndigenousTextsinLargeScaleDigitalRepositoriesACaseStudyofLocatingPagesofMoriTextintheHathiTrust.html
dh2020.adho.org
crawl=CC-MAIN-2021-10
3514
application/http; msgtype=response
sha1:WSZKN4C6ZOBQQQK4PIPET5HNCDKTU7NA
<urn:uuid:1b0d2e05-409b-4267-b927-faabf5e76f8c>
2021-03-04T13:26:19Z
87.106.166.188
application/xhtml+xml
sha1:3BSYYNQKGB5GT3ZJDUVWNGSYPUXC2WK2
<urn:uuid:6176e734-2e27-475e-aec0-53491942706a>
https://dh2020.adho.org/wp-content/uploads/2020/07/669_AnOpenDataApproachtoRevealingIndigenousTextsinLargeScaleDigitalRepositoriesACaseStudyofLocatingPagesofMoriTextintheHathiTrust.html
response
<urn:uuid:bf5b8481-5640-4311-8ce0-f647f2c18454>
isPartOf: CC-MAIN-2021-10
publisher: Common Crawl
description: Wide crawl of the web for February/March 2021
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-240.ec2.internal
software: Apache Nutch 1.18 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.2-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/