Tim Sherratt

Sharing recent updates and work-in-progress

Jan 2019

Adventures in stemming, or what happens when you search Trove for 'naturalization'

Fun fact — the Porter stemming algorithm treats the words ‘naturalisation’ and ‘naturalization’ differently. Naturalisation is stemmed to ‘naturalis’, naturalization to ‘natur’. You can try this yourself using this NLTK stemming demo.

Why does this matter? If you try searching for ‘naturalization’ in Trove you get almost 14 million results, most of which aren’t relevant because they’re matching words like ‘nature’.

Of course you can switch off stemming in Trove by using the text: modifier. A search for text:naturalization returns only 17,000 results. But now we’ll be missing related terms such as ‘naturalized’, so we have to explicity include any forms of the word that we want.

I’m not sure if this applies to other s/z spellings, but given that historical sources (in Australia at least) often tend to use both forms it’s something to keep in mind when you’re constructing your searches. #dhhacks
https://updates.timsherratt.org/2019/01/15/adventures-in-stemming.html
updates.timsherratt.org
crawl=CC-MAIN-2022-49
10993
application/http; msgtype=response
sha1:YTNNOZ4SOBK5ZDSBVQRKZ2IERCBYXGMA
<urn:uuid:860457e7-a3f0-4555-9838-56322f203375>
2022-12-08T19:59:30Z
104.200.22.214
text/html
sha1:VR6NLRGSCLE5YATP2DXEKPQYK5XDGKX4
<urn:uuid:784588f1-68e3-48de-ade8-1f0465b3a319>
https://updates.timsherratt.org/2019/01/15/adventures-in-stemming.html
response
<urn:uuid:223985a5-7bbb-4b47-9bc6-3f45023a2d14>
isPartOf: CC-MAIN-2022-49
publisher: Common Crawl
description: Wide crawl of the web for November/December 2022
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-146
software: Apache Nutch 1.19 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.4-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/