<jupyter_start><jupyter_text>Accessing Text Corpora and Lexical Resources  Accessing Text Corpora Gutenberg Corpus Checking the files available in the Project Gutenberg corpus for nltk:<jupyter_code>from nltk.corpus import gutenberg
gutenberg.fileids()<jupyter_output><empty_output><jupyter_text>Some information about each text in the corpus: average word length, average sentence length, and the average frequency of a token in the text.Note: `.raw()` outputs the contents of a given file without any processing. Thus, for  example, `len(gutenberg.raw('austen-emma'.txt)` will output the count of characters in the text, including whitespaces. `.sent()` divides the text into sentences.<jupyter_code>for fileid in gutenberg.fileids():
    num_chars = len(gutenberg.raw(fileid))
    num_words = len(gutenberg.words(fileid))
    num_sents = len(gutenberg.sents(fileid))
    num_vocab = len(set(w.lower() for w in gutenberg.words(fileid)))
    print(round(num_chars/num_words), round(num_words/num_sents), round(num_words/num_vocab), fileid)<jupyter_output>5 25 26 austen-emma.txt
5 26 17 austen-persuasion.txt
5 28 22 austen-sense.txt
4 34 79 bible-kjv.txt
5 19 5 blake-poems.txt
4 19 14 bryant-stories.txt
4 18 12 burgess-busterbrown.txt
4 20 13 carroll-alice.txt
5 20 12 chesterton-ball.txt
5 23 11 chesterton-brown.txt
5 18 11 chesterton-thursday.txt
4 21 25 edgeworth-parents.txt
5 26 15 melville-moby_dick.txt
5 52 11 milton-paradise.txt
4 12 9 shakespeare-caesar.txt
4 12 8 shakespeare-hamlet.txt
4 12 7 shakespeare-macbeth.txt
5 36 12 whitman-leaves.txt<jupyter_text>Let's try to find the longest sentence in Macbeth.<jupyter_code>macbeth_sentences = gutenberg.sents('shakespeare-macbeth.txt')
macbeth_sentences
longest_len = max(len(s) for s in macbeth_sentences)
print("Length of longest sentence: ", longest_len)
print(([s for s in macbeth_sentences if len(s) == longest_len]))<jupyter_output>[['Doubtfull', 'it', 'stood', ',', 'As', 'two', 'spent', 'Swimmers', ',', 'that', 'doe', 'cling', 'together', ',', 'And', 'choake', 'their', 'Art', ':', 'The', 'mercilesse', 'Macdonwald', '(', 'Worthie', 'to', 'be', 'a', 'Rebell', ',', 'for', 'to', 'that', 'The', 'multiplying', 'Villanies', 'of', 'Nature', 'Doe', 'swarme', 'vpon', 'him', ')', 'from', 'the', 'Westerne', 'Isles', 'Of', 'Kernes', 'and', 'Gallowgrosses', 'is', 'supply', "'", 'd', ',', 'And', 'Fortune', 'on', 'his', 'damned', 'Quarry', 'smiling', ',', 'Shew', "'", 'd', 'like', 'a', 'Rebells', 'Whore', ':', 'but', 'all', "'", 's', 'too', 'weake', ':', 'For', 'braue', 'Macbeth', '(', 'well', 'hee', 'deserues', 'that', 'Name', ')', 'Disdayning', 'Fortune', ',', 'with', 'his', 'brandisht', 'Steele', ',', 'Which', 'smoak', "'", 'd', 'with', 'bloody', 'execution', '(', 'Like', 'Valours', 'Minion', ')', 'caru', "'", 'd', 'out', 'his', 'passage', ',', 'Till', 'hee', 'fac', "'", 'd', 'the', 'Slaue', ':', 'Which', 'neu', "'", 'r', 's[...]<jupyter_text>Web and Chat Text Project Gutenberg gives us a taste of text in established language. It is important to consider informal language as well for NLP tasks. For this, we can check out NLTK's webtext corpus.<jupyter_code>from nltk.corpus import webtext
for fileid in webtext.fileids():
    print(fileid, " = ", webtext.raw(fileid)[:50], '...')<jupyter_output>firefox.txt  =  Cookie Manager: "Don't allow sites that set remove ...
grail.txt  =  SCENE 1: [wind] [clop clop clop] 
KING ARTHUR: Who ...
overheard.txt  =  White guy: So, do you have any plans for this even ...
pirates.txt  =  PIRATES OF THE CARRIBEAN: DEAD MAN'S CHEST, by Ted ...
singles.txt  =  25 SEXY MALE, seeks attrac older single lady, for  ...
wine.txt  =  Lovely delicate, fragrant Rhone wine. Polished lea ...<jupyter_text>Also available is a corpus of instant messaging chat sessions with anonymized data and text.<jupyter_code>from nltk.corpus import nps_chat
chatroom = nps_chat.posts('10-19-20s_706posts.xml')
print(chatroom[123])<jupyter_output>['i', 'do', "n't", 'want', 'hot', 'pics', 'of', 'a', 'female', ',', 'I', 'can', 'look', 'in', 'a', 'mirror', '.']<jupyter_text>Brown CorpusFrom the book... The Brown Corpus was the first million-word electronic corpus of English, created in 1961 at Brown University. This corpus contains text from 500 sources, and the sources have been categorized by genre, such as news, editorial, and so on. 1.1 gives an example of each genre (for a complete list, see http://icame.uib.no/brown/bcm-los.html).<jupyter_code>from nltk.corpus import brown
print(brown.categories())<jupyter_output>['adventure', 'belles_lettres', 'editorial', 'fiction', 'government', 'hobbies', 'humor', 'learned', 'lore', 'mystery', 'news', 'religion', 'reviews', 'romance', 'science_fiction']<jupyter_text>Words and sentences in the brown corpus can be accessed through a variety of ways, including through fileids and categories.<jupyter_code>brown.words(categories='news')
brown.words(fileids=['cg22'])
brown.sents(categories=['news','editorial','reviews'])<jupyter_output><empty_output><jupyter_text>Brown Corpus is a convenient for studying systematic differences between genres. This study of linguistics is also known as **stylistics**.Let's compare the usage of modal verbs across genres. First, let's count the modal verbs for the *news* genre.<jupyter_code>news_text = brown.words(categories='news')
fdist = nltk.FreqDist(w.lower() for w in news_text)
modals = ['can', 'could', 'may', 'might', 'must', 'will']
for modal in modals:
    print(modal,':',fdist[modal], end=' | ')<jupyter_output>can : 94 | could : 87 | may : 93 | might : 38 | must : 53 | will : 389 |<jupyter_text>Now we'll obtain the above counts for a few genres using NLTK's conditional frequency distribution functionality.<jupyter_code>cfd = nltk.ConditionalFreqDist((genre,word) for genre in brown.categories() for word in brown.words(categories=genre))
genres = ['news', 'religion', 'hobbies', 'science_fiction', 'romance', 'humor']
modals = ['can', 'could', 'may', 'might', 'must', 'will']
cfd.tabulate(conditions=genres, samples=modals)<jupyter_output>can could   may might  must  will 
           news    93    86    66    38    50   389 
       religion    82    59    78    12    54    71 
        hobbies   268    58   131    22    83   264 
science_fiction    16    49     4    12     8    16 
        romance    74   193    11    51    45    43 
          humor    16    30     8     8     9    13<jupyter_text>This idea of using word counts to distinguish between genres is taken up again in a later chapter.  Reuters Corpus The Reuters Corpus contains 10,788 news documents classified into 90 topics, totaling 1.3 million. The documents are further grouped into two sets called "training" and "test". This split can be used for training and testing algorithms that predict the topic of a document.<jupyter_code>from nltk.corpus import reuters
reuters.fileids()[:5]
reuters.categories()[:5]<jupyter_output><empty_output><jupyter_text>Note: Categories in the reuters corpus overlap with each other, as a news story can have more than one category/topic.<jupyter_code>reuters.categories('training/9865')
reuters.categories(['training/9865', 'training/9880'])
reuters.fileids('barley')[:5]<jupyter_output><empty_output><jupyter_text>We can similarly obtain words or sentences in terms of files and categories.<jupyter_code>reuters.words(categories=['barley','corn'])
reuters.sents('training/9865')<jupyter_output><empty_output><jupyter_text>Inaugural Address Corpus The inaugural address corpus is a collection of texts of presidential addresses. The corpus also has a time dimension to it.<jupyter_code>from nltk.corpus import inaugural
inaugural.fileids()[:5]
[fileid[:4] for fileid in inaugural.fileids()][:5]<jupyter_output><empty_output><jupyter_text>Looking at how the words *America* and *Citizen* are used over time:<jupyter_code>from matplotlib import *
%matplotlib inline
pyplot.rcParams["figure.figsize"] = (20,5)
cfd = nltk.ConditionalFreqDist((target, fileid[:4]) for fileid in inaugural.fileids()
           for w in inaugural.words(fileid) for target in ['america', 'citizen']
           if w.lower().startswith(target))
cfd.plot()<jupyter_output><empty_output><jupyter_text>Corpora in Other LanguagesSome examples of NLTK corpora in languages other than English.<jupyter_code>nltk.download('cess_esp')
nltk.corpus.cess_esp.words()
nltk.download('indian')
nltk.corpus.indian.words()<jupyter_output>[nltk_data] Downloading package indian to /home/sajal/nltk_data...
[nltk_data]   Unzipping corpora/indian.zip.<jupyter_text>Analysing the differences in word lengths in different languages, in a corpora of Universal Declaration of Human Rights in 300+ languages.<jupyter_code>from nltk.corpus import udhr
languages = ['Chickasaw', 'English', 'German_Deutsch','Greenlandic_Inuktikut', 'Hungarian_Magyar', 'Ibibio_Efik']
cfd = nltk.ConditionalFreqDist((lang,len(word)) for lang in languages for word in udhr.words(lang + '-Latin1'))
cfd.plot()<jupyter_output><empty_output><jupyter_text>Conditional Frequency Distributions We have already used conditional frequency distributions previously in this notebook. Let's now define it. A conditional frequency distribution is a collection of frequency distributions, each one for a different "condition". An example of this condition can be the category of a text. Using NLTK's `cfd` functionality, we can obtain a frequency distribution of words for different categories.A conditional frequency distribution needs to pair each event (example: an observed word) with a condition (example: a category).<jupyter_code>text = ['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', ...]
pairs = [('news', 'The'), ('news', 'Fulton'), ('news', 'County'), ...]<jupyter_output><empty_output><jupyter_text>Counting Words by Genre Where `FreqDist` takes a simple list as input, `ConditionalFreqDist()` takes a list of pairs.<jupyter_code>cfd = nltk.ConditionalFreqDist((genre, word) 
                               for genre in brown.categories() 
                               for word in brown.words(categories=genre))<jupyter_output><empty_output><jupyter_text>If we just look at two genres, news and romance, the pairs could be calculated like this:<jupyter_code>genre_word = [(genre,word) for genre in ['news','romance'] for word in brown.words(categories=genre)]
len(genre_word)<jupyter_output><empty_output><jupyter_text>Pairs at the beginning of the list genre_word will be of the form `('news', word)` while those at the end will be of the form `('romance', word)`.<jupyter_code>genre_word[:4]
genre_word[-4:]<jupyter_output><empty_output><jupyter_text>We can now use the list of pairs and create a `ConditionalFreqDist`.<jupyter_code>cfd = nltk.ConditionalFreqDist(genre_word)
cfd
cfd.conditions()
cfd['news'].most_common(10)<jupyter_output><empty_output><jupyter_text>Generating Random Text with Bigrams We can use conditional frequency distributions to create a table of bigrams, which can further be used to generate text.<jupyter_code>sent = ['In', 'the', 'beginning', 'God', 'created', 'the', 'heaven', 'and', 'the', 'earth', '.']
list(nltk.bigrams(sent))
def generate_model(cfdist, word, num=15):
    for i in range(num):
        print(word, end=' ')
        word = cfdist[word].max()

text = nltk.corpus.genesis.words('english-kjv.txt')
bigrams = nltk.bigrams(text)
cfd = nltk.ConditionalFreqDist(bigrams)<jupyter_output><empty_output><jupyter_text>The function generate_model contains a loop to generate text. As its second parameter we supply it with a word as our initial context, and then once inside the loop, we print the current value of the variable word, and reset word to be the most likely token in that context (using max()); next time through the loop, we use that word as our new context.<jupyter_code>cfd['living']
generate_model(cfd, 'living')<jupyter_output>living creature that he said , and the land of the land of the land<jupyter_text>Lexical Resources A lexicon, or lexical resource, is a collection of words and/or phrases along with associated information such as part of speech and sense definitions.  Wordlist Corpora Let's build a function that returns uncommon or misspelt words but removing items from an existing wordlist.<jupyter_code>def unusual_words(text):
    text_vocab = set(w.lower() for w in text if w.isalpha())
    english_vocab = set(w.lower() for w in nltk.corpus.words.words())
    unusual = text_vocab - english_vocab
    return sorted(unusual)
unusual_words(nltk.corpus.gutenberg.words('austen-sense.txt'))[:5]
unusual_words(nltk.corpus.nps_chat.words())[:5]<jupyter_output><empty_output><jupyter_text>For those who are already familar with NLTK, it conatins a corpus of stopwords, that is, high-frequency words like *the*, *to* etc.<jupyter_code>from nltk.corpus import stopwords
stopwords.words('english')[:5]<jupyter_output><empty_output><jupyter_text>Let's define a function to compute what fraction of words in a text are not in the stopwords list:<jupyter_code>def content_fraction(text):
    stop_words = stopwords.words('english')
    content = [w for w in text if w.lower() not in stop_words]
    return len(content)/len(text)
content_fraction(nltk.corpus.reuters.words())<jupyter_output><empty_output><jupyter_text>Solving a word puzzleA wordlist is a useful tool for solving a wordpuzzle. Say we have a list of letters *egivrvonl*, and we want to see the set of possible words we can build, using an obligatory letter *r*.<jupyter_code># Frequencies of letters in the word
puzzle_letters = nltk.FreqDist('egivrvonl')
puzzle_letters
obligatory = 'r'
wordlist = nltk.corpus.words.words()<jupyter_output><empty_output><jupyter_text>Now we can find a a list of words, above a length of 6 letters, using the above information<jupyter_code>[w for w in wordlist if len(w) >= 6 and obligatory in w and nltk.FreqDist(w) <= puzzle_letters]<jupyter_output><empty_output><jupyter_text>Names Corpus<jupyter_code>names = nltk.corpus.names
names.fileids()
male_names = names.words('male.txt')
female_names = names.words('female.txt')<jupyter_output><empty_output><jupyter_text>Finding male names that also appear as female names:<jupyter_code>[w for w in male_names if w in female_names][:10]<jupyter_output><empty_output><jupyter_text>Finding the most common last letters in male and female names.<jupyter_code>cfd = nltk.ConditionalFreqDist((fileid, name[-1]) for fileid in names.fileids() for name in names.words(fileid))
pyplot.rcParams["figure.figsize"] = (10,5)
cfd.plot()<jupyter_output><empty_output><jupyter_text>A Pronouncing DictionaryTaking a look at the CMU Pronouncing Dictionary for US English, designed for use by speech synthesizers.<jupyter_code>entries = nltk.corpus.cmudict.entries()
len(entries)
for entry in entries[42371:42379]:
    print(entry)<jupyter_output>('fir', ['F', 'ER1'])
('fire', ['F', 'AY1', 'ER0'])
('fire', ['F', 'AY1', 'R'])
('firearm', ['F', 'AY1', 'ER0', 'AA2', 'R', 'M'])
('firearm', ['F', 'AY1', 'R', 'AA2', 'R', 'M'])
('firearms', ['F', 'AY1', 'ER0', 'AA2', 'R', 'M', 'Z'])
('firearms', ['F', 'AY1', 'R', 'AA2', 'R', 'M', 'Z'])
('fireball', ['F', 'AY1', 'ER0', 'B', 'AO2', 'L'])<jupyter_text>The lexicon provides a list of phonetic codes for each words. Some words have multiple pronunciations, like *Fire* in the above example. Let's try looking for words whose pronunciation consists of three phones, starting with P and ending with T.<jupyter_code>for word, pron in entries:
    if len(pron) == 3:
        ph1, ph2, ph3 = pron
        if ph1 == 'P' and ph3 == 'T':
            print(word, ph2, end=' ')<jupyter_output>pait EY1 pat AE1 pate EY1 patt AE1 peart ER1 peat IY1 peet IY1 peete IY1 pert ER1 pet EH1 pete IY1 pett EH1 piet IY1 piette IY1 pit IH1 pitt IH1 pot AA1 pote OW1 pott AA1 pout AW1 puett UW1 purt ER1 put UH1 putt AH1<jupyter_text>We can also use the lexicon to find all words whose pronunciation ends with a syllable sounding like nicks.<jupyter_code>syllable = ['N','IH0','K','S']
print([word for word, pron in entries if pron[-4:] == syllable])<jupyter_output>["atlantic's", 'audiotronics', 'avionics', 'beatniks', 'calisthenics', 'centronics', 'chamonix', 'chetniks', "clinic's", 'clinics', 'conics', 'conics', 'cryogenics', 'cynics', 'diasonics', "dominic's", 'ebonics', 'electronics', "electronics'", "endotronics'", 'endotronics', 'enix', 'environics', 'ethnics', 'eugenics', 'fibronics', 'flextronics', 'harmonics', 'hispanics', 'histrionics', 'identics', 'ionics', 'kibbutzniks', 'lasersonics', 'lumonics', 'mannix', 'mechanics', "mechanics'", 'microelectronics', 'minix', 'minnix', 'mnemonics', 'mnemonics', 'molonicks', 'mullenix', 'mullenix', 'mullinix', 'mulnix', "munich's", 'nucleonics', 'onyx', 'organics', "panic's", 'panics', 'penix', 'pennix', 'personics', 'phenix', "philharmonic's", 'phoenix', 'phonics', 'photronics', 'pinnix', 'plantronics', 'pyrotechnics', 'refuseniks', "resnick's", 'respironics', 'sconnix', 'siliconix', 'skolniks', 'sonics', 'sputniks', 'technics', 'tectonics', 'tektronix', 'telectronics', 'telephonics', 'tonics', 'un[...]<jupyter_text>We can notice above that one pronunciation can be spelt in several ways:  nics, niks, nix and even ntic's with a silent t. Let's try to look at mismatches between pronunciation and writing by looking at words that end with the phonetic 'M', but end with 'N' in the written form.<jupyter_code>[w for w,pron in entries if pron[-1] == 'M' and w[-1] == 'n']<jupyter_output><empty_output><jupyter_text>Now, let's find out cases where a word is pronounced starting with the phonetic N, but doesn't start with the letter N in written form.<jupyter_code>sorted(set(w[:2] for w, pron in entries if pron[0] == 'N' and w[0] != 'n'))<jupyter_output><empty_output><jupyter_text>Phones also contain digits to represent primary stress (1), secondary stress (2) and no stress (0). In the below example we define a function to extract the stress digits and then scan our lexicon to find words having a particular stress pattern.<jupyter_code>def stress(pron):
    return [char for phone in pron for char in phone if char.isdigit()]
print([w for w, pron in entries if stress(pron) == ['0','1','0','2','0']][:20])
print([w for w, pron in entries if stress(pron) == ['0','2','0','1','0']][:20])<jupyter_output>['abbreviation', 'abbreviations', 'abomination', 'abortifacient', 'abortifacients', 'academicians', 'accommodation', 'accommodations', 'accreditation', 'accreditations', 'accumulation', 'accumulations', 'acetylcholine', 'acetylcholine', 'adjudication', 'administration', "administration's", 'administrations', "administrations'", 'aduliadae']<jupyter_text>Conditional frequency distribution can help us here to find minimally contrasting set of words. In particular, we will look for p-words containing three sounds, and group them according to their first and last sounds.<jupyter_code>p3 = [(pron[0]+'-'+pron[2], word) for (word,pron) in entries if pron[0] == 'P' and len(pron) == 3]
cfd =  nltk.ConditionalFreqDist(p3)
for template in sorted(cfd.conditions()):
    if len(cfd[template]) > 10:
          words = sorted(cfd[template])
          wordstring = ' '.join(words)
          print(template, wordstring[:70] + "...")<jupyter_output>P-CH patch pautsch peach perch petsch petsche piche piech pietsch pitch pit...
P-K pac pack paek paik pak pake paque peak peake pech peck peek perc perk ...
P-L pahl pail paille pal pale pall paul paule paull peal peale pearl pearl...
P-N paign pain paine pan pane pawn payne peine pen penh penn pin pine pinn...
P-P paap paape pap pape papp paup peep pep pip pipe pipp poop pop pope pop...
P-R paar pair par pare parr pear peer pier poor poore por pore porr pour...
P-S pace pass pasts peace pearse pease perce pers perse pesce piece piss p...
P-T pait pat pate patt peart peat peet peete pert pet pete pett piet piett...
P-UW1 peru peugh pew plew plue prew pru prue prugh pshew pugh...
P-Z p's p.'s p.s pais paiz pao's pas pause paws pays paz peas pease pei's ...<jupyter_text>Comparative Wordlists NLTK includes *Swadesh wordlists* which contain lists of common words in several languages.<jupyter_code>from nltk.corpus import swadesh
print(swadesh.fileids())
print(swadesh.words('en'))<jupyter_output>['I', 'you (singular), thou', 'he', 'we', 'you (plural)', 'they', 'this', 'that', 'here', 'there', 'who', 'what', 'where', 'when', 'how', 'not', 'all', 'many', 'some', 'few', 'other', 'one', 'two', 'three', 'four', 'five', 'big', 'long', 'wide', 'thick', 'heavy', 'small', 'short', 'narrow', 'thin', 'woman', 'man (adult male)', 'man (human being)', 'child', 'wife', 'husband', 'mother', 'father', 'animal', 'fish', 'bird', 'dog', 'louse', 'snake', 'worm', 'tree', 'forest', 'stick', 'fruit', 'seed', 'leaf', 'root', 'bark (from tree)', 'flower', 'grass', 'rope', 'skin', 'meat', 'blood', 'bone', 'fat (noun)', 'egg', 'horn', 'tail', 'feather', 'hair', 'head', 'ear', 'eye', 'nose', 'mouth', 'tooth', 'tongue', 'fingernail', 'foot', 'leg', 'knee', 'hand', 'wing', 'belly', 'guts', 'neck', 'back', 'breast', 'heart', 'liver', 'drink', 'eat', 'bite', 'suck', 'spit', 'vomit', 'blow', 'breathe', 'laugh', 'see', 'hear', 'know (a fact)', 'think', 'smell', 'fear', 'sleep', 'live', 'die', 'kill', 'fight',[...]<jupyter_text>We can access cognate (having the same linguistic meaning in different languages) words using the entries method and specifying a list of languages.<jupyter_code>fr2en = swadesh.entries(['fr','en'])
print(fr2en[:10])<jupyter_output>[('je', 'I'), ('tu, vous', 'you (singular), thou'), ('il', 'he'), ('nous', 'we'), ('vous', 'you (plural)'), ('ils, elles', 'they'), ('ceci', 'this'), ('cela', 'that'), ('ici', 'here'), ('là', 'there')]<jupyter_text>And convert it into a dictionary by:<jupyter_code>translate = dict(fr2en)
translate['nous']<jupyter_output><empty_output><jupyter_text>A similar application is to compare words in various Germanic and Romance Languages:<jupyter_code>languages = ['en', 'de', 'nl', 'es', 'fr', 'pt', 'la']
for i in range(139,143):
    print(swadesh.entries(languages)[i])<jupyter_output>('say', 'sagen', 'zeggen', 'decir', 'dire', 'dizer', 'dicere')
('sing', 'singen', 'zingen', 'cantar', 'chanter', 'cantar', 'canere')
('play', 'spielen', 'spelen', 'jugar', 'jouer', 'jogar, brincar', 'ludere')
('float', 'schweben', 'zweven', 'flotar', 'flotter', 'flutuar, boiar', 'fluctuare')<jupyter_text>WordNetFrom the book: *WordNet is a semantically-oriented dictionary of English, similar to a traditional thesaurus but with a richer structure. NLTK includes the English WordNet, with 155,287 words and 117,659 synonym sets.*  Senses and SynonymsEssentially, synonyms are words that have the same meaning. Let's use WordNet to explore these.<jupyter_code>from nltk.corpus import wordnet as wn
wn.synsets('motorcar')<jupyter_output><empty_output><jupyter_text>The entity above, `car.n.01` is called a synset, or "synonym set", a collection of synonymous words, or lemmas.<jupyter_code>wn.synset('car.n.01').lemma_names()<jupyter_output><empty_output><jupyter_text>Synsets also come with a prose definition and example sentences:<jupyter_code>wn.synset('car.n.01').definition()
wn.synset('car.n.01').examples()<jupyter_output><empty_output><jupyter_text>Let's look at some common operations involving synsets, starting with getting all the lemmas of a synset:<jupyter_code>wn.synset('car.n.01').lemmas()<jupyter_output><empty_output><jupyter_text>Looking up a particular lemma:<jupyter_code>wn.lemma('car.n.01.automobile')<jupyter_output><empty_output><jupyter_text>Getting the synset of a given lemma:<jupyter_code>wn.lemma('car.n.01.automobile').synset()<jupyter_output><empty_output><jupyter_text>And, getting the name of the lemma:<jupyter_code>wn.lemma('car.n.01.automobile').name()<jupyter_output><empty_output><jupyter_text>Unlike the word motorcar, which is unambiguous and has one synset, the word car is ambiguous, having five synsets:<jupyter_code>wn.synsets('car')
for synset in wn.synsets('car'):
    print(synset.lemma_names())
# Conveniently access all lemmas involving the word 'car'
wn.lemmas('car')<jupyter_output><empty_output><jupyter_text>WordNet HierarchyFrom the book: *WordNet synsets correspond to abstract concepts, and they don't always have corresponding words in English. These concepts are linked together in a hierarchy. Some concepts are very general, such as Entity, State, Event — these are called unique beginners or root synsets. Others, such as gas guzzler and hatchback, are much more specific. *Using WordNet, we can navigate between these concepts. As an example, let's look at concepts that are more specific, called the *hyponyms*.<jupyter_code>motorcar = wn.synset('car.n.01')
types_of_motorcar = motorcar.hyponyms()
types_of_motorcar[0]
print(sorted(lemma.name() for synset in types_of_motorcar for lemma in synset.lemmas()))<jupyter_output>['Model_T', 'S.U.V.', 'SUV', 'Stanley_Steamer', 'ambulance', 'beach_waggon', 'beach_wagon', 'bus', 'cab', 'compact', 'compact_car', 'convertible', 'coupe', 'cruiser', 'electric', 'electric_automobile', 'electric_car', 'estate_car', 'gas_guzzler', 'hack', 'hardtop', 'hatchback', 'heap', 'horseless_carriage', 'hot-rod', 'hot_rod', 'jalopy', 'jeep', 'landrover', 'limo', 'limousine', 'loaner', 'minicar', 'minivan', 'pace_car', 'patrol_car', 'phaeton', 'police_car', 'police_cruiser', 'prowl_car', 'race_car', 'racer', 'racing_car', 'roadster', 'runabout', 'saloon', 'secondhand_car', 'sedan', 'sport_car', 'sport_utility', 'sport_utility_vehicle', 'sports_car', 'squad_car', 'station_waggon', 'station_wagon', 'stock_car', 'subcompact', 'subcompact_car', 'taxi', 'taxicab', 'tourer', 'touring_car', 'two-seater', 'used-car', 'waggon', 'wagon']<jupyter_text>We can also navigate up the hierarchy by visiting hypernyms. Some words have multiple paths upwards as they can be classified in more than one way.<jupyter_code>motorcar.hypernyms()
paths = motorcar.hypernym_paths()
len(paths)
[synset.name() for synset in paths[0]]
[synset.name() for synset in paths[1]]
# Root Hypernym of a synset
motorcar.root_hypernyms()<jupyter_output><empty_output><jupyter_text>Lexical RelationsHypernyms and hyponyms are called lexical relations because they relate one synset to another. These two relations navigate up and down the "is-a" hierarchy. Another important way to navigate the WordNet network is from items to their components (meronyms) or to the things they are contained in (holonyms). For example, the parts of a tree are its trunk, crown, and so on; the part_meronyms().<jupyter_code>wn.synset('tree.n.01').part_meronyms()<jupyter_output><empty_output><jupyter_text>The substance a tree is made of includes heartwood and sapwood; the substance_meronyms().<jupyter_code>wn.synset('tree.n.01').substance_meronyms()<jupyter_output><empty_output><jupyter_text>A collection of trees forms a forest; the member_holonyms().<jupyter_code>wn.synset('tree.n.01').member_holonyms()<jupyter_output><empty_output><jupyter_text>Let's take a look at the word *mint* which has several closely related senses.<jupyter_code>for synset in wn.synsets('mint', wn.NOUN):
    print(synset.name() + ':', synset.definition())<jupyter_output>batch.n.02: (often followed by `of') a large number or amount or extent
mint.n.02: any north temperate plant of the genus Mentha with aromatic leaves and small mauve flowers
mint.n.03: any member of the mint family of plants
mint.n.04: the leaves of a mint plant used fresh or candied
mint.n.05: a candy that is flavored with a mint oil
mint.n.06: a plant where money is coined by authority of the government<jupyter_text>Looking at the part and substance holonyms, we can also see that mint.n.04 is part of mint.n.02 and the substance from which mint.n.05 is made.<jupyter_code>wn.synset('mint.n.04').part_holonyms()
wn.synset('mint.n.04').substance_holonyms()<jupyter_output><empty_output><jupyter_text>Let's look at relationship between verbs now. The verb *walk* entails *step*. Some verbs can have multiple entailments.<jupyter_code>wn.synset('walk.v.01').entailments()
wn.synset('eat.v.01').entailments()<jupyter_output><empty_output><jupyter_text>We can also use the wordnet to obtain antyonyms.<jupyter_code>wn.lemma('supply.n.02.supply').antonyms()
wn.lemma('rush.v.01.rush').antonyms()
wn.lemma('horizontal.a.01.horizontal').antonyms()<jupyter_output><empty_output><jupyter_text>Semantic SimilaritySynsets are linked by a complex network of lexical relations. For a particular synset, we can traverse the WordNet network to find synsets with related meanings. Each synset has one or more hypernym paths that link it to a root synonym. Two synsets linked to the same root may have common hypernyms. If two synsets share a very specific hypernym (i.e. one which lies low in the hypernym tree hierarchy) -  they may be closely related.<jupyter_code>right = wn.synset('right_whale.n.01')
orca = wn.synset('orca.n.01')
minke = wn.synset('minke_whale.n.01')
tortoise = wn.synset('tortoise.n.01')
novel = wn.synset('novel.n.01')
right.lowest_common_hypernyms(minke)
right.lowest_common_hypernyms(orca)
right.lowest_common_hypernyms(tortoise)
right.lowest_common_hypernyms(novel)<jupyter_output><empty_output><jupyter_text>Note that whale is very specific (and baleen whale even more so), while vertebrate is more general and entity is completely general. We can quantify the concept of generality by looking at the depth of each synset:<jupyter_code>wn.synset('whale.n.02').min_depth()<jupyter_output><empty_output><jupyter_text>wn.synset('vertebrate.n.01').min_depth()<jupyter_code>wn.synset('entity.n.01').min_depth()<jupyter_output><empty_output><jupyter_text>Similarity measures defined in WordNet, based on the shortest path that connects concepts in the hypernym hierarchy, can be used as a way to quantify the similarity between two synsets.<jupyter_code>right.path_similarity(orca)
right.path_similarity(tortoise)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup