<jupyter_start><jupyter_text>Implementation1. Import Libraries<jupyter_code>from nltk.corpus import stopwords # Stopwords
from nltk.tokenize import word_tokenize, sent_tokenize # Tokenizer<jupyter_output><empty_output><jupyter_text>- Corpus:- A collection of text is known as Corpus.- Tokenizers:- This divides a text into a series of tokens2. Next, we remove the `stopwords` in the sentence for narrowing the words.StopWords :- Words such as is, an, a, the, for that do not add value to the meaning of a sentence. 3. We create a `Frequency Table` of words.This `dict` would keep record of how many times each word will appear in the text after removing `stopwords`.<jupyter_code># Diwali Essay Example
text = """The religious significance of this festival has differences. It varies from one region to another in India. There is an association of many deities, cultures, and traditions with Diwali. The reason for these differences is probably local harvest festivals. Hence, there was a fusion of these harvest festivals into one pan-Hindu festival.

According to the Ramayana, Diwali is the day of the return of Rama. This day Lord Rama returned to Ayodhya along with his wife Sita. This return was made after Rama defeated demon King Ravana. Furthermore, Rama’s brother Lakshmana and Hanuman also came back to Ayodhya victorious.

There is another popular tradition for the reason of Diwali. Here Lord Vishnu as an incarnation of Krishna killed Narakasura. Narakasura was certainly a demon. Above all, this victory brought the release of 16000 captive girls.

Furthermore, this victory shows the triumph of good over evil. This is due to Lord Krishna being good and Narakasura being evil.

Association of Diwali to Goddess Lakshmi is the belief of many Hindus. Lakshmi is the wife of Lord Vishnu. She also happens to be the Goddess of wealth and prosperity.

According to a legend, Diwali is the night of Lakshmi wedding. This night she chose and wed Vishnu. Eastern India Hindus associate Diwali with Goddess Durga or kali. Some Hindus believe Diwali to be the start of a new year."""<jupyter_output><empty_output><jupyter_text>```pythonstopWords = set(stopwords.words('english'))words = word_tokenize(text)freqtable = dict()``` 4. Depending on the words it contains and the frequency table, we will now assign a score to each sentenceWe will `sent_tokenize()` method to create an list of sentences. Also, a `dict` is to be created to keep track of the score of each sentence for later use.<jupyter_code>sentences = sent_tokenize(text)
sentenceValue = dict()<jupyter_output><empty_output><jupyter_text>5. To compare the sentences within the `text`, assign a scoreWe can compare the score by finding `average` score of particular sentence. This score is now a `Threshold`.<jupyter_code>sumValues = 0
for sentence in sentenceValue:
    sumValues += sentenceValue[sentence]
    average = int(sumValues / len(sentenceValue))<jupyter_output><empty_output><jupyter_text>6. Now, apply this `threshold` to store sentences and print the `summary`<jupyter_code>stopWords = set(stopwords.words('english'))
words = word_tokenize(text)

# Frequency Table
freqTable = dict()

for word in words:
    word = word.lower()
    if word in stopWords:
        continue
    
    if word in freqTable:
        freqTable[word] += 1
    else:
        freqTable[word] = 1
        
# Creating dict to keep the score of each sentence
sentences = sent_tokenize(text)
sentenceValue = dict()

for sentence in sentences:
    for word, freq in freqTable.items():
        if word in sentence.lower():
            if sentence in sentenceValue:
                sentenceValue[sentence] += freq
            else:
                sentenceValue[sentence] = freq

sumValues = 0
for sentence in sentenceValue:
    sumValues += sentenceValue[sentence]
    
# Average value of a sentence from the original text
avg = int(sumValues / len(sentenceValue))

# Storing sentences in the summary
summary = ''
for sentence in sentences:
    if (sentence in sentenceValue) and (sentenceValue[sentence] > (1.2 * avg)):
        summary += " " + sentence
print(summary)<jupyter_output>There is an association of many deities, cultures, and traditions with Diwali. According to the Ramayana, Diwali is the day of the return of Rama. Furthermore, Rama’s brother Lakshmana and Hanuman also came back to Ayodhya victorious. According to a legend, Diwali is the night of Lakshmi wedding.
StarcoderdataJupyterStructuredCleanDedup