<jupyter_start><jupyter_text>News Headlines SentimentUse the news api to pull the latest news articles for bitcoin and ethereum and create a DataFrame of sentiment scores for each coin. Use descriptive statistics to answer the following questions:1. Which coin had the highest mean positive score?2. Which coin had the highest negative score?3. Which coin had the highest positive score?<jupyter_code># Initial imports
import os
import pandas as pd
from dotenv import load_dotenv
from nltk.sentiment.vader import SentimentIntensityAnalyzer
analyzer = SentimentIntensityAnalyzer()

%matplotlib inline
load_dotenv()
# Read your api key environment variable
api_key = os.getenv("NEWS_API_KEY")

# Create a newsapi client
from newsapi import NewsApiClient
newsapi = NewsApiClient(api_key=api_key)
# Fetch the Bitcoin news articles
btc_news_en = newsapi.get_everything(
    q="bitcoin",
    language="en"
)
# Fetch the Ethereum news articles
eth_news_en = newsapi.get_everything(
    q="ethereum AND ether",
    language="en"
)
# Create the Bitcoin sentiment scores DataFrame
btc_sentiments = []

for article in btc_news_en["articles"]:
    try:
        text = article["content"]
        date = article["publishedAt"][:10]
        sentiment = analyzer.polarity_scores(text)
        compound = sentiment["compound"]
        pos = sentiment["pos"]
        neu = sentiment["neu"]
        neg = sentiment["neg"]
        
        btc_sentiments.append({
            "text": text,
            "compound": compound,
            "positive": pos,
            "negative": neg,
            "neutral": neu
            
        })
        
    except AttributeError:
        pass
    
# Create DataFrame
btc_df = pd.DataFrame(btc_sentiments)
btc_df.head()
# Create the ethereum sentiment scores DataFrame
eth_sentiments = []

for article in eth_news_en["articles"]:
    try:
        text = article["content"]
        date = article["publishedAt"][:10]
        sentiment = analyzer.polarity_scores(text)
        compound = sentiment["compound"]
        pos = sentiment["pos"]
        neu = sentiment["neu"]
        neg = sentiment["neg"]
        
        eth_sentiments.append({
            "text": text,
            "compound": compound,
            "positive": pos,
            "negative": neg,
            "neutral": neu
            
        })
        
    except AttributeError:
        pass
    
# Create DataFrame
eth_df = pd.DataFrame(eth_sentiments)
eth_df.head()
# Describe the Bitcoin Sentiment
btc_df.describe()
# Describe the Ethereum Sentiment
eth_df.describe()<jupyter_output><empty_output><jupyter_text>Questions:Q: Which coin had the highest mean positive score?A: Q: Which coin had the highest compound score?A: Q. Which coin had the highest positive score?A:  ---  TokenizerIn this section, you will use NLTK and Python to tokenize the text for each coin. Be sure to:1. Lowercase each word2. Remove Punctuation3. Remove Stopwords<jupyter_code>from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer, PorterStemmer
from string import punctuation
import re
import nltk

lemmatizer = WordNetLemmatizer()
# Expand the default stopwords list if necessary
sentence_tokenized = [sent_tokenize(i) for i in btc_df["text"]]

print(sentence_tokenized)
# Complete the tokenizer function
def tokenizer(text):
    """Tokenizes text."""
    sw = set(stopwords.words('english'))
    regex = re.compile("[^a-zA-Z ]")
    #regex = re.compile("btc_df["text"]")
    
    re_clean = regex.sub('', text)
    words = word_tokenize(re_clean)
    lem = [lemmatizer.lemmatize(word) for word in words]
    output = [word.lower() for word in lem if word.lower() not in sw]
    
    # Create a list of the words

    # Convert the words to lowercase

    # Remove the punctuation
    
    # Remove the stop words
    
    # Lemmatize Words into root words
    
    return output

# Create a new tokens column for bitcoin
btc_df["btc_tokenized"] = btc_df["text"].apply(tokenizer)

btc_df
# Create a new tokens column for ethereum
eth_df["eth_tokenized"] = eth_df["text"].apply(tokenizer)
eth_df<jupyter_output><empty_output><jupyter_text>---  NGrams and Frequency AnalysisIn this section you will look at the ngrams and word frequency for each coin. 1. Use NLTK to produce the n-grams for N = 2. 2. List the top 10 words for each coin.<jupyter_code>from collections import Counter
from nltk import ngrams
# Generate the Bitcoin N-grams where N=2
#def bigram_counter(btc_tokenized): 
    # Combine all articles in corpus into one large string
big_string_btc = ''.join(btc_df.text)
    #processed = process_text(big_string)
bigrams = ngrams(big_string_btc.split(), n=2)
top_10 = dict(Counter(bigrams).most_common(10))
pd.DataFrame(list(top_10.items()), columns=['bigram', 'count'])
# Generate the Ethereum N-grams where N=2
#def bigram_counter(btc_tokenized): 
    # Combine all articles in corpus into one large string
big_string_eth = ''.join(eth_df.text)
    #processed = process_text(big_string)
bigrams = ngrams(big_string_eth.split(), n=2)
top_10 = dict(Counter(bigrams).most_common(10))
pd.DataFrame(list(top_10.items()), columns=['bigram', 'count'])
# Use the token_count function to generate the top 10 words from each coin
def token_count(tokens, N=10):
    """Returns the top N tokens from the frequency count"""
    return Counter(tokens).most_common(N)
# Get the top 10 words for Bitcoin
token_count(tokenizer(big_string_btc))
# Get the top 10 words for Ethereum
token_count(tokenizer(big_string_eth))<jupyter_output><empty_output><jupyter_text>Word CloudsIn this section, you will generate word clouds for each coin to summarize the news for each coin<jupyter_code>from wordcloud import WordCloud
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
import matplotlib as mpl
mpl.rcParams['figure.figsize'] = [20.0, 10.0]
# Generate the Bitcoin word cloud
wc = WordCloud().generate(" ".join(tokenizer(big_string_btc)))
plt.imshow(wc)
# Generate the Ethereum word cloud
wc = WordCloud().generate(" ".join(tokenizer(big_string_eth)))
plt.imshow(wc)<jupyter_output><empty_output><jupyter_text>Named Entity RecognitionIn this section, you will build a named entity recognition model for both coins and visualize the tags using SpaCy.<jupyter_code>import spacy
from spacy import displacy
# Optional - download a language model for SpaCy
# !python -m spacy download en_core_web_sm
# Load the spaCy model
nlp = spacy.load('en_core_web_sm')<jupyter_output><empty_output><jupyter_text>Bitcoin NER<jupyter_code># Concatenate all of the bitcoin text together
print(big_string_btc)
# Run the NER processor on all of the text
doc = nlp(big_string_btc)
doc.user_data["title"]= "BTC NER"
displacy.render(doc, style='ent')

# Add a title to the document
#doc.user_data["title"]= "BTC NER"
# Render the visualization
# YOUR CODE HERE!
# List all Entities
for ent in doc.ents:
    print(ent.text, ent.label_)<jupyter_output>Microsoft ORG
Ukraine GPE
nine years DATE
more than $10 million MONEY
Microsoft ORG
2016 to 2018 DATE
US GPE
first ORDINAL
US GPE
October DATE
days ago DATE
about $1 billion MONEY
bitcoin GPE
the Silk Road FAC
2013 DATE
one CARDINAL
November 2017 DATE
two-month DATE
Bitcoin GPE
10,000 MONEY
first ORDINAL
the next 20 days DATE
19,665 MONEY
the past few weeks DATE
bitcoin GPE
digital ORG
hype GPE
chars]2018 ORG
2020 DATE
Tesla Tequila ORG
April DATE
Elon Musk ORG
Teslaquila WORK_OF_ART
U.S. GPE
Individual X WORK_OF_ART
chars]4 CARDINAL
U.S. GPE
November DATE
today DATE
500 CARDINAL
10.8 percent PERCENT
November DATE
monthly DATE
April DATE
fourth ORDINAL
Williams PERSON
<NAME> PERSON
<NAME> PERSON
Eddard PERSON
acolyte PERSON
the Faceless Men WORK_OF_ART
Monday DATE
$1 billion MONEY
Silk Road ORG
Ross Ulbricht PERSON
this week DATE
chars]1 CARDINAL
1 CARDINAL
Microsoft ORG
Ukraine GPE
nine years DATE
more than $10 million MONEY
Microsoft ORG
chars]8 CARDINAL
7 CARDINAL
Bitcoin [...]<jupyter_text>---  Ethereum NER<jupyter_code># Concatenate all of the bitcoin text together
print(big_string_eth)
# Run the NER processor on all of the text
doc = nlp(big_string_eth)
doc.user_data["title"]= "ETH NER"
displacy.render(doc, style='ent')

# Add a title to the document
# YOUR CODE HERE!
# Render the visualization
# YOUR CODE HERE!
# List all Entities
for ent in doc.ents:
    print(ent.text, ent.label_)<jupyter_output>2020 DATE
U.S. GPE
Wednesday DATE
morning TIME
Elon Musk WORK_OF_ART
ETH ORG
Ethereum 2.0s ORG
Wednesday DATE
U.S. GPE
Two CARDINAL
Nearly $1 billion MONEY
BTC ORG
recent weeks DATE
40% PERCENT
the last month DATE
500 MONEY
last week DATE
first ORDINAL
June DATE
bitcoin GPE
<NAME> PERSON
first ORDINAL
ul><li PERSON
Buterin ORG
VB2 WORK_OF_ART
100 CARDINAL
32 CARDINAL
Ripples XRP PERSON
this week DATE
XRP ORG
Wednesday DATE
Ethereum 2.0 ORG
ul><li PERSON
Bitcoin GPE
BTC ORG
around $15,694 MONEY
21:00 TIME
4 p.m. ET TIME
2.6% PERCENT
the previous 24 DATE
Bi PERSON
Tel Aviv GPE
Thursday DATE
Retrie PERSON
Ether DecentrsikedBhereum Poot ORG
Netaork Request ORG
Decentralized Ethereum LatDt ORG
below $18,000 MONEY
bitcoin GPE
ul><li PERSON
Bitcoin GPE
BTC ORG
around $18 MONEY
ETH ORG
Ethereum 2.0s ORG
Wednesday DATE
a wild year DATE
2019 DATE
a third CARDINAL
about a month DATE
billion-dollar MONEY
days DATE
bitcoin GPE
bitcoin GPE
ul><li PERSON
Bitcoin GPE
BTC ORG
October DATE
DeFi[...]
StarcoderdataJupyterStructuredCleanDedup