<jupyter_start><jupyter_text>Lab 2: Text Classification=============In this problem set, you will build a system for automatically classifying song lyrics comments by era. You will:- Do some basic text processing, tokenizing your input and converting it into a bag-of-words representation- Build a machine learning classifier based on the generative model, using Naive Bayes- Evaluate your classifiers and examine what they have learned- Build a logistic regression classifier (discriminative model) using scikit-learnTotal Points: 120 points  0. SetupIn order to develop this assignment, you will need [python 3.6](https://www.python.org/downloads/) and the following libraries. Most if not all of these are part of [anaconda](https://www.continuum.io/downloads), so a good starting point would be to install that.- [jupyter](http://jupyter.readthedocs.org/en/latest/install.html)- numpy (This will come if you install scipy like above, but if not install separately)- [matplotlib](http://matplotlib.org/users/installing.html)- [nosetests](https://nose.readthedocs.org/en/latest/)- [pandas](http://pandas.pydata.org/) DataframesHere is some help on installing packages in python: https://packaging.python.org/installing/. You can use ```pip --user``` to install locally without sudo.  About this assignment- This is a Jupyter notebook. You can execute cell blocks by pressing control-enter.- All of your coding will be in the python file ```lab2.py```. - The file ```tests/tests_visible.py``` contains the Gradescope autograder unit tests that will be available for you to run locally. You should run them as you work on the assignment to see that you're on the right track. You are free to look at their source code, if that helps. You can run the tests by running ```python run_tests.py``` or ```python run_tests.py -j``` for more description. - You may want to add more tests, but that is completely optional. - **To submit this assignment, submit ```lab2.py``` on Gradescope.**  Important Instructions for this assignment- Since each test case takes about 1 minute to run individually and the collective test suite takes about 20-30 minutes to run in its entirety, we recommend that when you implement an individual function you can comment out the remaining test case functions in tests/test_visible.py and only keep the corresponding test case and the def Setup(self) (i.e first function) in an uncommented state.- We estimate that your completed code should be able to complete running on all the test cases in about 20-30 minuetes. However, if your code takes longer to run, follow the next bullet point.- The gradescope autograder has a runtime limit of 40 minutes, so if your code times out with the autograder unable to run on all the test cases, then we have a solution for you. The ECE 365 Gradescope page has two assignments: **NLP Lab 2 Code** and **NLP Lab 2 Screenshot**. You will submit your code to **NLP Lab 2 Code**, which will run the autograder. If your code is unable to finish running on all the test cases before timeout, then you would need to submit a screenshot of the local test case output on the **NLP Lab 2 Screenshot** assignment. First run ```python run_tests.py -j``` in the assignment directory and then take a screenshot of the  prompt which shows your final score. An example screenshot is shown below.- You only need to submit the screenshot if the gradescope autograder is unable to run your code on all the test cases. Submitting your code in **NLP Lab 2 Code**  is a requirement and you will not recieve any credit from your screenshot submission if you have not submitted your code.  ![title](screenshot.png)<jupyter_code>import sys
from importlib import reload
import lab2
print('My Python version')

print('python: {}'.format(sys.version))
import nose

import pandas as pd
import numpy as np
import scipy as sp
import matplotlib
import matplotlib.pyplot as plt

import torch
from torch.autograd import Variable
from torch import optim

%matplotlib inline
print('My library versions')

print('pandas: {}'.format(pd.__version__))
print('numpy: {}'.format(np.__version__))
print('scipy: {}'.format(sp.__version__))
print('matplotlib: {}'.format(matplotlib.__version__))
print('nose: {}'.format(nose.__version__))
print('torch: {}'.format(torch.__version__))<jupyter_output>My library versions
pandas: 1.2.1
numpy: 1.19.4
scipy: 1.6.0
matplotlib: 3.3.3
nose: 1.3.7
torch: 1.7.1<jupyter_text>To test whether your libraries are the right version, run:`nosetests tests/test_environment.py`<jupyter_code># use ! to run shell commands in notebook
! nosetests tests/test_environment.py<jupyter_output>.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK<jupyter_text>1. Preprocessing**Total: 20 points** Read the data into a dataframe<jupyter_code>df_train = pd.read_csv('lyrics-train.csv')<jupyter_output><empty_output><jupyter_text>A dataframe is a structured representation of your data. You can preview a dataframe using `head()`<jupyter_code>df_train.head()<jupyter_output><empty_output><jupyter_text>Bags of wordsYour first task is to convert the text to a bag-of-words representation. For this data, a lot of the preprocessing is already done: the text is lower-cased, and punctuation is removed. You need only create a `counter` for each instance.- **Deliverable 1.1**: Complete the function `lab2.bag_of_words`. (5 points)- **Test**: `tests/test_visible.py:test_d1_1_bow`<jupyter_code># run this block to update the notebook as you change the preproc library
reload(lab2);
y_tr,x_tr = lab2.read_data('lyrics-train.csv',preprocessor=lab2.bag_of_words)
y_dv,x_dv = lab2.read_data('lyrics-dev.csv',preprocessor=lab2.bag_of_words)
y_te,x_te = lab2.read_data('lyrics-test-hidden.csv',preprocessor=lab2.bag_of_words)<jupyter_output><empty_output><jupyter_text>Unseen wordsOne challenge for classification is that words will appear in the test data that do not appear in the training data. Compute the number of words that appear in `lyrics-dev.csv`, but not in `lyrics-train.csv`. To do this, implement the following deliverables:- **Deliverable 1.2**: implement `lab2.compute_oov`, returning a list of words that appear in one list of bags-of-words, but not another. Also implement `lab2.aggregate_counts` (10 points)- **Tests**: `tests/test_visible.py:test_d1_3a_oov` and `tests/test_visible.py:test_d1_2agg`<jupyter_code>from collections import Counter
reload(lab2);<jupyter_output><empty_output><jupyter_text>To write fast code, you can find bottlenecks using the %%timeit cell magic. (The following line will run for about 5 mins.)<jupyter_code>%%timeit
lab2.aggregate_counts(x_tr)
counts_dv = lab2.aggregate_counts(x_dv)<jupyter_output><empty_output><jupyter_text>You can see the most common items in a counter by calling `counts.most_common()`:<jupyter_code>counts_dv.most_common(5)
counts_tr = lab2.aggregate_counts(x_tr)
reload(lab2);
len(lab2.compute_oov(counts_dv,counts_tr))
len(lab2.compute_oov(counts_tr,counts_dv))
lab2.oov_rate(counts_dv,counts_tr)<jupyter_output><empty_output><jupyter_text>30% of the words in the dev set do not appear in the training set.  Pruning the vocabularyLet's prune the vocabulary to include only words that appear at least ten times in the training data.- **Deliverable 1.3:** Implement `lab2.prune_vocabulary` (5 points)- **Test**: `tests/test_visible.py:test_d1_4_prune`<jupyter_code>reload(lab2);
x_tr_pruned, vocab = lab2.prune_vocabulary(counts_tr,x_tr,10)
x_dv_pruned, _ = lab2.prune_vocabulary(counts_tr,x_dv,10)
x_te_pruned, _ = lab2.prune_vocabulary(counts_tr,x_te,10)
len(vocab)
i = 94
print(len(x_dv[i]),len(x_dv_pruned[i]))
print(sum(x_dv[i].values()),sum(x_dv_pruned[i].values()))<jupyter_output>88 79
187 176<jupyter_text>2. Linear classificationNow we'll show you how to implement the linear classification rule, $\hat{y} = \text{argmax}_y \theta^{\top} f(x,y)$.You will use these functions in all classifiers in this assignment.**Total: 10 points**<jupyter_code>reload(lab2);<jupyter_output><empty_output><jupyter_text>The feature function vector $f(x,y)$ can be viewed as a dict, in which the values are counts, and the keys are tuples $(y,x_j)$, where $y$ is a label and $x_j$ is a base feature. Note that we must also include the offset feature, ```lab2.OFFSET```. Desired output is shown below:<jupyter_code>fv = lab2.make_feature_vector({'test':1,'case':2},'1980s')
print(fv)<jupyter_output>defaultdict(None, {('1980s', '**OFFSET**'): 1, ('1980s', 'test'): 1, ('1980s', 'case'): 2})<jupyter_text>Let's compute the entire set of labels.<jupyter_code>labels = set(y_tr) #figure out all possible labels
print(labels)<jupyter_output>{'1990s', 'pre-1980', '2000s', '1980s'}<jupyter_text>Now we implement the prediction rule, $\hat{y} = \text{argmax}_y \theta^{\top} f(x,y)$.The output should be:- A predicted label- The scores of all labelsYou can test this function using these simple hand-crafted weights.<jupyter_code>from collections import defaultdict
reload(lab2)
# weight vectors must be defaultdicts
theta_hand = defaultdict(float,
                         {('2000s','money'):0.1,
                          ('2000s','name'):0.2,
                          ('1980s','tonight'):0.1,
                          ('2000s','man'):0.1,
                          ('1990s','fly'):0.1,
                          ('pre-1980',lab2.OFFSET):0.1
                         })
lab2.predict(x_tr_pruned[0],theta_hand,labels)<jupyter_output><empty_output><jupyter_text>Now let's see how good these weights are, by evaluating on the dev set.<jupyter_code>reload(lab2);
# this applies your predict function to all the instances in ```x_dv```
y_hat = lab2.predict_all(x_dv_pruned,theta_hand,labels)
print(lab2.acc(y_hat,y_dv))<jupyter_output>0.3422222222222222<jupyter_text>3. Naive BayesYou'll now implement a Naive Bayes classifier in this section.**Total: 45 points**<jupyter_code>reload(lab2);<jupyter_output><empty_output><jupyter_text>- **Deliverable 3.1**: (warmup) implement ```get_corpus_counts``` in ```lab2.py```. (5 points)- **Test**: `tests/test_visible.py:test_d3_1_corpus_counts`This function should compute the word counts for a given label.<jupyter_code>eighties_counts = lab2.get_corpus_counts(x_tr_pruned,y_tr,"1980s");
print(eighties_counts['today'])
print(eighties_counts['yesterday'])<jupyter_output>50
14<jupyter_text>- **Deliverable 3.2**: Implement ```estimate_pxy``` in ```lab2.py```. (15 points)- **Test**: `tests/test_visible.py:test_d3_2_pxy`This function should compute the *smoothed* multinomial distribution $\log P(x \mid y)$ for a given label $y$.Note that this function takes the vocabulary as an argument. You have to assign a probability even for words that do not appear in documents with label $y$, if they are in the vocabulary.Hint: You can use ```get_corpus_counts``` in this function if you want to, but you don't have to.<jupyter_code>reload(lab2);
log_pxy = lab2.estimate_pxy(x_tr_pruned,y_tr,"1980s",0.1,vocab)<jupyter_output><empty_output><jupyter_text>Probabilities must sum to one! (or very close)<jupyter_code>sum(np.exp(list(log_pxy.values())))<jupyter_output><empty_output><jupyter_text>Let's look at the log-probabilities of the words from the hand-tuned weights<jupyter_code>print({word:log_pxy[word] for (_,word),weight in theta_hand.items() if weight>0})
log_pxy_more_smooth = lab2.estimate_pxy(x_tr_pruned,y_tr,"1980s",1000,vocab)
print({word:log_pxy_more_smooth[word] for (_,word),weight in theta_hand.items() if weight>0})<jupyter_output>{'money': -8.443741629859995, 'name': -8.43282250408468, 'tonight': -8.191919045665923, 'man': -8.295226983039361, 'fly': -8.497300695217104, '**OFFSET**': 0}<jupyter_text>- **Deliverable 3.3**: Now you are ready to implement ```estimate_nb``` in ```lab2.py```. (15 points)- **Test**: `tests/test_visible.py:test_d3_3a_nb`- The goal is that the score given by ```lab2.predict``` is equal to the joint probability $P(x,y)$, as described in the notes. Therefore, make sure your return output can be feed into ```lab2.predict```. - Don't forget the offset feature, whose weights should be set to the prior $\log P(y)$.- The log-probabilities for the offset feature should not be smoothed.- You can call the functions you have defined above, but you don't have to.<jupyter_code>reload(lab2);
theta_nb = lab2.estimate_nb(x_tr_pruned,y_tr,0.1)<jupyter_output><empty_output><jupyter_text>Let's predict for a single instance.<jupyter_code>lab2.predict(x_tr_pruned[155],theta_nb,labels)
aaa = lab2.predict(x_tr_pruned[155],theta_nb,labels)
print(aaa)
aaa = lab2.predict(x_tr_pruned[55],theta_nb,labels)
print(aaa)<jupyter_output>('2000s', defaultdict(None, {'1990s': -2125.1966084804503, 'pre-1980': -2136.834842396802, '2000s': -2099.2474010561396, '1980s': -2153.019927798136}))
('1980s', defaultdict(None, {'1990s': -1851.2801316678658, 'pre-1980': -1798.5641514619065, '2000s': -1840.50646909292, '1980s': -1735.3527509275218}))<jupyter_text>Let's predict for all instances of the development set.<jupyter_code>y_hat = lab2.predict_all(x_dv_pruned,theta_nb,labels)
print(lab2.acc(y_hat,y_dv))
# this block shows how we write and read predictions for evaluation
lab2.write_predictions(y_hat,'nb-dev.preds')
y_hat_dv = lab2.read_predictions('nb-dev.preds')
lab2.acc(y_hat_dv,y_dv)
# execute this block to write predictions for the test set
y_hat = lab2.predict_all(x_te_pruned,theta_nb,labels)
lab2.write_predictions(y_hat,'nb-test.preds')<jupyter_output><empty_output><jupyter_text>- **Deliverable 3.4**: Write a function in ```lab2.py``` called ```find_best_smoother```, which finds the smoothing value that gives best performance on the dev data.  (5 points)- **Test**: `tests/test_visible.py:test_d3_4a_nb_best`Your function should be trying at least the following values in `vals` below.Then, using this smoothing value, run your Naive Bayes classifier on the test set, and output the results.<jupyter_code>vals = np.logspace(-3,2,11)
print(vals)
reload(lab2);
best_smoother, scores = lab2.find_best_smoother(x_tr_pruned,y_tr,x_dv_pruned,y_dv,vals)
plt.semilogx(list(scores.keys()),list(scores.values()),'o-');
plt.xlabel('smoothing')
plt.ylabel('dev set accuracy');<jupyter_output><empty_output><jupyter_text>**Reflect:**- what might explain the dramatic drop in accuracy when the smoothing is increased from $10$ to $30$?- before you check, predict whether the accuracy will continue to significantly drop if you further increase the smoothing to $10000$. **Your Answer Here**:  Save the best parameters for later comparison.<jupyter_code>theta_nb = lab2.estimate_nb(x_tr_pruned,y_tr,best_smoother)
y_hat = lab2.predict_all(x_te_pruned,theta_nb,labels)
lab2.write_predictions(y_hat,'nb-best-test.preds')<jupyter_output><empty_output><jupyter_text>4. Logistic regressionYou will implement logistic regression in scikit-learn.**Total: 15 points**  4.1 Converting data to numpyNumpy is a package for numerical computing in python.You will need to convert your bag-of-words list of counters to a numpy array. - **Deliverable 4.1**: Implement `lab2.py:make_numpy()` (5 points)- **Test**: `tests/test_visible.py:test_d4_1_numpy`- **Hint**: one approach is to start with `numpy.zeros((height,width))`, and then fill in the cells by iterating through the bag-of-words list<jupyter_code>X = np.zeros((4,2))
print(X)
X[1,1] = -1
X[2,0] = 1.5
print(X)
reload(lab2);
X_tr = lab2.make_numpy(x_tr_pruned,vocab)
X_dv = lab2.make_numpy(x_dv_pruned,vocab)
X_te = lab2.make_numpy(x_te_pruned,vocab)
label_set = sorted(list(set(y_tr)))
print(label_set)
Y_tr = np.array([label_set.index(y_i) for y_i in y_tr])
Y_dv = np.array([label_set.index(y_i) for y_i in y_dv])
len(set(Y_tr))<jupyter_output><empty_output><jupyter_text>4.2 Building a logistic regression model Import the model you want to use and make an instance of the Model.<jupyter_code>from sklearn.linear_model import LogisticRegression
scikit_log_reg = LogisticRegression()<jupyter_output><empty_output><jupyter_text>Logistic Regression Model training.<jupyter_code>logisticRegr=scikit_log_reg.fit(X_tr, Y_tr)<jupyter_output>/usr/local/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:763: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(<jupyter_text>Get accuracy of training data and dev data. accuracy is defined as:(fraction of correct predictions): correct predictions / total number of data points<jupyter_code>train_acc = logisticRegr.score(X_tr, Y_tr)
dev_acc = logisticRegr.score(X_dv, Y_dv)

print(train_acc)
print(dev_acc)<jupyter_output>0.95375
0.48444444444444446<jupyter_text>**Deliverable 4.2**The noisy progress of the loss and dev set accuracy suggests that something is wrong with our training hyperparameters. Tune the ```LogisticRegression``` parameters until you can get to a dev set accuracy of at least 0.5. You may find a set of tunable parameters here: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html Complete lab2.better_model function(10 points)<jupyter_code>reload(lab2);
scikit_log_reg = lab2.better_model()
logisticRegr=scikit_log_reg.fit(X_tr, Y_tr)
train_acc = logisticRegr.score(X_tr, Y_tr)
dev_acc = logisticRegr.score(X_dv, Y_dv)
print(train_acc)
print(dev_acc)
### BEGIN HIDDEN TESTS
scikit_log_reg = lab2.better_model()
logisticRegr=scikit_log_reg.fit(X_tr, Y_tr)
dev_acc = logisticRegr.score(X_dv, Y_dv)
assert dev_acc >= 0.50
### END HIDDEN TESTS
Y_hat_te = logisticRegr.predict(X_te)
np.save('logreg-es-test.preds.npy', np.array(Y_hat_te))<jupyter_output><empty_output><jupyter_text>5. Feature analysis**Total: 20 points**  5.1 Top Features for Logistic Regression **Deliverable 5.1**: Implement ```get_top_features_LR``` to output the k most indicative features (**highest features weights**) and the k least indicative features (**lowest features weights**) for each label. (10 points)**Hint**: ```scikit_log_reg.coef_``` is the coefficient of the features.  Let's load the vanilla LR model for comparison.<jupyter_code>scikit_log_reg = LogisticRegression()
logisticRegr=scikit_log_reg.fit(X_tr, Y_tr)
reload(lab2);
print(lab2.get_top_features_LR(scikit_log_reg, vocab,label_set,'pre-1980',k=10))
print(lab2.get_top_features_LR(scikit_log_reg, vocab,label_set,'1980s',k=10))
print(lab2.get_top_features_LR(scikit_log_reg, vocab,label_set,'1990s',k=10))
print(lab2.get_top_features_LR(scikit_log_reg, vocab,label_set,'2000s',k=10))
### BEGIN HIDDEN TESTS
scikit_log_reg = LogisticRegression()
logisticRegr=scikit_log_reg.fit(X_tr, Y_tr)
assert set(lab2.get_top_features_LR(scikit_log_reg, vocab,label_set,'pre-1980',k=10)[0]) == set(['lord', 'boogie', 'very', 'feelin', 'darling', 'dancing', 'till', 'mornin', 'fool', 'percussion'])
assert set(lab2.get_top_features_LR(scikit_log_reg, vocab,label_set,'pre-1980',k=10)[1]) == set(['step', 'under', 'meant', 'runaway', 'perfect', 'yo', 'open', 'front', 'body', 'hit'])

assert set(lab2.get_top_features_LR(scikit_log_reg, vocab,label_set,'1980s',k=10)[0]) == set(['wall', 'america', 'standing', 'tumble', 'poison', 'shout', 'chance', 'heat', 'cut', 'took'])
assert set(lab2.get_top_features_LR(scikit_log_reg, vocab,label_set,'1980s',k=10)[1]) == set(['floor', 'hes', 'god', 'percussion', 'thinkin', 'finally', 'window', 'mama', 'lord', 'sing'])

assert set(lab2.get_top_features_LR(scikit_log_reg, vocab,label_set,'1990s',k=10)[0]) == set(['hit', 'yo', 'cuz', 'saw', 'dick', 'cradle', 'front', 'push', 'needed', 'rush'])
assert set(lab2.get_top_features_LR(scikit_log_reg, vocab,label_set,'1990s',k=10)[1]) == set(['dancing', 'second', 'chance', 'born', 'use', 'those', 'pretty', 'meaning', 'today', 'other'])

assert set(lab2.get_top_features_LR(scikit_log_reg, vocab,label_set,'2000s',k=10)[0]) == set(['wit', 'shut', 'shorty', 'club', 'three', 'jeans', 'side', 'ass', 'full', 'bitch'])
assert set(lab2.get_top_features_LR(scikit_log_reg, vocab,label_set,'2000s',k=10)[1]) == set(['lovin', 'rhythm', 'hip', 'lover', 'must', 'honey', 'boogie', 'woman', 'youve', 'fool'])
### END HIDDEN TESTS<jupyter_output>/usr/local/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:763: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(<jupyter_text>5.2 Top Features for Naive Bayes **Deliverable 5.2**: Implement ```get_top_features_NB``` to output the k most indicative features (**highest features weights**) and the k least indicative features (**lowest features weights**) for each label. (10 points)<jupyter_code>reload(lab2);
print(lab2.get_top_features_NB(theta_nb, label_set,'pre-1980',k=10))
print(lab2.get_top_features_NB(theta_nb, label_set,'1980s',k=10))
print(lab2.get_top_features_NB(theta_nb, label_set,'1990s',k=10))
print(lab2.get_top_features_NB(theta_nb, label_set,'2000s',k=10))
### BEGIN HIDDEN TESTS
theta_nb = lab2.estimate_nb(x_tr_pruned,y_tr,best_smoother)
assert set(lab2.get_top_features_NB(theta_nb, label_set,'pre-1980',k=10)[0]) == set(['you', 'the', 'i', 'to', 'and', 'a', 'me', 'my', 'it', 'love'])
# assert set(get_top_features_NB(theta_nb, label_set,'pre-1980',k=10)[1]) == set(['master', 'wishful', 'killin', 'benefit', 'zono', 'muzik', 'mewhy', 'overall', 'animal', 'skeet'])

assert set(lab2.get_top_features_NB(theta_nb, label_set,'1980s',k=10)[0]) == set(['you', 'the', 'i', 'to', 'me', 'a', 'and', 'it', 'my', 'love'])
# assert set(get_top_features_NB(theta_nb, label_set,'1980s',k=10)[1]) == set(['lamborghini', 'yeahthe', 'wishful', 'benefit', 'babei', 'zono', 'overall', 'billion', 'fiend', 'skeet'])

assert set(lab2.get_top_features_NB(theta_nb, label_set,'1990s',k=10)[0]) == set(['you', 'i', 'the', 'to', 'me', 'and', 'a', 'it', 'my', 'your'])
# assert set(get_top_features_NB(theta_nb, label_set,'1990s',k=10)[1]) == set(['ladada', 'toot', 'spotlights', 'reverse', 'zono', 'muzik', 'overall', 'tho', 'billion', 'skeet'])

assert set(lab2.get_top_features_NB(theta_nb, label_set,'2000s',k=10)[0]) == set(['you', 'i', 'the', 'me', 'and', 'to', 'a', 'it', 'my', 'in'])
# assert set(get_top_features_NB(theta_nb, label_set,'2000s',k=10)[1]) == set(['eternal', 'shiver', 'stepper', 'escapade', 'jojo', 'tambourine', 'dop', 'wishful', 'total', 'muzik'])
### END HIDDEN TESTS<jupyter_output><empty_output><jupyter_text>**Reflect:**- Compare the development dataset accuracy of LR and NB, which model do you think is better? - Given those indicative features of LR and NB, which model do you think is better? - You may read https://medium.com/@sangha_deb/naive-bayes-vs-logistic-regression-a319b07a5d4c for more information on a comparison between discriminative and generative models.  **Your Answer Here**:   6. Precision, Recall, and F1Besides accuracy, systems in natural language processing are evaluated using precision, recall, and F1. Such measures are essential when evaluating on an unbalanced dataset in terms of classes (labels). **Total: 10 points**  Confusion MatrixA confusion matrix is a table that is often used to describe the performance of a classification model (or "classifier") on a set of data for which the true values are known. In this section, we show one python packages (Seaborn) for making confusion matrixes.<jupyter_code>from sklearn import metrics
import seaborn as sns

predictions = logisticRegr.predict(X_dv)
cm = metrics.confusion_matrix(Y_dv, predictions)
plt.figure()
ax = sns.heatmap(cm, annot=True, fmt=".1f", linewidths=1, square = True, cmap = 'Blues_r');
ax.set_ylim(0 ,4)
plt.ylabel('Actual label');
plt.xlabel('Predicted label');
all_sample_title = 'Accuracy Score: {0:.4f}'.format(dev_acc)
plt.title(all_sample_title, size = 15);
plt.show();<jupyter_output><empty_output><jupyter_text>**Reflect**: What do you observe on the above confusion matrix? If you are the leading manager for this team project, which portion of the data would you ask your team to focus on?  **Your Answer Here**:   Precision, Recall, and F1Write a function below that takes in a predicted labels 'Y_hat' and gold labels 'Y', and returns the precision, recall, and F1 for each label.F1 is the harmonic mean of precision and recall. F1 = 2 * (precision * recall) / (precision + recall)(10 points)<jupyter_code>print(lab2.get_PRF(predictions, Y_dv, label_set, 'pre-1980'))
print(lab2.get_PRF(predictions, Y_dv, label_set, '1980s'))
print(lab2.get_PRF(predictions, Y_dv, label_set, '1990s'))
print(lab2.get_PRF(predictions, Y_dv, label_set, '2000s'))
label_set
### BEGIN HIDDEN TESTS
scikit_log_reg = LogisticRegression()
logisticRegr=scikit_log_reg.fit(X_tr, Y_tr)
predictions = logisticRegr.predict(X_dv)
a,b,c = lab2.get_PRF(predictions, Y_dv, label_set, 'pre-1980')
assert abs(a-0.5078125) < 0.01
assert abs(b-0.5241935483870968) < 0.01
assert abs(c-0.5158730158730158) < 0.01

a,b,c = lab2.get_PRF(predictions, Y_dv, label_set, '1980s')
assert abs(a-0.32967032967032966) < 0.01
assert abs(b-0.28846153846153844) < 0.01
assert abs(c-0.30769230769230765) < 0.01

a,b,c = lab2.get_PRF(predictions, Y_dv, label_set, '1990s')
assert abs(a-0.391304347826087) < 0.01
assert abs(b-0.37894736842105264) < 0.01
assert abs(c-0.3850267379679144) < 0.01

a,b,c = lab2.get_PRF(predictions, Y_dv, label_set, '2000s')
assert abs(a-0.6258992805755396) < 0.01
assert abs(b-0.6850393700787402) < 0.01
assert abs(c-0.6541353383458647) < 0.01
### END HIDDEN TESTS<jupyter_output>/usr/local/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:763: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(
StarcoderdataJupyterStructuredCleanDedup