<jupyter_start><jupyter_text>Steps in which i impliment thisstep1 : Import Datastep2 : Word Count - CountVectorizerstep3 : Term Frequency Inverse Document Frequencystep4 : Naive Bayes Classifierstep5 : Result<jupyter_code>import sklearn.datasets as skd<jupyter_output><empty_output><jupyter_text>Step1 : Importing Data offline<jupyter_code>categories = ['alt.atheism', 'soc.religion.christian','comp.graphics', 'sci.med']
news_train = skd.load_files('/home/manjeet/Videos/Documentation/ML-algo/Text-classification-using-Naive-Bayes/data/train',categories=categories,encoding='ISO-8859-1')
news_test = skd.load_files('/home/manjeet/Videos/Documentation/ML-algo/Text-classification-using-Naive-Bayes/data/test',categories=categories,encoding='ISO-8859-1')
#After loading the data, the variable news_test and news_train
#stores as a dictionary
news_train.keys()
news_train['target'] #here just i'm accessing the target
news_train['target_names']
news_train.target_names<jupyter_output><empty_output><jupyter_text>Step : 2 Word Count - CountVectorizer See Section-1 of Rough work for Text classification .ipynb file<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer()
x_train_tf = count_vect.fit_transform(news_train.data)
x_train_tf.shape<jupyter_output><empty_output><jupyter_text>Step3 : Term Frequency Inverse Document Frequency See Section-2 of Rough work for Text classification .ipynb file<jupyter_code>from sklearn.feature_extraction.text import TfidfTransformer
tfidf_transformer = TfidfTransformer()
x_train_tfidf = tfidf_transformer.fit_transform(x_train_tf)
x_train_tfidf.shape<jupyter_output><empty_output><jupyter_text>step4 : Naive Bayes Classifier<jupyter_code>#here we are using Multinomial Naive Bayes Classifier

#sklearn already has inbuilt Multinomial Naive Bayes Classifier package

#Using this package we can directly train our model with the
#matrix obtained from TfidTransformer

from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB().fit(x_train_tfidf,news_train.target) #parameter is data and it's label

# see section-3 for the demo that how it is doing
# also see my .pdf file of this part.<jupyter_output><empty_output><jupyter_text>step5 : Result see section 4 of Rough work for Text classification .ipynb file<jupyter_code>x_test_tf = count_vect.transform(news_test.data)
x_test_tf.shape
x_test_tfidf = tfidf_transformer.transform(x_test_tf)
predicted = clf.predict(x_test_tfidf)
from sklearn import metrics
from sklearn.metrics import accuracy_score
print("Accuracy : ", accuracy_score(news_test.target,predicted))
print(metrics.classification_report(news_test.target,predicted,target_names=news_test.target_names))

metrics.confusion_matrix(news_test.target,predicted)<jupyter_output>Accuracy :  0.8348868175765646
                        precision    recall  f1-score   support

           alt.atheism       0.97      0.60      0.74       319
         comp.graphics       0.96      0.89      0.92       389
               sci.med       0.97      0.81      0.88       396
soc.religion.christian       0.65      0.99      0.78       398

             micro avg       0.83      0.83      0.83      1502
             macro avg       0.89      0.82      0.83      1502
          weighted avg       0.88      0.83      0.84      1502<jupyter_text>Using Pipline Line conceptHere we are doing same thing which is done by us in the above<jupyter_code>import sklearn.datasets as skd

categories = ['alt.atheism', 'soc.religion.christian','comp.graphics', 'sci.med']
news_train = skd.load_files('/home/manjeet/Videos/Documentation/ML-algo/Text-classification-using-Naive-Bayes/data/train',categories=categories,encoding='ISO-8859-1')
news_test = skd.load_files('/home/manjeet/Videos/Documentation/ML-algo/Text-classification-using-Naive-Bayes/data/test',categories=categories,encoding='ISO-8859-1')

from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB

text_clf = Pipeline([('vect', TfidfVectorizer()), 
                      ('clf', MultinomialNB()) ])

# train the model
text_clf.fit(news_train.data, news_train.target)
# Predict the test cases
predicted = text_clf.predict(news_test.data)

from sklearn import metrics
from sklearn.metrics import accuracy_score
import numpy as np

print('Accuracy achieved is ' + str(np.mean(predicted == news_test.target)))
print(metrics.classification_report(news_test.target, predicted, target_names=news_test.target_names)),
metrics.confusion_matrix(news_test.target, predicted)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup