<jupyter_start><jupyter_text>ML Pipeline PreparationFollow the instructions below to help you create your ML pipeline. 1. Import libraries and load data from database.- Import Python libraries- Load dataset from database with [`read_sql_table`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_table.html)- Define feature and target variables X and Y<jupyter_code># import libraries
import pandas as pd
import numpy as np
import os
import pickle
from sqlalchemy import create_engine
import re
import nltk
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.model_selection import train_test_split
from sklearn.multioutput import MultiOutputClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier,AdaBoostClassifier
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import make_scorer, accuracy_score, f1_score, fbeta_score, classification_report
from scipy.stats import hmean
from scipy.stats.mstats import gmean

nltk.download(['punkt', 'wordnet', 'averaged_perceptron_tagger'])
# load data from database
engine = create_engine('sqlite:///s0umDisasterProject.db')

df = pd.read_sql_table('df',engine)
X = df['message']
Y = df.iloc[:,4:]

df.head()<jupyter_output><empty_output><jupyter_text>2. Write a tokenization function to process your text data<jupyter_code>def tokenize(text):
    url_regex = 'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+'
    detected_urls = re.findall(url_regex, text)
    for url in detected_urls:
        text = text.replace(url, "urlplaceholder")

    tokens = word_tokenize(text)
    lemmatizer = WordNetLemmatizer()

    clean_tokens = []
    for tok in tokens:
        clean_tok = lemmatizer.lemmatize(tok).lower().strip()
        clean_tokens.append(clean_tok)

    return clean_tokens<jupyter_output><empty_output><jupyter_text>3. Build a machine learning pipelineThis machine pipeline should take in the `message` column as input and output classification results on the other 36 categories in the dataset. You may find the [MultiOutputClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.multioutput.MultiOutputClassifier.html) helpful for predicting multiple target variables.<jupyter_code># PRELIMINARY PIPELINE
def model_pipeline():
    pipeline = Pipeline([
        ('vect', CountVectorizer(tokenizer=tokenize)),
        ('tfidf', TfidfTransformer()),
        ('clf', MultiOutputClassifier(RandomForestClassifier())),
    ], verbose=True)

    return pipeline

# NEW PIPELINE (AFTER SOME TRIAL-AND-ERROR ALGORITHM TEST AND GRID SEARCH)
def new_model_pipeline():
    pipeline = Pipeline([
        ('features', FeatureUnion([

            ('text_pipeline', Pipeline([
                ('vect', CountVectorizer(tokenizer=tokenize)),
                ('tfidf', TfidfTransformer())
            ])),

            ('starting_verb', StartingVerbExtractor())
        ])),

        ('clf', MultiOutputClassifier(AdaBoostClassifier()))
    ], verbose=True)

    return pipeline<jupyter_output><empty_output><jupyter_text>4. Train pipeline- Split data into train and test sets- Train pipeline<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, Y)

model = []
model = new_model_pipeline()
model.fit(X_train, y_train);<jupyter_output>[Pipeline] .......... (step 1 of 2) Processing features, total=  58.5s
[Pipeline] ............... (step 2 of 2) Processing clf, total= 2.2min<jupyter_text>Let's see is the model tells us something reasonable!<jupyter_code>msg = ['Hello I see fire in the street and many houses are destroyed, homeless people everywhere']
test_output = model.predict(msg)
print(y_train.columns.values[(test_output.flatten()==1)])<jupyter_output>['request' 'aid_related' 'shelter' 'refugees' 'buildings'
 'weather_related' 'direct_report' 'direct']<jupyter_text>5. Test your modelReport the f1 score, precision and recall for each output category of the dataset. You can do this by iterating through the columns and calling sklearn's `classification_report` on each.<jupyter_code>y_pred = model.predict(X_test)
category_names = Y.columns
Y_pred = pd.DataFrame(data=y_pred,
                          index=y_test.index,
                          columns=category_names)
    
print(classification_report(y_test, Y_pred, target_names=category_names))<jupyter_output>precision    recall  f1-score   support

               request       0.73      0.55      0.63      1130
                 offer       0.09      0.04      0.05        28
           aid_related       0.75      0.60      0.67      2657
          medical_help       0.63      0.25      0.35       524
      medical_products       0.63      0.30      0.41       331
     search_and_rescue       0.60      0.22      0.32       195
              security       0.10      0.02      0.03       121
              military       0.57      0.34      0.42       198
           child_alone       0.00      0.00      0.00         0
                 water       0.71      0.62      0.67       427
                  food       0.79      0.68      0.73       730
               shelter       0.75      0.57      0.65       594
              clothing       0.73      0.38      0.50       114
                 money       0.51      0.28      0.36       150
        missing_people       0.54      [...]<jupyter_text>6. Improve your modelUse grid search to find better parameters.   ADD CUSTOM ESTIMATOR<jupyter_code>class StartingVerbExtractor(BaseEstimator, TransformerMixin):

    def starting_verb(self, text):
        sentence_list = nltk.sent_tokenize(text)
        for sentence in sentence_list:
            pos_tags = nltk.pos_tag(tokenize(sentence))
            first_word, first_tag = pos_tags[0]
            if first_tag in ['VB', 'VBP'] or first_word == 'RT':
                return True
        return False

    def fit(self, X, y=None):
        return self

    def transform(self, X):
        X_tagged = pd.Series(X).apply(self.starting_verb)
        return pd.DataFrame(X_tagged)<jupyter_output><empty_output><jupyter_text>IMPROVE MODEL PIPELINE<jupyter_code>def new_model_pipeline():
    pipeline = Pipeline([
        ('features', FeatureUnion([

            ('text_pipeline', Pipeline([
                ('vect', CountVectorizer(tokenizer=tokenize)),
                ('tfidf', TfidfTransformer())
            ])),

            ('starting_verb', StartingVerbExtractor())
        ])),

        ('clf', MultiOutputClassifier(AdaBoostClassifier()))
    ])

    return pipeline<jupyter_output><empty_output><jupyter_text>CUSTOMIZE AND RUN GRID SEARCH**Warning** It may take several hours on a standard laptop (with CPU parallelization)<jupyter_code>model = new_model_pipeline()

parameters = {
    'features__text_pipeline__vect__ngram_range': ((1, 1), (1, 2)),
    'features__text_pipeline__vect__max_df': (0.75, 1.0),
    'features__text_pipeline__vect__max_features': (None, 5000),
    'features__text_pipeline__tfidf__use_idf': (True, False),
#    'clf__n_estimators': [10, 100],
#    'clf__learning_rate': [0.01, 0.1],
#    'features__transformer_weights': (
#        {'text_pipeline': 1, 'starting_verb': 0.5},
#        {'text_pipeline': 0.5, 'starting_verb': 1},
#        {'text_pipeline': 0.8, 'starting_verb': 1},
#    )
}

scorer = make_scorer(multioutput_fscore,greater_is_better = True)

cv = GridSearchCV(model, param_grid=parameters, scoring = scorer,verbose = 2, n_jobs = -1)

cv.fit(X_train, y_train)<jupyter_output>Fitting 3 folds for each of 16 candidates, totalling 48 fits
[CV] features__text_pipeline__tfidf__use_idf=True, features__text_pipeline__vect__max_df=0.75, features__text_pipeline__vect__max_features=None, features__text_pipeline__vect__ngram_range=(1, 1) 
[CV] features__text_pipeline__tfidf__use_idf=True, features__text_pipeline__vect__max_df=0.75, features__text_pipeline__vect__max_features=None, features__text_pipeline__vect__ngram_range=(1, 1) 
[CV] features__text_pipeline__tfidf__use_idf=True, features__text_pipeline__vect__max_df=0.75, features__text_pipeline__vect__max_features=None, features__text_pipeline__vect__ngram_range=(1, 1) 
[CV] features__text_pipeline__tfidf__use_idf=True, features__text_pipeline__vect__max_df=0.75, features__text_pipeline__vect__max_features=None, features__text_pipeline__vect__ngram_range=(1, 2) 
[CV]  features__text_pipeline__tfidf__use_idf=True, features__text_pipeline__vect__max_df=0.75, features__text_pipeline__vect__max_features=None, features_[...]<jupyter_text>7. Test your modelShow the accuracy, precision, and recall of the tuned model.  Since this project focuses on code quality, process, and  pipelines, there is no minimum performance metric needed to pass. However, make sure to fine tune your models for accuracy, precision and recall to make your project stand out - especially for your portfolio!  8. Try improving your model further. Here are a few ideas:* try other machine learning algorithms* add other features besides the TF-IDF **DISCLAIMER** Please notice that complete results are already shown in the previous paragraph, where the new model is run by default. Most relevant differences compared to the first-guess model are:1) AdaBoost classifier instead of RandomForest2) New feaure (first verb)The tuned model, compared to the first-guess sample with a limited number of trees (10) and various decision tree-based algorithms, shows the following improvements:1) Overall accuracy: from 93% to 94.5%2) Customized multi-class multi-label f1-score: from 90% to 93%  9. Export your model as a pickle file<jupyter_code># save the model to disk
filename = 'classifier.pkl'
pickle.dump(model, open(filename, 'wb'))<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup