<jupyter_start><jupyter_text><NAME>'s Safe Driving Prediction (AutoML Local Compute)Now let's use Azure Automated ML! Hint: use shift + enter to run the code cells below. Once the cell turns from [*] to [], you can be sure the cell has run.   Import Needed PackagesImport the packages needed for this solution notebook. The most widely used package for machine learning is [scikit-learn](https://scikit-learn.org/stable/), [pandas](https://pandas.pydata.org/docs/getting_started/index.htmlgetting-started), and [numpy](https://numpy.org/). These packages have various features, as well as a lot of clustering, regression and classification algorithms that make it a good choice for data mining and data analysis.<jupyter_code>import os
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
import joblib
from sklearn import metrics<jupyter_output><empty_output><jupyter_text>Get Azure ML Workspace to use<jupyter_code>from azureml.core import Workspace, Dataset

# Get Workspace defined in by default config.json file
ws = Workspace.from_config()<jupyter_output><empty_output><jupyter_text>Load data from file into regular Pandas DataFrame<jupyter_code>DATA_DIR = "../../data/"
data_df = pd.read_csv(os.path.join(DATA_DIR, 'porto_seguro_safe_driver_prediction_train.csv'))

print(data_df.shape)
# print(data_df.describe())
data_df.head(5)<jupyter_output><empty_output><jupyter_text>Split Data into Train and Validatation SetsPartitioning data into training, validation, and holdout sets allows you to develop highly accurate models that are relevant to data that you collect in the future, not just the data the model was trained on. In machine learning, features are the measurable property of the object you’re trying to analyze. Typically, features are the columns of the data that you are training your model with minus the label. In machine learning, a label (categorical) or target (regression) is the output you get from your model after training it.<jupyter_code># Split in train/validation datasets (Validation=20%, Train=80%)
# Only split in test/train
train_df, validation_df = train_test_split(data_df, test_size=0.2, random_state=0)
train_df.describe()<jupyter_output><empty_output><jupyter_text>Train with Azure AutoML automatically searching for the 'best model' (Best algorithms and best hyper-parameters)  List and select primary metric to drive the AutoML classification problem<jupyter_code>from azureml.train import automl
    
# Get a list of valid metrics for your given task
automl.utilities.get_primary_metrics('classification')

# List of possible primary metrics is here:
# https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-train#primary-metric<jupyter_output><empty_output><jupyter_text>Define AutoML Experiment settings<jupyter_code>import logging

# You can provide additional settings as a **kwargs parameter for the AutoMLConfig object
# automl_settings = {
#      "whitelist_models": ['LightGBM']
# }

from azureml.train.automl import AutoMLConfig

automl_config = AutoMLConfig(task='classification',
                             primary_metric='AUC_weighted',                           
                             training_data=train_df,
                             validation_data=validation_df,
                             label_column_name="target",                                                    
                             enable_early_stopping= True,
                             blacklist_models=['LogisticRegression', 'ExtremeRandomTrees', 'RandomForest'], 
                             iterations=10,
                             # experiment_exit_score= 0.63,
                             featurization= 'auto',
                             debug_log='automated_ml_errors.log',
                             verbosity= logging.INFO,
                             enable_onnx_compatible_models=False
                             # **automl_settings
                             )

# Explanation of Settings: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-train#configure-your-experiment-settings

# AutoMLConfig info on: 
# https://docs.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.automlconfig.automlconfig<jupyter_output><empty_output><jupyter_text>Run Experiment with multiple child runs under the covers<jupyter_code>from azureml.core import Experiment

experiment_name = "SDK_local_porto_seguro_driver_pred"
print(experiment_name)

experiment = Experiment(workspace=ws, 
                        name=experiment_name)

run = experiment.submit(automl_config, show_output=True)<jupyter_output><empty_output><jupyter_text>Explore results with Widget<jupyter_code># Explore the results of automatic training with a Jupyter widget: https://docs.microsoft.com/en-us/python/api/azureml-widgets/azureml.widgets?view=azure-ml-py
from azureml.widgets import RunDetails
RunDetails(run).show()<jupyter_output><empty_output><jupyter_text>Retrieve the 'Best' Scikit-Learn Model<jupyter_code>best_run, fitted_model = run.get_output()
print(best_run)
print('--------')
print(fitted_model)<jupyter_output><empty_output><jupyter_text>Make Predictions  Prep Validation Data: Extract X values (feature columns) from dataset and convert to NumPi array for predicting<jupyter_code>import pandas as pd

#Remove Label/y column
if 'target' in validation_df.columns:
    y_validation_df = validation_df.pop('target')

x_validation_df = validation_df

x_validation_df.describe()
y_validation_df.describe()<jupyter_output><empty_output><jupyter_text>Make predictions in bulk<jupyter_code># Try the best model making predictions with the test dataset
y_predictions = fitted_model.predict(x_validation_df)

print('10 predictions: ')
print(y_predictions[:10])<jupyter_output><empty_output><jupyter_text>Evaluate ModelEvaluating performance is an essential task in machine learning. In this case, because this is a classification problem, the data scientist elected to use an AUC - ROC Curve. When we need to check or visualize the performance of the multi - class classification problem, we use AUC (Area Under The Curve) ROC (Receiver Operating Characteristics) curve. It is one of the most important evaluation metrics for checking any classification model’s performance.<img src="https://www.researchgate.net/profile/Oxana_Trifonova/publication/276079439/figure/fig2/AS:614187332034565@1523445079168/An-example-of-ROC-curves-with-good-AUC-09-and-satisfactory-AUC-065-parameters.png"     alt="Markdown Monster icon"     style="float: left; margin-right: 12px; width: 320px; height: 239px;" />  Calculate the Accuracy with Validation or Test Dataset<jupyter_code>from sklearn.metrics import accuracy_score

print('Accuracy with Scikit-Learn model:')
print(accuracy_score(y_validation_df, y_predictions))

fpr, tpr, thresholds = metrics.roc_curve(y_validation_df, y_predictions)
print('AUC (Area Under the Curve) with Scikit-Learn model:')
metrics.auc(fpr, tpr)<jupyter_output><empty_output><jupyter_text>See files associated with the 'Best run'<jupyter_code>print(best_run.get_file_names())

# best_run.download_file('azureml-logs/70_driver_log.txt') # To check errors<jupyter_output><empty_output><jupyter_text>Download Scikit-Learn model pickle file from the run<jupyter_code># Download the model .pkl file to local (Using the 'run' object)
best_run.download_file('outputs/model.pkl')<jupyter_output><empty_output><jupyter_text>Load model in memory from .pkl file<jupyter_code># Load the model into memory
import joblib
fitted_model = joblib.load('model.pkl')
print(fitted_model)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup