<jupyter_start><jupyter_text>(c) <NAME> - Beginners Machine Learning - London Introduction to Unsupervised Machine Learning with AWS SagemakerIn this interesting 3hr workshop, you will take the massive dataset of UFO sightings (80,000 reports over the past century) from [National UFO Reporting Center (NUFORC)](http://www.nuforc.org/) and use Amazon's machine learning services ([AWS Sagemaker](https://aws.amazon.com/sagemaker/)) to identify the top 10 locations that are most likely to have UFO sightings. To do so, you will need to use an unsupervised machine learning algorithm.You will then take your trained model, deserialise it, convert its output to a csv format and visualise it on a map using AWS [Quicksight](https://aws.amazon.com/quicksight/) to see where these locations are. Then you can try correlating these locations with landmarks.The general machine learning workflow with AWS Sagemaker is shown below. For this assignment we will not evaluate or deploy the model but only use its output to visualise the results on a world map. What is Unsupervised Machine Learning? With unsupervised learning, data features are fed into the learning algorithm, which determines how to label them (usually with numbers 0,1,2..) and based on what. This “based on what” part dictates which unsupervised learning algorithm to follow.Most unsupervised learning-based applications utilize the sub-field called **Clustering**. One of the most famous topics under the realm of Unsupervised Learning in Machine Learning is k-Means Clustering. Even though this clustering algorithm is fairly simple, it can look challenging to newcomers into the field.  What is the difference between supervised and unsupervised machine learning?The main difference between Supervised and Unsupervised learning algorithms is the absence of data labels in the latter. What does clustering mean?**Clustering** is the process of grouping data samples together into clusters based on a certain feature that they share — exactly the purpose of unsupervised learning in the first place.Source: [Clustering using K-means algorithm](https://towardsdatascience.com/clustering-using-k-means-algorithm-81da00f156f6) How does the K-Means Algorithm work?Being a clustering algorithm, **k-Means** takes data points as input and groups them into `k` clusters. This process of grouping is the training phase of the learning algorithm. The result would be a model that takes a data sample as input and returns the cluster that the new data point belongs to, according the training that the model went through.Source - [How Does k-Means Clustering in Machine Learning Work?](https://towardsdatascience.com/how-does-k-means-clustering-in-machine-learning-work-fdaaaf5acfa0)Source: [How Does k-Means Clustering in Machine Learning Work?](https://towardsdatascience.com/how-does-k-means-clustering-in-machine-learning-work-fdaaaf5acfa0)Check out the the two articles below to learn more about how the K-Means Algorithm work:- [Clustering using K-means algorithm](https://towardsdatascience.com/clustering-using-k-means-algorithm-81da00f156f6)- [How Does k-Means Clustering in Machine Learning Work?](https://towardsdatascience.com/how-does-k-means-clustering-in-machine-learning-work-fdaaaf5acfa0) Where can you use k-means?The **k-means algorithm** can be a good fit for finding patterns or groups in large datasets that have not been explicitly labeled. Here are some example use cases in different domains:**E-commerce**- Classifying customers by purchase history or clickstream activity.**Healthcare**- Detecting patterns for diseases or success treatment scenarios.- Grouping similar images for image detection.**Finance**- Detecting fraud by detecting anomalies in the dataset. For example, detecting credit card frauds by abnormal purchase patterns.**Technology**- Building a network intrusion detection system that aims to identify attacks or malicious activity.**Meteorology**- Detecting anomalies in sensor data collection such as storm forecasting.  Step 1: Importing Data For this part of the assignment, we need to import the following packages: - **Amazon SageMaker Python SDK**: Amazon SageMaker Python SDK is an open source library for training and deploying machine-learned models on Amazon SageMaker. See [Documentation](https://sagemaker.readthedocs.io/en/stable/index.html)- **Python Built-in Library** [datetime](https://docs.python.org/2/library/datetime.html)- **Numpy** and **Pandas**<jupyter_code># TODO: Import the above packages below
import ____ as pd
import ____ as np
import ____
from ____ import datetime<jupyter_output><empty_output><jupyter_text>> **Exercise:** Construct a url to the the dataset location in your S3 bucket using the following expression and save it to `data_location`.<jupyter_code># TODO: Construct the url path to your dataset file that you have just uploaded to your newly created S3 bucket
bucket = "____"
prefix = "ufo_dataset"
data_key = "ufo_complete.csv"

# Construct a url string and save it to data_location variable
data_location = "s3://{}/{}/{}".format(bucket, prefix, data_key)

# print data_location
print(data_location)
# Internally do not process the file in chunks when loading the csv onto a dataframe 
# to ensure avoid mixed type inferences when importing the large UFO dataset. 
df = pd.read_csv(data_location, low_memory= False)
# Inspect the tail of the dataframe
df.tail()
# Inspect the shape of the dataframe
df.shape<jupyter_output><empty_output><jupyter_text>Step 2: Clearning, transforming and preparing the data<jupyter_code># TODO: Select the 'latitude' and 'longitude' columns and save it as a new dataframe `df_geo` with .copy().
df_geo = df[["____", "____"]].copy()
# Inspect the tail of df_geo
df_geo.tail()
# Fully inspect the df_geo dataframe
df_geo.info()<jupyter_output><empty_output><jupyter_text>Upon successfull inspection of the above dataframe, you should note the following with this dataframe:- There are no `null` or missing values in both columns. However, we still need to check for other incorrect entries that are not **coordinates**. Examples include: `0`, `string`s, etc.- The `latitude` column has a `dtype` of `object`. This means the column may have missing or string values where the rest of the values are numbers. If the entries in the column are non-homogenous, pandas will store the column as a `string` or `object` data type. To clean the data in this column we can use pandas' `pd.to_numeric()` method to convert the data in this column to `float` for processing. The machine learning algorithm expects the data passed in to it to be numerical digits `float`s or `int`s not `string`s. - See [Documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_numeric.html) on how to use this method.> **Exercise:** Convert the `latitude` column's datatype to `float`. You can pass in the `errors = "coerce"` option to `pd.to_numeric()` method to enforce the conversion. When conversion is not possible - i.e. values are `string`s - these strings will be replaced with `NaNs`. Therefore, you need to use a `.dropna()` method to drop rows where `NaNs` exist. Then check whether the column formats have been converted to numerical data types `float` and if any missing values are still present. **Note**: You can pass in `inplace = True` argument to `.dropna()` methods so that operations are performed in place and to avoid re-assignments.<jupyter_code># TODO: Convert the column values to numeric and whenever this is not possible replace the value with NaNs
df_geo["latitude"] = pd.____(df_geo.____, errors = "____")
# Count the number of null values in the dataframe - Expecting this to be non-zero
print("Number of null values in the dataframe before dropping rows is {}".format(df_geo.isnull().any().sum()))

# TODO: Drop all rows that NaN Values
df_geo.____(inplace=____)

# Count the number of null values in the dataframe - Expecting this to be zero
print("Number of null values in the dataframe before dropping rows is {}". format(df_geo.isnull().any().sum()))
# Count how many rows in the df have 0 values
print(df_geo[(df_geo.longitude == 0) | (df_geo.latitude == 0) ].count())
# TODO: Select all rows that have non-zero coordinate values and re-assign the selection to df_geo
df_geo = df_geo[(df_geo.longitude != ____) &(df_geo.latitude != ____) ]
# Check that the there are no coordinate values in the df_geo dataframe with 0
print(df_geo[(df_geo.longitude == 0) &(df_geo.latitude == 0)])
# Re-checking the dataframe to ensure both columns have numerical datatype such as `float` or `int`.
df_geo.info()
# Check if we have any missing values (NaNs) in our dataframe
missing_values = df_geo.isnull().values.any()
print("Are there any missing values? {}".format(missing_values))
# If there are any missing values in the dataframe, show them
if (missing_values):
    df_geo[df_geo.isnull().any(axis = 1)]
# TODO: store the cleaned up dataframe column values as a 2D numpy array (matrix) with datatype of float32
data_train = df_geo.values.astype("____")

# Print the 2D numpy array
data_train<jupyter_output><empty_output><jupyter_text>Step 3: Visualising the last 5000 reports of the data on the map One of the useful packages for visualising the data on a map is called **plotly**. We can import the following module from plotly package as `px`:- **plotly**'s [express](https://plot.ly/python/plotly-express/) - Plotly Express is a terse, consistent, high-level wrapper around `plotly.graph_objects` for rapid data exploration and figure generation.For data available as a tidy pandas DataFrame, we can use the Plotly Express function `px.scatter_geo` for a geographical scatter plot. The `color` argument is used to set the color of the markers from a given column of the DataFrame.<jupyter_code>import plotly.express as px

# Showing only the last 5000 rows only on a map
fig = px.scatter_geo(df_geo.iloc[-5000: -1, :], lat="latitude", lon = "longitude", 
                     title="UFO Reports by Latitude/Longitude in the world - Last 5000 Reports", color = "longitude")
fig.show()<jupyter_output><empty_output><jupyter_text>You may notice that most of the recent 5000 UFO reports have been located in the United States. Let's take a closer look at United States by using `plotly`'s `geo` layout feature to show sightings on the US map.<jupyter_code>from plotly.offline import iplot

data = [dict(
        type = 'scattergeo',
        locationmode = 'USA-states',
        lat = df_geo.iloc[-5000:-1, 0],
        lon = df_geo.iloc[-5000:-1, 1],
        mode = 'markers',
        marker = dict(
            size = 5.5,
            opacity = 0.75,
            color = 'rgb(0, 163, 81)',
            line = dict(color = 'rgb(255, 255, 255)', width = 1))
        )]

layout = dict(
         title = 'UFO Reports by Latitude/Longitude in United States - Last 5000 Reports',
         geo = dict(
             scope = 'usa',
             projection = dict(type = 'albers usa'),
             showland = True,
             landcolor = 'rgb(250, 250, 250)',
             subunitwidth = 1,
             subunitcolor = 'rgb(217, 217, 217)',
             countrywidth = 1,
             countrycolor = 'rgb(217, 217, 217)',
             showlakes = True,
             lakecolor = 'rgb(255, 255, 255)')
        )

figure = dict(data = data, layout = layout)
iplot(figure)<jupyter_output><empty_output><jupyter_text>Step 3: Create and train our model<jupyter_code># Define number of clusters and output location URL to save the trained model
num_clusters = 10
output_location = "s3://" + bucket + "/model-artifacts"<jupyter_output><empty_output><jupyter_text>To pass a training command to Amazon Sagemaker, we need to grab the details of the current execution role **ARN ID** whose credentials we are using to call the Sagemaker API. > **Exercise:** Grab the ARN ID of your current Execution role using the `sagemaker` SDK - See [Documentation](https://sagemaker.readthedocs.io/en/stable/session.html?highlight=get%20executionsagemaker.session.get_execution_role)<jupyter_code># TODO: Get the execution role ARN ID to pass to the sagemaker API later on
role = sagemaker.____()

# Check that you have this step correctly performed
print(role)<jupyter_output><empty_output><jupyter_text>We now can use Amazon's built-in K-means ML algorithm to find `k` clusters of data in our unlabeled UFO dataset.Amazon SageMaker uses a modified version of the web-scale k-means clustering algorithm. Compared with the original version of the algorithm, the version used by Amazon SageMaker is more accurate. Like the original algorithm, it scales to massive datasets and delivers improvements in training time. To do this, the version used by Amazon SageMaker streams mini-batches (small, random subsets) of the training data. The k-means algorithm expects tabular data, where rows represent the observations that you want to cluster, and the columns represent attributes of the observations. See [Documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/k-means.html)To ask AWS sagemaker for training a model using this algorithm we need to define a **K-means Estimator**. KMeans estimators can be configured by setting **hyperparameters**. These hyperparameters are arguments passed into the estimator's Constructor Function. This estimator requires the following hyperparameters to be passed in `sagemaker.KMeans()`:- `role` (str) – An AWS IAM role (either name or full ARN)- `train_instance_count` (int) – Number of Amazon EC2 instances to use for training. We only need 1 for this exercise.- `train_instance_type` (str) – Type of EC2 instance to use for training, for example, ‘ml.c4.xlarge’. This is the **compute resources** that you want Amazon SageMaker to use for model training. Compute resources are ML compute instances that are managed by Amazon SageMaker.- `k` (int) – The number of clusters to produce. We need to 10 for this exercise.- `output_path` (str) - The URL of the S3 bucket where you want to store the output of the job.<jupyter_code># TODO: Define the training API request to AWS Sagemaker
kmeans = sagemaker.____(role = ____,
               train_instance_count = ____,
               train_instance_type = "____",
               output_path = ____,
               k = ____)<jupyter_output><empty_output><jupyter_text>The following diagram shows how you train and deploy a model with Amazon SageMakern - See [Documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-training.html) For this assignment, Amazon SageMaker provides training algorithms that are great out-of-the-box solution for quick model training.  We have used some helper code above to clean and prepare the dataset and configure AWS Sagemaker API calls but do not need to specify training code or even a training code image from EC2 Container Registry. We only need to pass in the dataset for training with AWS's KMeans default algorithm. If we wanted to specify our own algorithms or use one of the popular deep learning frameworks - tensorflow/etc. - then we provide additional training code.To train a model in Amazon SageMaker, you create a **training job** using the `kmeans.fit()` method. - See [Documentation](https://sagemaker.readthedocs.io/en/stable/kmeans.html?highlight=kmeans.fitsagemaker.KMeans.fit)The training job requires the following information passed in to `.fit()` method:- `record_set(data_train)` - The training records to train the KMeans Estimator on. Here `data_train` must be passed in to the `kmeans.record_set()` method to convert our 2D numpy array data to a `RecordSet` object that is required by the algorithm. - See [Documentation](https://sagemaker.readthedocs.io/en/stable/sagemaker.amazon.amazon_estimator.html?highlight=record_set()sagemaker.amazon.amazon_estimator.AmazonAlgorithmEstimatorBase.record_set)- `job_name` (str) - Training job name. If not specified, the estimator generates a default job name, based on the training image name and current timestamp.Amazon SageMaker then launches the ML compute instances and uses the training dataset to train the model. It saves the resulting model artifacts and other output in the S3 bucket you specified for that purpose.Here we are going to construct a job name using the following expression and Python's built-in `datetime` module. This ensures our `job_name` is unique. Each training job requires a **unique** `job_name`. Otherwise, AWS will throw an error.<jupyter_code># Construct a unique job_name using datetime module
job_name = "kmeans-geo-job-{}".format(datetime.now().strftime("%Y%m%d%H%M%S"))

# Print job_name
print("Here is the job name: {}".format(job_name))<jupyter_output><empty_output><jupyter_text>> **Exercise**: Create a training job using `kmeans.fit()`. Use the AWS documentation links above to figure out how to pass in the arguments to `kmeans.fit()` for the training job to commence. If you do this step right, you should see outputs like this appear underneath the code cell:```2019-07-29 00:54:46 Starting - Starting the training job...2019-07-29 00:54:47 Starting - Launching requested ML instances...2019-07-29 00:55:44 Starting - Preparing the instances for training......2019-07-29 00:56:24 Downloading - Downloading input data...2019-07-29 00:57:05 Training - Downloading the training image.....2019-07-29 00:57:31 Uploading - Uploading generated training model2019-07-29 00:57:31 Completed - Training job completedBillable seconds: 68CPU times: user 1.78 s, sys: 18.7 ms, total: 1.8 sWall time: 3min 13s```<jupyter_code>%%time

# TOOD: Create a training job and time it. Running this code cell will send a training job request to AWS Sagemaker
kmeans.fit(kmeans.record_set(_____), job_name= _____)<jupyter_output><empty_output><jupyter_text>**Congratulations** on building and training a model on the cloud using unsupervised machine learning algorithm and saving it! Next we are going to deserialise the model so that we can use its output.  Step 4: Model Deserialisation To deserialise the compressed model output saved on our S3 bucket we need to import the following packages.- **Boto** is the Amazon Web Services (AWS) SDK for Python. It enables Python developers to create, configure, and manage AWS services, such as EC2 and S3. Boto provides an easy to use, object-oriented API, as well as low-level access to AWS service. See [Documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html)> **Exercise**: Import `boto3` package, then use the AWS Python SDK boto3 to download the compressed model from the S3 bucket to a file. You will need to construct a url to the model and save it to `path_to_model` variable. Then pass `path_to_model` to the following command `boto3.resource("s3").Bucket(bucket).download_file(path_to_model, file_name_to_save_to)`. - See [boto3 Documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html?highlight=s3.objectS3.Client.download_file)<jupyter_code># TODO: Import boto3
import ____

# Construct a url to the model. Compressed model is saved under the model-artifacts folder
path_to_model = "model-artifacts/" + job_name + "/output/model.tar.gz"

# TODO: Use the AWS Python SDK boto3 to download the compressed model output from S3 bucket onto `model.tar.gz` file.
boto3.____("____").____(____).download_file(____, "model.tar.gz")<jupyter_output><empty_output><jupyter_text>To deserialise the compressed model output saved on our S3 bucket we need to import the following packages.- **Python's Built-in module** `os` - See [Documentation](https://docs.python.org/2/library/os.htmlos.system)Python's built-in system module `os.system()` can be used to execute a shell command `tar -zxvf` on the `model.tar.gz` compressed gzipped file. This command shell helps to extract tar files out a `tar.gz` archives. The `-zxvf` flags can passed in to `os.system()` to perform the following commands: - `-z` - The file is a “gzipped” file- `-x` - Extract files- `-v` - Verbose, print the file names as they are extracted one by one- `-f` - Use the following tar archive for the operationSee [Linux's tar Man Pages](https://linux.die.net/man/1/tar) for more details on the `tar` shell command. > **Exercise:** Use `os.system()` method to run the `tar` command on the compressed gzip file `model.tar.gz` with the above flags.<jupyter_code># TODO: Import the required packages for deserialisation
import os

# TODO: Use Python's built-in os package to open the compressed model output
os.system("____ -____ model.tar.gz")<jupyter_output><empty_output><jupyter_text>`os.system()` later can be used to execute the `unzip` shell command on `model_algo-1`. `unzip` shell command lists, tests, or extracts files from a ZIP archive. See [Linux unzip Man Pages](https://linux.die.net/man/1/unzip) for more details on the `unzip` command.> **Exercise:** Use `os.system()` method to unzip `model_algo-1`.<jupyter_code># TODO: Use Python's built-in os package to unzip model_algo-1 file. 
os.system("____ model_algo-1")<jupyter_output><empty_output><jupyter_text>To load the unzipped model output parameters, we need to install `mxnet` package.> **Exercise**: Use `!pip install` to install `mxnet`.<jupyter_code># TODO: Install mxnet package
!pip install ____<jupyter_output><empty_output><jupyter_text>To load the model output parameters we need to import the following package:- **MXNet**: A flexible and efficient library for deep learning. - See [Documentation](https://mxnet.apache.org/versions/master/api/python/index.html) > **Exercise**: Use `mxnet`'s `.ndarray.load()` method to load the model output parameters and assign it to `Kmeans_model_params` variable - See [Documentation](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html)<jupyter_code># TODO: Import mxnet
import ____ as mx

# TODO: Use mxnet to load the model parameters
Kmeans_model_params = mx.____.____("model_algo-1")<jupyter_output><empty_output><jupyter_text>> **Exercise**: Convert the model parameters to a dataframe called `cluster_centroids_kmeans` using `pd.DataFrame()`. You can grab the model output parameters using `Kmeans_model_params[0].asnumpy()` to pass to `pd.DataFrame()`.<jupyter_code># TODO: Convert the Kmeans_model_params to a dataframe using pandas and numpy: cluster_centroids_kmeans
cluster_centroids_kmeans = pd.____(____[0].____())

# TODO: Set the column names of the cluster_centroids_kmeans dataframe to match the df_geo column names
cluster_centroids_kmeans.____ = df_geo.____

# Print cluster_centroids_kmeans
print(cluster_centroids_kmeans)<jupyter_output><empty_output><jupyter_text>To write the content of the model output using An in-memory stream for text I/O we need to import the following package:- **Python's Built-in Package** `io` - See [Documentation](https://docs.python.org/3/library/io.htmlio.StringIO)<jupyter_code># TODO: Import Python's built-on package io
import ____

# When a csv_buffer object is created, it is initialized using StringIO() constructor
# Here no string is given to the StringIO() so the csv_buffer object is empty.
csv_buffer = io.StringIO()

# TODO: Use pandas .to_csv() method to weite the cluster_centroids_kmeans dataframe to a csv file
cluster_centroids_kmeans.____(csv_buffer, index = False)

# TODO: Let's use Amazon S3
s3_resource = boto3.resource("____")

# Use the .Object() method to upload an object in the given `bucket`
# Save the content of the csv_buffer file using the .put() method
s3_resource.Object(bucket, "results/ten_locations_kmeans.csv").put(Body = csv_buffer.getvalue())<jupyter_output><empty_output><jupyter_text>Let's quickly visualise where these top 10 coordinates are! We will use **AWS Quicksights** later on to for reporting these locations.<jupyter_code># TODO: Visualise the top 10 locations in the world most likely to have UFO Sightings
fig = px.scatter_geo(cluster_centroids_kmeans, lat="____", lon = "____", 
                     title="Top 10 Locations in the world mostly likely to have UFO Sightings", color = "longitude")
fig.show()
# TODO: Visualise the top locations in the US most likely to have UFO Sightings
data = [dict(
        type = '____',
        locationmode = 'USA-states',
        lat = ____.iloc[:, 0],
        lon = ____.iloc[:, 1],
        mode = 'markers',
        marker = dict(
            size = 5.5,
            opacity = 0.75,
            color = 'rgb(0, 163, 81)',
            line = dict(color = 'rgb(255, 255, 255)', width = 1))
        )]

layout = dict(
         title = 'Top locations in the United States most likely to have UFO Sightings',
         geo = dict(
             scope = '____',
             projection = dict(type = 'albers usa'),
             showland = True,
             landcolor = 'rgb(250, 250, 250)',
             subunitwidth = 1,
             subunitcolor = 'rgb(217, 217, 217)',
             countrywidth = 1,
             countrycolor = 'rgb(217, 217, 217)',
             showlakes = True,
             lakecolor = 'rgb(255, 255, 255)')
        )

figure = dict(data = ____, layout = ____)
iplot(____)<jupyter_output><empty_output><jupyter_text>Interesting findings! Now answer the following questions:- Which cities are the closest to these top 10 locations?- Which states in the United States are these top coordinates located in?- What landmarks - airports, research centres, etc. - do these coordinates correlate with?<jupyter_code># TODO: Your answers here
cities = ["___", "___", "___", "___", "___", "___", "___", "___", "___", "___"]
us_states = ["___", "___", "___", "___", "___", "___"]
landmarks = ["___", "___", "___", "___", "___", "___", "___", "___", "___", "___"]<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup