# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# ### This is a sample how we can detect outliers in each cluster.

# In this case study, we show how we can group data in the multi-dimensional space and select outliers in each group.
#
# Dataset (a CSV file) has many columns. Each column is a dimension and row is an observation, case or better say a data point in that n-dimensions' space. It is easy to present clusters (groups) in 2 or 3 dimensions' space but it is hard to visualize groups in 10 or 50-dimensions' space. For our sample, we use 2 dimensions where each group is a circle of dots on the plot. But in reality, there could be 100, 500-dimensions' space. Luckily, the math under the hood stays the same no matter how many dimensions we have.  
#
# #### 1) First we cluster data into groups (clusters). Each cluster has its own center. 

import matplotlib.pyplot as plt
import seaborn as sns; sns.set()
import numpy as np
from sklearn.datasets.samples_generator import make_blobs
from sklearn.cluster import KMeans
from util_distances_in_clusters import plot_kmeans, calc_distances, annotate_data_points

# Generate some data
X, y_true = make_blobs(n_samples=20, centers=2, cluster_std=0.80, random_state=0)
X = X[:, ::-1] # flip axes for better plotting

# Plot the data with centers' of clusters
kmeans = KMeans(n_clusters=2, random_state=0)
labels = kmeans.fit(X).predict(X)
ax = plot_kmeans(plt, kmeans, X, title='Two clusters\' sample')

# Plot same data as above but with arrows from centers' of clusters
ax = plot_kmeans(plt, kmeans, X, title='Two clusters\' sample', draw_lines=True)

# #### 2) Second we calculate distances from every data point to its cluster center. And show distances in terms of STD.

# Plot observations with labels as distances to the center of each cluster center in terms of STD
std_distances = calc_distances(X, kmeans)
ax = plot_kmeans(plt, kmeans, X, title='Two clusters\' sample')
annotate_data_points(ax, std_distances)

# In the above picture, you can see two clusters and each has its own center. We label each data point with a number of STD - how far that point lies from the center of its cluster in terms of STD. 
#
# We calculate STD among all distances in each cluster from each data point to the center of its cluster. Then every distance is divided by its cluster's STD.

# #### 3) Now let's plot just one cluster with a slider to see how we can select outliers based on STD threshold.

# %matplotlib inline
from outliers_plot_with_slider import run_demo_nb
run_demo_nb()

# In the above plot use your mouse and move the slider. You see red data points those greater than a given number of STDs.

StarcoderdataJupyterScriptsDedupFiltered