# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] _uuid="866e7eb37a03e1a48e71ba4e4c8f4baca43472e8"
#  # <div align="center">  The Data Scientist’s Toolbox Tutorial </div>
#  ### <div align="center"><b>Quite Practical and Far from any Theoretical Concepts</b></div>
#  
# <div style="text-align:center">last update: <b>11/13/2018</b></div>

# + [markdown] _uuid="e02d495da0fb0ad24e0341e91848f4c4cfc35bdb"
# ---------------------------------------------------------------------
# Fork, Run and Follow this kernel on GitHub:
# > ###### [ GitHub](https://github.com/mjbahmani/10-steps-to-become-a-data-scientist)
#
#
# -------------------------------------------------------------------------------------------------------------
#  **I hope you find this kernel helpful and some <font color="red">UPVOTES</font> would be very much appreciated**
#  
#  -----------
#

# + [markdown] _uuid="85b27cf82d3023fd69c338df2be7afb2d7afaf32"
#  <a id="top"></a> <br>
# **Notebook Content**
# 1. [Introduction](#1)
#     1. [Import](#2)
#     1. [Version](#3)
#     1. [setup](#4)
# 1. [Python](#5)
#     1. [Python Syntax compared to other programming languages](#6)
#     1. [Basics](#2)
#     1. [Functions](#3)
#     1. [Types and Sequences](#4)
#     1. [More on Strings](#5)
#     1. [Reading and Writing CSV files](#6)
#     1. [Dates and Times](#7)
#     1. [Objects and map()](#8)
#     1. [Lambda and List Comprehensions](#9)
#     1. [OOP](#10)
# 1. [Numpy](#12)
#     1. [Creating Arrays](#13)
#     1. [Combining Arrays](#14)
#     1. [Operations](#15)
#     1. [Math Functions](#16)
#     1. [Indexing / Slicing](#17)
#     1. [Copying Data](#18)
#     1. [Iterating Over Arrays](#19)
#     1. [The Series Data Structure](#20)
#     1. [Querying a Series](#21)
# 1. [Pandas](#22)
#     1. [The DataFrame Data Structure](#22)
#     1. [Dataframe Indexing and Loading](#23)
#     1. [Missing values](#24)
#     1. [Merging Dataframes](#25)
#     1. [Making Code Pandorable](#26)
#     1. [Group by](#27)
#     1. [Scales](#28)
#     1. [Pivot Tables](#29)
#     1. [Date Functionality](#30)
#     1. [Distributions in Pandas](#31)
#     1. [Hypothesis Testing](#32)
# 1. [Matplotlib](#33)
#     1. [Scatterplots](#34)
#     1. [Line Plots](#35)
#     1. [Bar Charts](#36)
#     1. [Histograms](#37)
#     1. [Box and Whisker Plots](#38)
#     1. [Heatmaps](#39)
#     1. [Animations](#40)
#     1. [Interactivity](#41)
#     1. [DataFrame.plot](#42)
# 1. [seaborn](#43)
#     1. [5-1 Seaborn Vs Matplotlib](#43)
#     1. [5-2 10 Useful Python Data Visualization Libraries](#43)
# 1. [SKlearn](#44)
#     1. [Introduction](#45)
#     1. [Algorithms](#46)
#     1. [Framework](#47)
#     1. [Applications](#48)
#     1. [Data](#49)
#     1. [Supervised Learning: Classification](#50)
#     1. [Separate training and testing sets](#51)
#     1. [linear, binary classifier](#52)
#     1. [Prediction](#53)
#     1. [Back to the original three-class problem](#54)
#     1. [Evaluating the classifier](#55)
#     1. [Using the four flower attributes](#56)
#     1. [Unsupervised Learning: Clustering](#57)
#     1. [Supervised Learning: Regression](#58)
# 1. [Plotly](#59)
#     1. [New to Plotly?](#59)
# 1. [CheatSheet](#60)
# 1. [Conclusion](#61)
# 1. [References](#62)

# + [markdown] _uuid="6b741a5af31141057f72904b11954acfb57f5772"
# <a id="1"></a> <br>
# ## 1-Introduction

# + [markdown] _uuid="1e7678d8e7831683d0aefffdab152562c18c3188"
# In this kernel, we have a comprehensive tutorials for Five packages in python after that you can read my other kernels about machine learning

# + [markdown] _uuid="918161eb40e34a545c75850d6a7dee64d9bf5cd1"
#  <img src="http://s8.picofile.com/file/8340141626/packages1.png"  height="420" width="420">
#  
#  ###### [Go to top](#top)

# + [markdown] _uuid="953258de456acd2780aba3d18c1fc817ae689ad7"
# <a id="2"></a> <br>
# ## 1-1 Import

# + _kg_hide-input=true _uuid="4f8b6d44c99e1141bd001556f554c7efc6b77acf"
from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
import matplotlib.pyplot as plt
import plotly.graph_objs as go
import scipy.stats as stats
import plotly.plotly as py
import seaborn as sns
import pandas as pd
import numpy as np
import matplotlib
import warnings
import sklearn
import scipy
import json
import sys
import csv
import os
# -

# !pip install plotly

# + [markdown] _uuid="9a30328de5327e576716f7a57ef119e63a6e77c3"
# <a id="3"></a> <br>
# ## 1-2 Version

# + _kg_hide-input=true _uuid="3925ca44958df0a27f3b51de1dba2a18a2650110"
print('matplotlib: {}'.format(matplotlib.__version__))
print('sklearn: {}'.format(sklearn.__version__))
print('scipy: {}'.format(scipy.__version__))
print('seaborn: {}'.format(sns.__version__))
print('pandas: {}'.format(pd.__version__))
print('numpy: {}'.format(np.__version__))
print('Python: {}'.format(sys.version))

# + [markdown] _uuid="51a98c51b1bf27253f6521a2951a5d61625642bf"
# <a id="4"></a> <br>
# ##  1-3 Setup

# + _kg_hide-input=true _uuid="cbf46ff8ec8faf08c4aa9c84eb796ffcf0a8c4c8"
warnings.filterwarnings('ignore')
sns.set(color_codes=True)
plt.style.available
# %matplotlib inline
# %precision 2

# + [markdown] _uuid="5efeff35ad9951e40551d0763eaf26f08bb4119e"
#  <a id="5"></a> <br>
# # 2-Python
#
# **Python** is a modern, robust, high level programming language. It is very easy to pick up even if you are completely new to programming.
#
# It is used for:
#
# 1. web development (server-side),
# 1. software development,
# 1. mathematics,
# 1. system scripting.
#
# <a id="6"></a> <br>
# ## 2-1 Python Syntax compared to other programming languages
#
# 1. Python was designed to for readability, and has some similarities to the English language with influence from mathematics.
# 1. Python uses new lines to complete a command, as opposed to other programming languages which often use semicolons or parentheses.
# 1. Python relies on indentation, using whitespace, to define scope; such as the scope of loops, functions and classes. Other programming languages often use curly-brackets for this purpose.
# ###### [Go to top](#top)

# + [markdown] _uuid="a276b26993c902362e522d8a29a67ef670db103d"
# <a id="7"></a> <br>
# # 2-2 Python: Basics
#

# + _kg_hide-input=true _uuid="732b4733f620d36d4ac78b27261933559f7086de"
import this

# + [markdown] _uuid="46d2f5efcea352bda6ee7f8721be0001548440f1"
# <a id="8"></a> <br>
# ### 2-2-1 Variables
# A name that is used to denote something or a value is called a variable. In python, variables can be declared and values can be assigned to it as follows,

# + _kg_hide-input=true _uuid="52c065c046f96806741321dcc3e40dffec8f75fe"
x = 2
y = 5
xy = 'Hey'


# + [markdown] _uuid="762c34fcdeed69fc79fb84d7bb86c5baf8f9152e"
# <a id="9"></a> <br>
# ### 2-2-2 Operators

# + [markdown] _uuid="9dcd3e4436aa8050f116b6f7157cc986e762c794"
# | Symbol | Task Performed |
# |----|---|
# | +  | Addition |
# | -  | Subtraction |
# | /  | division |
# | %  | mod |
# | *  | multiplication |
# | //  | floor division |
# | **  | to the power of |
#
# ### Relational Operators
# | Symbol | Task Performed |
# |----|---|
# | == | True, if it is equal |
# | !=  | True, if not equal to |
# | < | less than |
# | > | greater than |
# | <=  | less than or equal to |
# | >=  | greater than or equal to |
# ### Bitwise Operators
# | Symbol | Task Performed |
# |----|---|
# | &  | Logical And |
# | l  | Logical OR |
# | ^  | XOR |
# | ~  | Negate |
# | >>  | Right shift |
# | <<  | Left shift |
#
# ###### [Go to top](#top)

# + [markdown] _uuid="281c637be82f68519f371353afa603fe842bd5fa"
# <a id="10"></a> <br>
# # 2-3 Python : Functions

# + [markdown] _uuid="b513caa4821bc20e6b833577ba4d894bce011e32"
# <br>
# `add_numbers` is a function that takes two numbers and adds them together.

# + _kg_hide-input=true _uuid="700ebb9c4d8c7c57948194e3d848b4b480e06f9a"
def add_numbers(x, y):
    return x + y

add_numbers(1, 2)


# + [markdown] _uuid="0fb081fa2f33e01e616a82aad306ab6490f48a03"
# <br>
# `add_numbers` updated to take an optional 3rd parameter. Using `print` allows printing of multiple expressions within a single cell.

# + _uuid="b7dd7bc4f7f52acac95ac94586c74f4b59a22acb"
def add_numbers(x,y,z=None):
    if (z==None):
        return x+y
    else:
        return x+y+z

print(add_numbers(1, 2))
print(add_numbers(1, 2, 3))


# + [markdown] _uuid="51d91959b8088baa0884eaf191edc9520e012625"
# <br>
# `add_numbers` updated to take an optional flag parameter.

# + _kg_hide-input=true _uuid="8f109910dd7dacbac0212d03b3a0d55e8d92b4bf"
def add_numbers(x, y, z=None, flag=False):
    if (flag):
        print('Flag is true!')
    if (z==None):
        return x + y
    else:
        return x + y + z
    
print(add_numbers(1, 2, flag=True))


# + [markdown] _uuid="052a1ebfb11cba627a4ed7ae13360e15cfd2f5bc"
# <br>
# Assign function `add_numbers` to variable `a`.

# + _kg_hide-input=true _uuid="774773afde862fefb51b22d9e676b44744410c6f"
def add_numbers(x,y):
    return x+y

a = add_numbers
a(1,2)

# + [markdown] _uuid="54244074f574884d9c241727f559466cdc646793"
# <a id="11"></a> <br>
# # 2-4 Python : Types and Sequences

# + [markdown] _uuid="fe3bfa4b99b20b534a24cd4b04953de4c81a2032"
# <br>
# Use `type` to return the object's type.

# + _uuid="ea5486e825425487854b66e135142787a8696547"
type('This is a string')

# + _uuid="6b1b6307a4b2363c5f13c86a94a6c6ed83cf05d1"
type(None)

# + _uuid="1bfe51959705a633d7ae10b8e807ba752809f8a2"
type(1)

# + _uuid="9c6ea01bf5fac295d6ca5dea7a867fd769d9854f"
type(1.0)

# + _uuid="08e7ecceb2d59d5964d039b1307736a106abef30"
type(add_numbers)

# + [markdown] _uuid="843a9232374917b366c4afadc94fd123e13bc2cf"
# <br>
# Tuples are an immutable data structure (cannot be altered).

# + _uuid="02c0f949470c05c302705b8cbde9dbc4c5e9ec58"
x = (1, 'a', 2, 'b')
type(x)

# + [markdown] _uuid="32e3d57ead5741f615aaf0ec276583163a6fc1a7"
# <br>
# Lists are a mutable data structure.

# + _uuid="e559da4fdd4adcf955e36ee33209728c7f04c873"
x = [1, 'a', 2, 'b']
type(x)

# + [markdown] _uuid="1c25590aa57c37e383007ea43e9aebd08dbb7ef8"
# <br>
# Use `append` to append an object to a list.

# + _uuid="46b3996bb68391469648cfaf732d9f891aad8e76"
x.append(3.3)
print(x)

# + [markdown] _uuid="a0a9f9ad271ff5cab43f7d95c00f917dfa42cda3"
# <br>
# This is an example of how to loop through each item in the list.

# + _uuid="fdbcbc6f491284146fb7ec2288a48dca70d028f0"
for item in x:
    print(item)

# + [markdown] _uuid="a863bf6c33d37823812c0d373bc685f8018328c3"
# <br>
# Or using the indexing operator:

# + _uuid="34ebdc7a81ca55c6d96576e8e1032bb2e2a8c38d"
i=0
while( i != len(x) ):
    print(x[i])
    i = i + 1

# + [markdown] _uuid="7accd0a14f6f94b5f8f87c28e772fd7e70f5aa89"
# <br>
# Use `+` to concatenate lists.

# + _uuid="2e2d33c027d2d2e66444ea8c5bc16ac182604699"
[1,2] + [3,4]

# + [markdown] _uuid="8dac836d3bfd72d4ba18e0ad35e918fd5578e0c4"
# <br>
# Use `*` to repeat lists.

# + _uuid="33fbab44facf2994b9a1f4f842b5b68db790c60c"
[1]*3

# + [markdown] _uuid="1bf6aa261fae9ee56cc8e2ec7ba789818619bfc7"
# <br>
# Use the `in` operator to check if something is inside a list.

# + _uuid="9dcaf39b71a922a3a61f6565cb9025fe89291cc8"
1 in [1, 2, 3]

# + [markdown] _uuid="2fdc7e3a4da86bf2fc3daef510d521ff4f9e780a"
# <br>
# Now let's look at strings. Use bracket notation to slice a string.
#
# ###### [Go to top](#top)

# + _uuid="31a527ad3d4ac52203a81f2aa05d6ccbcbefd7d0"
x = 'This is a string'
print(x[0]) #first character
print(x[0:1]) #first character, but we have explicitly set the end character
print(x[0:2]) #first two characters


# + [markdown] _uuid="a2da025858712f120a1708fa8de0da26c6b8b1de"
# <br>
# This will return the last element of the string.

# + _uuid="24cb16eaabaa7e13b0a4a69dfc6cdf71ffb0238b"
x[-1]

# + [markdown] _uuid="892cb74f3b1ba32053e1b854681e9520d9ae7578"
# <br>
# This will return the slice starting from the 4th element from the end and stopping before the 2nd element from the end.

# + _uuid="b947154182b9eb7aa08f5986eb768020e9eefdcb"
x[-4:-2]

# + [markdown] _uuid="7d88463b1e8c9555030af32e1e99cdd522c40610"
# <br>
# This is a slice from the beginning of the string and stopping before the 3rd element.

# + _uuid="961b4829e231286702b78c13f6ce68c780df7c56"
x[:3]

# + [markdown] _uuid="d365c72e42116bbd6c7ae960d77af0d89d35f201"
# <br>
# And this is a slice starting from the 3rd element of the string and going all the way to the end.

# + _uuid="82c83cb0bfe09b269037a97ef0b00cd66254ab89"
x[3:]

# + _uuid="b3914291465c80995a05d30f2842f38b1e2d7255"
firstname = 'MJ'
lastname = 'Bahmani'

print(firstname + ' ' + lastname)
print(firstname*3)
print('mj' in firstname)


# + [markdown] _uuid="5e92142f11b79daaa369a0a031bab1979602457e"
# <br>
# `split` returns a list of all the words in a string, or a list split on a specific character.

# + _uuid="63a7aea0881d73a1c2caab2b2bc0bf09d46e0402"
firstname = 'Mr Dr M<NAME>'.split(' ')[0] # [0] selects the first element of the list
lastname = 'Mr Dr M<NAME>'.split(' ')[-1] # [-1] selects the last element of the list
print(firstname)
print(lastname)

# + _uuid="a46edc198922ad4957bcdc3d990f43aaae35a464"
'MJ' + str(2)

# + [markdown] _uuid="50a1372d759bcc6e0b4d8852588ba9fe1c3a1200"
# <br>
# Dictionaries associate keys with values.

# + _uuid="b6dd088e810aca788fc74f76d23777b2115e12d0"
x = {'<NAME>': '<EMAIL>', 'irmatlab': '<EMAIL>'}
x['<NAME>'] # Retrieve a value by using the indexing operator


# + _uuid="6ec5106de063d06587a532835173cb3e15c4c074"
x['<NAME>'] = None
x['<NAME>']

# + [markdown] _uuid="909deaebc50f94808fbdbe831ffbe1137066a2fe"
# <br>
# Iterate over all of the keys:

# + _uuid="993e0c88238de00475ddd22a2f4594ce14aa936d"
for name in x:
    print(x[name])

# + [markdown] _uuid="9248f6bfe90a9c867ca248b08ad81c8626eaf689"
# <br>
# Iterate over all of the values:

# + _uuid="9f995195b06da4b3e64cabbe950d19570fbb910c"
for email in x.values():
    print(email)

# + [markdown] _uuid="147f360dd1e8f878ae393c46648ec5f630d8d80c"
# <br>
# Iterate over all of the items in the list:

# + _uuid="ab973ccad15b9f3af27e6a2cbd2425ede962be86"
for name, email in x.items():
    print(name)
    print(email)

# + [markdown] _uuid="3a909b20788d5df862adec16c2d6d15e021a74dc"
# <br>
# You can unpack a sequence into different variables:

# + _uuid="296600127a5b164c23ad4aad0e622883f6ce56b1"
x = ('MJ', 'Bahmani', '<EMAIL>')
fname, lname, email = x

# + _uuid="b1d783512fd2e8d76b8d1c147ed51f8584d203b7"
fname

# + _uuid="a69c79b2a267725470ff843205f11c8a6e2a820e"
lname

# + [markdown] _uuid="6b069e526dd64ce6efba8cb51854ef2636e308b9"
# <a id="5"></a> <br>
# # 2-5 Python: More on Strings
#
# ###### [Go to top](#top)

# + _uuid="2aa934d05afb73cd7cdcb9ff223b2014798f3b18"
print('MJ' + str(2))

# + [markdown] _uuid="2a9b5f81e7ae92846e4bce5c8c34ff23490b83ef"
# <br>
# Python has a built in method for convenient string formatting.

# + _uuid="a8fa5e6a70f8a2e642f0799b514d96f5e8b4cf67"
sales_record = {
'price': 3.24,
'num_items': 4,
'person': 'MJ'}

sales_statement = '{} bought {} item(s) at a price of {} each for a total of {}'

print(sales_statement.format(sales_record['person'],
                             sales_record['num_items'],
                             sales_record['price'],
                             sales_record['num_items']*sales_record['price']))


# + [markdown] _uuid="7fee00e11e78abee9a0397711ca813c60b1b8c57"
# <a id="6"></a> <br>
# # 2-6 Python:Reading and Writing CSV files

# + [markdown] _uuid="3f79954fc298a34c3c231d35161dfb3bf743576d"
# <br>
# Let's import our datafile train.csv 
# ###### [Go to top](#top)

# + _uuid="ab5ab332ba1dc0fb1668d18df1b1771232cc4e35"


with open('train.csv') as csvfile:
    train = list(csv.DictReader(csvfile))
    
train[:1] # The first three dictionaries in our list.

# + [markdown] _uuid="ac3ffa35fd8725c3b1d81ab92470f1e5de19061b"
# <br>
# `csv.Dictreader` has read in each row of our csv file as a dictionary. `len` shows that our list is comprised of 234 dictionaries.

# + _uuid="3f51c4e51000c69c3b3113f4cc8ed1e5e5fa005f"
len(train)

# + [markdown] _uuid="59457b14190fe59e900f644450b66f542b2320c4"
# <br>
# `keys` gives us the column names of our csv.

# + _uuid="de6d753e74e0345bcc292cb6cfded49241b696f8"
train[0].keys()

# + [markdown] _uuid="391e4c865b352e70ae7bc5cbdef9559ccf396cb8"
# <br>
# How to do some math action on the data set

# + _uuid="c654cdc458ff6f8d960caa80268f61e9f213e9e8"
sum(float(d['Fare']) for d in train) / len(train)

# + [markdown] _uuid="a12ec3e5c23e0660c9cfce427333afabe5a4384c"
# <br>
# Use `set` to return the unique values for the type of Sex  in our dataset have.

# + _uuid="3541bdc6bd01a88c58e408bcc99411136a953538"
Sex = set(d['Sex'] for d in train)
Sex

# + [markdown] _uuid="51bd65ce547fceba323660a1370a407d004e9abe"
# <a id="7"></a> <br>
# # 2-7 Python: Dates and Times

# + _uuid="77aea246af2c06fe57b72bf45e424a4a6ae2e8c5"
import datetime as dt
import time as tm

# + [markdown] _uuid="f14fbbf5385512cc6d6631677a62ff620d51f9ae"
# <br>
# `time` returns the current time in seconds since the Epoch. (January 1st, 1970)
#
# ###### [Go to top](#top)

# + _uuid="73ffa7423c31ed5efb8bab44ead79a4162711920"
tm.time()

# + [markdown] _uuid="83a2aceba5919534adf74e59c16baa61d6e24bda"
# <br>
# Convert the timestamp to datetime.

# + _uuid="b0726eaf4ebb492d3da2396520a7328bbb00263c"
dtnow = dt.datetime.fromtimestamp(tm.time())
dtnow

# + [markdown] _uuid="df2a8c0e8237c6c88ea5d8f446604d6c60435b72"
# <br>
# Handy datetime attributes:

# + _uuid="3ca2dd821f01874e23fc1eccf13b6584ba21e795"
dtnow.year, dtnow.month, dtnow.day, dtnow.hour, dtnow.minute, dtnow.second # get year, month, day, etc.from a datetime

# + [markdown] _uuid="85160de1a52833048050277d7df6713172b367c9"
# <br>
# `timedelta` is a duration expressing the difference between two dates.

# + _uuid="c466ccf8a9f33641fec51525f0e8a03dbec8fd62"
delta = dt.timedelta(days = 100) # create a timedelta of 100 days
delta

# + [markdown] _uuid="c09e3bd1c17b5a67b8e340956fdb27b2e6b32b2d"
# <br>
# `date.today` returns the current local date.

# + _uuid="539408df43c0aa1b449551e14b84b49110b04beb"
today = dt.date.today()

# + _uuid="4022bf4458c6495ab1c4ddfcf82c155742641823"
today - delta # the date 100 days ago

# + _uuid="38399dc2975b4f44e083ae319f37217286d9533a"
today > today-delta # compare dates


# + [markdown] _uuid="a8bcac1b08550c3f143e21cf7ca9049449304a5c"
# <a id="8"></a> <br>
# # 2-8 Python: Objects and map()

# + [markdown] _uuid="28be3905d37e6b7d2fb09a5f1a0e0c9df7e48559"
# <br>
# An example of a class in python:

# + _uuid="2dc3faf4892b932cb1015902e83a13bd47125b20"
class Person:
    department = 'School of Information' #a class variable

    def set_name(self, new_name): #a method
        self.name = new_name
    def set_location(self, new_location):
        self.location = new_location


# + _uuid="1da9aaf6d97f50fff89e9d08b658adb3c691a262"
person = Person()
person.set_name('<NAME>')
person.set_location('MI, Berlin, Germany')
print('{} live in {} and works in the department {}'.format(person.name, person.location, person.department))

# + [markdown] _uuid="8422c57273a71b4a74b6b4a9eb5535444356823d"
# <br>
# Here's an example of mapping the `min` function between two lists.

# + _uuid="3052442e823ee0a450e404607e44cb3c321b508c"
store1 = [10.00, 11.00, 12.34, 2.34]
store2 = [9.00, 11.10, 12.34, 2.01]
cheapest = map(min, store1, store2)
cheapest

# + [markdown] _uuid="787b960d3c824c764a0eb7c2dab69d073eacfa9d"
# <br>
# Now let's iterate through the map object to see the values.

# + _uuid="a9d6751e7c95241dbd10a44f0179a40df8ee3a7a"
for item in cheapest:
    print(item)

# + [markdown] _uuid="e9c6c70988cd1459c7078f31fb9a43a275965d46"
# <a id="9"></a> <br>
# # 2-9-Python : Lambda and List Comprehensions

# + [markdown] _uuid="b6257386671f84333b4cfae3a979a2c1f8100e76"
# <br>
# Here's an example of lambda that takes in three parameters and adds the first two.

# + _uuid="a306ae5687e20ae64431306035359f33aa465382"
my_function = lambda a, b, c : a + b

# + _uuid="6bf98de2fab3b4239e41a093386d5fb9acad6367"
my_function(1, 2, 3)

# + [markdown] _uuid="5db80d702a02a1873aa09fa3acd26d922d74ff46"
# <br>
# Let's iterate from 0 to 9 and return the even numbers.
#
# ###### [Go to top](#top)

# + _uuid="90e7eb5e7689d6ce448b8c0c801c2caef70e6d44"
my_list = []
for number in range(0, 9):
    if number % 2 == 0:
        my_list.append(number)
my_list

# + [markdown] _uuid="1bfe68a8553e5718a04b14a321262e6e88407b2c"
# <br>
# Now the same thing but with list comprehension.

# + _uuid="ee501b9820296d1452845b8da201dbd69d4e0637"
my_list = [number for number in range(0,10) if number % 2 == 0]
my_list


# + [markdown] _uuid="e99f98ecb3393b8846fca621cdbad5dcaf1c60d1"
# <a id="10"></a> <br>
# # 2-10 OOP
# 1. **Class** − A user-defined prototype for an object that defines a set of attributes that characterize any object of the class. The attributes are data members (class variables and instance variables) and methods, accessed via dot notation.
#
# 1. **Class variable** − A variable that is shared by all instances of a class. Class variables are defined within a class but outside any of the class's methods. Class variables are not used as frequently as instance variables are.
#
# 1. **Data member** − A class variable or instance variable that holds data associated with a class and its objects.
#
# 1. **Function overloading** − The assignment of more than one behavior to a particular function. The operation performed varies by the types of objects or arguments involved.
#
# 1. **Instance variable** − A variable that is defined inside a method and belongs only to the current instance of a class.
#
# 1. **Inheritance** − The transfer of the characteristics of a class to other classes that are derived from it.
#
# 1. **Instance** − An individual object of a certain class. An object obj that belongs to a class Circle, for example, is an instance of the class Circle.
#
# 1. **Instantiation** − The creation of an instance of a class.
#
# 1. **Method** − A special kind of function that is defined in a class definition.
#
# 1. **Object** − A unique instance of a data structure that's defined by its class. An object comprises both data members (class variables and instance variables) and methods.
#
# 1. **Operator overloading** − The assignment of more than one function to a particular operator.[4]
#
# ###### [Go to top](#top)

# + _uuid="1c3c77a74a2f45e0cfbec9e2a43ac089aaa09558"
class FirstClass:
    test = 'test'
    def __init__(self,name,symbol):
        self.name = name
        self.symbol = symbol


# + _uuid="2dd1233bc33a4e9ca7bf528968267868f4d1bbe1"
eg3 = FirstClass('Three',3)

# + _uuid="8bb856d47cf1407cccd295fe241825c2999e0756"
print (eg3.test, eg3.name)


# + _uuid="3fa8ea01163ceea3f5cf46fe2f5eb843fa8a9cc7"
class FirstClass:
    def __init__(self,name,symbol):
        self.name = name
        self.symbol = symbol
    def square(self):
        return self.symbol * self.symbol
    def cube(self):
        return self.symbol * self.symbol * self.symbol
    def multiply(self, x):
        return self.symbol * x


# + _uuid="6086b2eaf54cc57b3447b7c0b47e4b67f208cba3"
eg4 = FirstClass('Five',5)

# + _uuid="7029000f20ae1efcd694754d0db4b7e546cdafae"
print (eg4.square())
print (eg4.cube())

# + _uuid="e60cf1e71ae99cb38e6b555ae43cb6948bdfc329"
eg4.multiply(2)

# + _uuid="52ccbcd46b12d802454097d1a4c6ee52c551c4a3"
FirstClass.multiply(eg4,2)


# + [markdown] _uuid="c68d1914adc7080d3ee9b5592420ff95ea2870e7"
# <a id="11"></a> <br>
# ### 2-10-1 Inheritance
#
# There might be cases where a new class would have all the previous characteristics of an already defined class. So the new class can "inherit" the previous class and add it's own methods to it. This is called as inheritance.
#
# Consider class SoftwareEngineer which has a method salary.

# + _uuid="ea270e5c33b3e6d6fb3c7b66a0081e8d80825a78"
class SoftwareEngineer:
    def __init__(self,name,age):
        self.name = name
        self.age = age
    def salary(self, value):
        self.money = value
        print (self.name,"earns",self.money)


# + _uuid="d63b58d87a62e2e692f25644b1bfd1b0c3c2ccff"
a = SoftwareEngineer('Kartik',26)

# + _uuid="14e20781ff108a4cc13ac68de99478293413b3d1"
a.salary(40000)

# + _uuid="89cf62c093a3c6369b77536f4700f70ddf84eebf"
dir(SoftwareEngineer)


# + _uuid="edcf9e2d90a66ca3370d889ffd2109827229825f"
class Artist:
    def __init__(self,name,age):
        self.name = name
        self.age = age
    def money(self,value):
        self.money = value
        print (self.name,"earns",self.money)
    def artform(self, job):
        self.job = job
        print (self.name,"is a", self.job)


# + _uuid="92e459938effe069d173a19f94819e68b34d4eca"
b = Artist('Nitin',20)

# + _uuid="a708a468bec5bc8314bc40ad7e830568805885bc"
b.money(50000)
b.artform('Musician')

# + _uuid="efa5a80085b1a7a6e98a88c5fd86da6c880421cf"
dir(Artist)

# + [markdown] _uuid="843c1001213bd2cfd201e99b682a5c5bfe78c0c6"
# <a id="12"></a> <br>
# ## 2-11 Python JSON
#

# + _uuid="153a10fa7584e50553490e6f765284192776ee5c"


# some JSON:
x =  '{ "name":"John", "age":30, "city":"New York"}'

# parse x:
y = json.loads(x)

# the result is a Python dictionary:
print(y["age"])

# + [markdown] _uuid="85876511f2a41d866fe4bfa5208372f51536e6c5"
# <a id="13"></a> <br>
# ## 2-11-1 Convert from Python to JSON
#

# + _uuid="06ce583a101287b07e6e95d28426ae20d5b4e64b"
# a Python object (dict):
x = {
  "name": "John",
  "age": 30,
  "city": "New York"
}

# convert into JSON:
y = json.dumps(x)

# the result is a JSON string:
print(y)

# + [markdown] _uuid="b016e08d500dc749010ed6495e3c02e3f1144b4e"
# You can convert Python objects of the following types, into JSON strings:
#
# * dict
# * list
# * tuple
# * string
# * int
# * float
# * True
# * False
# * None
#
# ###### [Go to top](#top)

# + _uuid="2906f3ee8c9392b9be7507c2c99a1fc3db9df95d"
print(json.dumps({"name": "John", "age": 30}))
print(json.dumps(["apple", "bananas"]))
print(json.dumps(("apple", "bananas")))
print(json.dumps("hello"))
print(json.dumps(42))
print(json.dumps(31.76))
print(json.dumps(True))
print(json.dumps(False))
print(json.dumps(None))

# + [markdown] _uuid="5cd4ddf30b6996185afdc94f68ad675b258ebb9a"
# Convert a Python object containing all the legal data types:

# + _uuid="7a5a584c5912ac52a6fc9796e23eb4799d5b9261"
x = {
  "name": "John",
  "age": 30,
  "married": True,
  "divorced": False,
  "children": ("Ann","Billy"),
  "pets": None,
  "cars": [
    {"model": "BMW 230", "mpg": 27.5},
    {"model": "Ford Edge", "mpg": 24.1}
  ]
}

print(json.dumps(x))

# + [markdown] _uuid="27a8da102cf08964e9cde82973cd653306e3356f"
# <a id="14"></a> <br>
# ## 2-12 Python PIP
#

# + [markdown] _uuid="5d13a95f504b710e977b4b589c200d5dadca5bc2"
# <a id="15"></a> <br>
# ### 2-12-1 What is a Package?
# A package contains all the files you need for a module.
#
# Modules are Python code libraries you can include in your project.
#
# ###### [Go to top](#top)

# + [markdown] _uuid="cfe1ddbfe7ac5b38f1fb0d36401aaa59bf706b06"
# <a id="16"></a> <br>
# ### 2-12-2 Install PIP
# If you do not have PIP installed, you can download and install it from this page: https://pypi.org/project/pip/

# + [markdown] _uuid="af880b807b69c45d77b21433937a13772453b481"
# <a id="17"></a> <br>
# ## 2-13 Python Try Except
# The **try** block lets you test a block of code for errors.
#
# The **except** block lets you handle the error.
#
# The **finally** block lets you execute code, regardless of the result of the try- and except blocks.

# + _uuid="078d0a98c43a742ff43460848dd6bf31a69975ce"
try:
  print(x)
except NameError:
  print("Variable x is not defined")
except:
  print("Something else went wrong")

# + _uuid="c5e3cbfe7a287cda43736597455b95e2117ccf4d"
try:
  print(x)
except:
  print("Something went wrong")
finally:
  print("The 'try except' is finished")

# + [markdown] _uuid="6d4ae311b6798fa03dff0516dfb59e7d271b040e"
# <a id="18"></a> <br>
# ## 2-14 Python Iterators
# An iterator is an object that contains a countable number of values.
#
# An iterator is an object that can be iterated upon, meaning that you can traverse through all the values.
#
# Technically, in Python, an iterator is an object which implements the iterator protocol, which consist of the methods __iter__() and __next__().
# ###### [Go to top](#top)

# + [markdown] _uuid="0fd29f43681c38533591b6cd0d3cff27e75b6b0c"
# Return a iterator from a tuple, and print each value:

# + _uuid="d4f1c3ecdad754158b847660bfc92343470e9c55"
mytuple = ("apple", "banana", "cherry")
myit = iter(mytuple)

print(next(myit))
print(next(myit))
print(next(myit))

# + [markdown] _uuid="5a15f75b705e078ec25f1fb628af76ed19da0333"
# <a id="19"></a> <br>
# ### 2- 14-1 Looping Through an Iterator
#

# + _uuid="830a5d61f4d7afec6525f32b8f8e9c0f4783acc0"
mytuple = ("apple", "banana", "cherry")

for x in mytuple:
  print(x)

# + [markdown] _uuid="e7410e1e9691615946998a833fec7bea80f50cc2"
# <a id="20"></a> <br>
# ## 2- 15 Dictionary
# A **dictionary** is a collection which is **unordered, changeable and indexed**. In Python dictionaries are written with curly brackets, and they have **keys and values**.

# + _uuid="85f51ded66c72e13b253c5775e5b1ac914459479"
thisdict =	{
  "brand": "Ford",
  "model": "Mustang",
  "year": 1964
}
print(thisdict)

# + [markdown] _uuid="437f23fd4289f0a757dc1c87a1e339a9c6f96c31"
# <a id="21"></a> <br>
# ## 2-16 Tuples
# A **tuple** is a collection which is **ordered and unchangeable**. In Python tuples are written with round brackets.
#
#

# + _uuid="253ab9bcc46ce48b2f5b702bacad75db5d95de7c"
thistuple = ("apple", "banana", "cherry")
print(thistuple)

# + [markdown] _uuid="0d377a64581e95a0022045181a2c81530eba31d1"
# <a id="22"></a> <br>
# ## 2-19 Set
# A set is a collection which is unordered and unindexed. In Python sets are written with curly brackets.
# ###### [Go to top](#top)

# + _uuid="0d4b6b6a5efa453ed3a594173586fb63bb6475dc"
thisset = {"apple", "banana", "cherry"}
print(thisset)

# + _uuid="dbb1998040fc3bdc67e0d690de391b4f47e117b8"
thisset = {"apple", "banana", "cherry"}

for x in thisset:
  print(x)

# + [markdown] _uuid="2f51152b3812f1d279b3a009fed112780aedbda4"
# <a id="23"></a> <br>
# ### 2-17-1 Add Items
# To add one item to a set use the add() method.
#
# To add more than one item to a set use the update() method.
# ###### [Go to top](#top)

# + _uuid="cccd895e65e0b5bbbfbecdd96b3322a43688de78"
thisset = {"apple", "banana", "cherry"}

thisset.add("orange")

print(thisset)

# + [markdown] _uuid="1a8697f93952e076f6f949997676d40518d7b5a6"
# <a id="11"></a> <br>
# # Python Packages
# * Numpy
# * Pandas
# * Matplotlib
# * Seaborn
# * Sklearn
# * plotly

# + [markdown] _uuid="bfb701e45e93aea0b3ed64e148ca2fdb53559038"
# <a id="24"></a> <br>
# # 3- Numerical Python (NumPy)

# + _uuid="db9a850ebb440ca960a0713d822e20090bc10601"
import numpy as np

# + [markdown] _uuid="79699175b4559f509181d359393167f801735485"
# <a id="25"></a> <br>
# ## 3-1 NumPy :Creating Arrays

# + [markdown] _uuid="fb5123cfa4687a819758ea82810984fa69d631e3"
# Create a list and convert it to a numpy array

# + _uuid="1cdc9404b31261269891723ddc59064802063041"
mylist = [1, 2, 3]
x = np.array(mylist)
x

# + [markdown] _uuid="85123a6bf0589918ff03fe9916b06635fa32b776"
# <br>
# Or just pass in a list directly

# + _uuid="dca7dd9319716e863760bb7c4e1e47a1c17d7b1b"
y = np.array([4, 5, 6])
y

# + [markdown] _uuid="5a29d1e6bb19131b3bde9ae197b562cf5c905f2a"
# <br>
# Pass in a list of lists to create a multidimensional array.

# + _uuid="e18e77b6d1becf1b7ded3a4daa361dbe3f985d96"
m = np.array([[7, 8, 9], [10, 11, 12]])
m

# + [markdown] _uuid="956c870e985074aadd2aa97b8a4c820c64bd7d2b"
# <br>
# Use the shape method to find the dimensions of the array. (rows, columns).

# + _uuid="7e5472233313b1eb806b7e9dfd2478f4155b23d0"
m.shape

# + [markdown] _uuid="d20433435f865bf4593ee8b5eae4cb2173794a57"
# <br>
# `arange` returns evenly spaced values within a given interval.

# + _uuid="87aded8cbe232bfcb4ce246ecf1bac0b70d9477e"
n = np.arange(0, 30, 2) # start at 0 count up by 2, stop before 30
n

# + [markdown] _uuid="ef6d0651972cb576188cd986830aae2d53bb20b8"
# <br>
# `reshape` returns an array with the same data with a new shape.

# + _uuid="6722026f830d545bfd21a2760f80c20e33c0d757"
n = n.reshape(3, 5) # reshape array to be 3x5
n

# + [markdown] _uuid="a47a0e9b094d81ae9ec235cd33d395ca34c379b3"
# <br>
# `linspace` returns evenly spaced numbers over a specified interval.

# + _uuid="099d0b4b7f3c6e5aef6cebd760e3c6b8111205d8"
o = np.linspace(0, 4, 9) # return 9 evenly spaced values from 0 to 4
o

# + [markdown] _uuid="e5abb93fd66c344624ad2bb5e91ec1cd8a9ab220"
# <br>
# `resize` changes the shape and size of array in-place.

# + _uuid="66de4ad6fbb8a2ffe7251b667de732fe0048c39c"
o.resize(3, 3)
o

# + [markdown] _uuid="95af33fa252b8ccc4afad8666e0a7ced2d83aee9"
# <br>
# `ones` returns a new array of given shape and type, filled with ones.

# + _uuid="015e5a604fceb5b335760d85f090812416ab0edd"
np.ones((3, 2))

# + [markdown] _uuid="415ba7eb6ffcca82d1ad73b6e652d77c2b40c4c7"
# <br>
# `zeros` returns a new array of given shape and type, filled with zeros.

# + _uuid="4833b4502eac60ec53ca4ab55f10653103be3584"
np.zeros((2, 3))

# + [markdown] _uuid="acbef4f6ab93b0ea5ed8bea30ec0eddbce8415f1"
# <br>
# `eye` returns a 2-D array with ones on the diagonal and zeros elsewhere.

# + _uuid="5e4e8112e5829b290385bd076041d9d0c43bdd4e"
np.eye(3)

# + [markdown] _uuid="bb258a18b84c56c26c19e5e467f8886a3209eb6d"
# <br>
# `diag` extracts a diagonal or constructs a diagonal array.

# + _uuid="0167bbb06dc5175a14e2d1844b3e92c3f601e156"
np.diag(y)

# + [markdown] _uuid="05f0ea0d69c70181a0ddee0b0a0670afc1a5761c"
# <br>
# Create an array using repeating list (or see `np.tile`)

# + _uuid="76f7c4dd56d1f9ba9c8dcc3bf3b0e67bc62326de"
np.array([1, 2, 3] * 3)

# + [markdown] _uuid="e9d6935951d0df2a4d60f66b302da7b876006c71"
# <br>
# Repeat elements of an array using `repeat`.

# + _uuid="7299b4990895d4ea8a59447489fa6fbc0cde5ea6"
np.repeat([1, 2, 3], 3)

# + [markdown] _uuid="01250c2ea726f387db5f3fab5a004e741e574a35"
# <a id="26"></a> <br>
# ## 3-2 Numpy:Combining Arrays
#
# ###### [Go to top](#top)

# + _uuid="0dc57919256360d1ac309813fb6e836f75d17484"
p = np.ones([2, 3], int)
p

# + [markdown] _uuid="98aa86e98478bcad2ce35eded8e2adb825bbf709"
# <br>
# Use `vstack` to stack arrays in sequence vertically (row wise).

# + _uuid="48b0f8194df6a4f63932d97b24883d6a5d69d0df"
np.vstack([p, 2*p])

# + [markdown] _uuid="a2cf2eb6fdf3deccca31df9979dd978241adadb8"
# <br>
# Use `hstack` to stack arrays in sequence horizontally (column wise).

# + _uuid="834984b63056d7161f261d868d85b83f24403287"
np.hstack([p, 2*p])

# + [markdown] _uuid="b82f24eec48ab5206a47d04a36e5e639dad7f9a1"
# <a id="27"></a> <br>
# ## 3-3 Numpy:Operations
#
# ###### [Go to top](#top)

# + [markdown] _uuid="d58a4bb35b9f1525736b28ffc7e2b6ead4035266"
# Use `+`, `-`, `*`, `/` and `**` to perform element wise addition, subtraction, multiplication, division and power.

# + _uuid="7ba8b42fcdcdc83e6f43df83b73665aa289e9786"
print(x + y) # elementwise addition     [1 2 3] + [4 5 6] = [5  7  9]
print(x - y) # elementwise subtraction  [1 2 3] - [4 5 6] = [-3 -3 -3]

# + _uuid="cd42ade6bd206750b5dde1086b06bde519f2b1df"
print(x * y) # elementwise multiplication  [1 2 3] * [4 5 6] = [4  10  18]
print(x / y) # elementwise divison         [1 2 3] / [4 5 6] = [0.25  0.4  0.5]

# + _uuid="3370d5b9b8ca04ed5be48331c3d6d4d08738ab5d"
print(x**2) # elementwise power  [1 2 3] ^2 =  [1 4 9]

# + [markdown] _uuid="e665d9db501694e29f54b02de56befaf69305629"
# <br>
# **Dot Product:**  
#
# $ \begin{bmatrix}x_1 \ x_2 \ x_3\end{bmatrix}
# \cdot
# \begin{bmatrix}y_1 \\ y_2 \\ y_3\end{bmatrix}
# = x_1 y_1 + x_2 y_2 + x_3 y_3$

# + _uuid="b45a94567cf5801e05869bde614e831603b1599a"
x.dot(y) # dot product  1*4 + 2*5 + 3*6

# + _uuid="b27ce9c3b1328461ba14f5efdd36079e3f827951"
z = np.array([y, y**2])
print(len(z)) # number of rows of array

# + [markdown] _uuid="30ab350aa70687cbdd087d86386b96504aec4479"
# <br>
# Let's look at transposing arrays. Transposing permutes the dimensions of the array.

# + _uuid="2630bf81db6d00483f7e0abd3a3c3af28144d55f"
z = np.array([y, y**2])
z

# + [markdown] _uuid="8bb90454186cab0c688be9a01ff84c7ca67fa6a9"
# <br>
# The shape of array `z` is `(2,3)` before transposing.

# + _uuid="bdcbfc2e1c9b985a83aefa1a5972aa919acb8365"
z.shape

# + [markdown] _uuid="3b796b3d6c4e9a90e4fb5332d708398bb848c2e4"
# <br>
# Use `.T` to get the transpose.

# + _uuid="8ca46bf46ed15a5fa3af7e2d325ac68bc8785f05"
z.T

# + [markdown] _uuid="d2949665fb40cdb932376219b6d78156265d5ebf"
# <br>
# The number of rows has swapped with the number of columns.

# + _uuid="e48e8eff60259a9c865cf2dae88a3ce1641e826c"
z.T.shape

# + [markdown] _uuid="08fac1d78a66fdd513372cd31eaa5f973655e80a"
# <br>
# Use `.dtype` to see the data type of the elements in the array.

# + _uuid="6e1ea214c877f23d670b72f06060c2dee1bdaee5"
z.dtype

# + [markdown] _uuid="70da3bcabbc0631bb11b48796106c07f4de41ba8"
# <br>
# Use `.astype` to cast to a specific type.

# + _uuid="60684ed5f18c88c4ef4e2fe4207fa94eede9d993"
z = z.astype('f')
z.dtype

# + [markdown] _uuid="e1199442dd07a4a4eef965c584d9e0f443c40013"
# <a id="28"></a> <br>
# ## 3-4 Numpy: Math Functions
#
# ###### [Go to top](#top)

# + [markdown] _uuid="01121300b7b5b8a83d213d4f065383da19d5f7d8"
# Numpy has many built in math functions that can be performed on arrays.

# + _uuid="e1eaeb06cf68d055f6a5536ea72a17606b6762c1"
a = np.array([-4, -2, 1, 3, 5])

# + _uuid="2918e83be55935fa03fd24924bc7f07a271c40d3"
a.sum()

# + _uuid="2e86a6ea98c6c86dbe7a96612af4f37452b23670"
a.max()

# + _uuid="1e8e48425f65e90a18a29fb7983dc6aa424f8445"
a.min()

# + _uuid="27131855b2c26a5f90c6524d4a8c7ed0266bf378"
a.mean()

# + _uuid="a020531df7f87af15577e986699e80f892300773"
a.std()

# + [markdown] _uuid="6d26c78ddd00e9387ba4214470debcc8147fd2bc"
# <br>
# `argmax` and `argmin` return the index of the maximum and minimum values in the array.

# + _uuid="dd6f6aee91fc8dd99cc5a8359dc81ce1d443e77f"
a.argmax()

# + _uuid="59aeec1ba92a0eb4c6928564e64445e6ce46cc3c"
a.argmin()

# + [markdown] _uuid="cf3ad4800506fd903882f60d811ca2548756e7c8"
# <a id="29"></a> <br>
#
# ## 3-5 Numpy:Indexing / Slicing
# ###### [Go to top](#top)

# + _uuid="82ca4b616a46de3280b5a50df4c0114298d07aea"
s = np.arange(13)**2
s

# + [markdown] _uuid="417c1b56b9d3eca21021422bde199036c3f08ec3"
# <br>
# Use bracket notation to get the value at a specific index. Remember that indexing starts at 0.

# + _uuid="6f90f5dee19a73afde4e9455b47c0dbe86c9ce6b"
s[0], s[4], s[-1]

# + [markdown] _uuid="9d0cf8a30f9b1bd1f8dc60a46d3271e5cac235d0"
# <br>
# Use `:` to indicate a range. `array[start:stop]`
#
#
# Leaving `start` or `stop` empty will default to the beginning/end of the array.

# + _uuid="1ea11cc5be369a751b250e7085cf507b122bcf88"
s[1:5]

# + [markdown] _uuid="dd42e9a4274baebf747c627767a273ef8ca9a26f"
# <br>
# Use negatives to count from the back.

# + _uuid="ce51a60f59516b22174f97a4f7c6da6c75322a8b"
s[-4:]

# + [markdown] _uuid="0fdd94c22e7ceb4cac00001bd98f09d5d879613e"
# <br>
# A second `:` can be used to indicate step-size. `array[start:stop:stepsize]`
#
# Here we are starting 5th element from the end, and counting backwards by 2 until the beginning of the array is reached.

# + _uuid="f28245bea6ab9a1bed7a76859ddfbb295ea95038"
s[-5::-2]

# + [markdown] _uuid="2f2295df96d9e7da01cde82bf87d5c7157bd9d21"
# <br>
# Let's look at a multidimensional array.

# + _uuid="582e15f695a2891826ef4eb8e913a0dc61913898"
r = np.arange(36)
r.resize((6, 6))
r

# + [markdown] _uuid="26fd89ebfe4b92cedad89152753918608b68eafe"
# <br>
# Use bracket notation to slice: `array[row, column]`.

# + _uuid="7bf0f1e77c5243c3e2b846f7b268c4891786621b"
r[2, 2]

# + [markdown] _uuid="fcbf322288c6671ef799299b5ddb828b68b8cab9"
# <br>
# And use : to select a range of rows or columns.

# + _uuid="64d430ac4626977723c6357865a5297699e5323d"
r[3, 3:6]

# + [markdown] _uuid="e109144f7316b357450c4b3cae3cb61ad327f5d9"
# <br>
# Here we are selecting all the rows up to (and not including) row 2, and all the columns up to (and not including) the last column.

# + _uuid="7b80dc3ba5c970afe2db37c23599d476399b1c68"
r[:2, :-1]

# + [markdown] _uuid="6c5126aa54a1e6b9d0d078f2226b68d3fb42c4a2"
# <br>
# This is a slice of the last row, and only every other element.

# + _uuid="054f80d47b0e2d249a9140321846ff7b50daba1f"
r[-1, ::2]

# + [markdown] _uuid="62e5354b353e1fe2e60036f2c3f94909a8af1b7b"
# <br>
# We can also perform conditional indexing. Here we are selecting values from the array that are greater than 30. (Also see `np.where`)

# + _uuid="5476c43a5e7d195804cac7517abbdbcdbc234829"
r[r > 30]

# + [markdown] _uuid="77ae22731783157aaecdd019be77004504b943af"
# <br>
# Here we are assigning all values in the array that are greater than 30 to the value of 30.
# ###### [Go to top](#top)

# + _uuid="6651ea99641526b7df83e7a9d9c1efe4eab9dc99"
r[r > 30] = 30
r

# + [markdown] _uuid="6189d2e7dc9038dc1c4f2ed95a47fad76d700890"
# <a id="30"></a> <br>
# ## 3-6 Numpy :Copying Data

# + [markdown] _uuid="ad2453d83a8db6de311e9265672e9b10b0337284"
# Be careful with copying and modifying arrays in NumPy!
#
#
# `r2` is a slice of `r`

# + _uuid="e6bb5dab7586dd4cfbc93b2cee05c57aae1e8518"
r2 = r[:3,:3]
r2

# + [markdown] _uuid="65d42e0a4cade74ae9b2b587f182efaa8a4b1dbf"
# <br>
# Set this slice's values to zero ([:] selects the entire array)

# + _uuid="9026a103a457054124258eefca3492008f884ef4"
r2[:] = 0
r2

# + [markdown] _uuid="f73d4004c9181b5cffddf6f13b452d7e40cea3f9"
# <br>
# `r` has also been changed!

# + _uuid="80c0d26d5cb0374f82929e3d79c183f5e116f4ea"
r

# + [markdown] _uuid="c8a21fd94e5dfcd322ec0017e89302533ac3cf2d"
# <br>
# To avoid this, use `r.copy` to create a copy that will not affect the original array

# + _uuid="d2b2e17295f75a5a3b8dacf5ca65b4c4f3b6ca47"
r_copy = r.copy()
r_copy

# + [markdown] _uuid="2ef3568f7ac932cc45931854410cbb9e4d909df8"
# <br>
# Now when r_copy is modified, r will not be changed.

# + _uuid="4f7fa9e1b65eea1e9514fbd6f91a2720822430ba"
r_copy[:] = 10
print(r_copy, '\n')
print(r)

# + [markdown] _uuid="752b09b0f8f1606d55bf0423d3f2d9cb162d3ce9"
# <a id="31"></a> <br>
# ## 3-7 Numpy: Iterating Over Arrays

# + [markdown] _uuid="6a7f8c639e5f8cbdbd0cf2fe4c78504955ec2ccb"
# Let's create a new 4 by 3 array of random numbers 0-9.

# + _uuid="c1496dab8d8f90434e8c44e63c225bfb3ca9713f"
test = np.random.randint(0, 10, (4,3))
test

# + [markdown] _uuid="bf971c58819a069592b3d5c195cf9e21faa4797b"
# <br>
# Iterate by row:

# + _uuid="993bb7b7f1be5a0b088caac550c32257dd1c9297"
for row in test:
    print(row)

# + [markdown] _uuid="f5b7fa4289acc25093efbbc99fe36355073bcd02"
# <br>
# Iterate by index:

# + _uuid="9de357edca10cf7708e1c5b37ef5a0ad337fcbbf"
for i in range(len(test)):
    print(test[i])

# + [markdown] _uuid="00ed3013376dff60b43f6f1eb5091580279063ad"
# <br>
# Iterate by row and index:

# + _uuid="a75a7881baf87a49a3f48236fc9a9281f2ace310"
for i, row in enumerate(test):
    print('row', i, 'is', row)

# + [markdown] _uuid="08f51011e8d4da3ab72a33ccb6daca3c97832eb4"
# <br>
# Use `zip` to iterate over multiple iterables.

# + _uuid="63709cf63fc6e5596bc540055a287d56a57c55df"
test2 = test**2
test2

# + _uuid="88776121d1063744cdb0d1df15320af35a40690f"
for i, j in zip(test, test2):
    print(i,'+',j,'=',i+j)

# + [markdown] _uuid="c2ec9941ed71b0d102881252688723804c536b65"
# <a id="32"></a> <br>
# ## 3-8  Numpy: The Series Data Structure
# One-dimensional ndarray with axis labels (including time series)

# + _uuid="ff60c47c0ee85b3534fa0eeb1fc6c18951e13a93"
animals = ['Tiger', 'Bear', 'Moose']
pd.Series(animals)

# + _uuid="3b9d3593c2f04eb52439d1e2f6eaced42103b385"
numbers = [1, 2, 3]
pd.Series(numbers)

# + _uuid="ac6b145a659c5c6e143e47a726be2d2bc904ea05"
animals = ['Tiger', 'Bear', None]
pd.Series(animals)

# + _uuid="608363b045521c88d96135d4651624753a0a97f8"
numbers = [1, 2, None]
pd.Series(numbers)

# + _uuid="81c84dd1739a442c3eca83911b9e9cd146beccf1"
import numpy as np
np.nan == None

# + _uuid="ee829b3241dc2b99e7aeb8b16daa61f43516a08e"
np.nan == np.nan

# + _uuid="c39a1b7d020fa502a055e4befa580529da3d9206"
np.isnan(np.nan)

# + _uuid="703803f890a0c351d5122b4f509c0835b949481d"
sports = {'Archery': 'Bhutan',
          'Golf': 'Scotland',
          'Sumo': 'Japan',
          'Taekwondo': 'South Korea'}
s = pd.Series(sports)
s

# + _uuid="6a4a03374c688db5bc1cb49cf0ae67a166f33ab5"
s.index

# + _uuid="16a45c1d82eb06da0e940aa9304455d5b6629723"
s = pd.Series(['Tiger', 'Bear', 'Moose'], index=['India', 'America', 'Canada'])
s

# + _uuid="d5c017e5124d3c8c62bbe91b16ae5e2fb76f2cd5"
sports = {'Archery': 'Bhutan',
          'Golf': 'Scotland',
          'Sumo': 'Japan',
          'Taekwondo': 'South Korea'}
s = pd.Series(sports, index=['Golf', 'Sumo', 'Hockey'])
s

# + [markdown] _uuid="91a03b68b0698ad0f0736d0a8106bb4c2023437d"
# <a id="33"></a> <br>
# # 3-9 Numpy: Querying a Series

# + _uuid="6c44f80eadf6e60f8d1d966594169e6e579fd91a"
sports = {'Archery': 'Bhutan',
          'Golf': 'Scotland',
          'Sumo': 'Japan',
          'Taekwondo': 'South Korea'}
s = pd.Series(sports)
s

# + _uuid="560e0377aa2d53cc14f61a42a6492f99932a6eab"
s.iloc[3]

# + _uuid="2c0e34116ab3b36363b7d7ffde48d3520b493d50"
s.loc['Golf']

# + _uuid="12dce7548346ce489c21f50e02a30e4a7dad6a81"
s[3]

# + _uuid="a44e0fd0b5ea294bfde2383f3ed660bb4dc7c032"
s['Golf']

# + _uuid="97f106d843bd9da560c4aaa36e9a2baf0fd5f820"
sports = {99: 'Bhutan',
          100: 'Scotland',
          101: 'Japan',
          102: 'South Korea'}
s = pd.Series(sports)

# + _uuid="6c8d10a115a42956c1bf95a925006f1a8f44ac77"
s = pd.Series([100.00, 120.00, 101.00, 3.00])
s

# + _uuid="a877d83b4ea29606ab220884b742b9436640a87b"
total = 0
for item in s:
    total+=item
print(total)

# + _uuid="eb4aaac9d42d6ad0df5b0d766bf142cec13ca640"
total = np.sum(s)
print(total)

# + _uuid="c5ec441b291558581a79c6bda7d67ee7df640ac4"
#this creates a big series of random numbers
s = pd.Series(np.random.randint(0,1000,10000))
s.head()

# + _uuid="cafe2341bfb1b9419fcff2624e9973a12beb579a"
len(s)

# + _uuid="b0028ee6c78f4715a1848458c3b5ef2ea75e601b"
# %%timeit -n 100
summary = 0
for item in s:
    summary+=item

# + _uuid="bbc4da9b2ba74a04d797ff837bddf7553b4625d9"
# %%timeit -n 100
summary = np.sum(s)

# + _uuid="a9ef0f8d0db28ffb3f5c92c86dbcffd8b9d01840"
s+=2 #adds two to each item in s using broadcasting
s.head()

# + _uuid="f092ba84abb6a9172a23030b25caa23b7dea1c3f"
for label, value in s.iteritems():
    s.set_value(label, value+2)
s.head()

# + _uuid="979c2ea48abace1804d37a316ecf8a7cb0b53aa2"
# %%timeit -n 10
s = pd.Series(np.random.randint(0,1000,100))
for label, value in s.iteritems():
    s.loc[label]= value+2

# + _uuid="84c6c0358bf55fd26254ae784ae2b60b3d7c3526"
# %%timeit -n 10
s = pd.Series(np.random.randint(0,1000,100))
s+=2


# + _uuid="35dabb561b6b3aaf6520311c68004880cadf5f7d"
s = pd.Series([1, 2, 3])
s.loc['Animal'] = 'Bears'
s

# + _uuid="1f40d7bded3fd8cea73b8ab31e945929d75a57f4"
original_sports = pd.Series({'Archery': 'Bhutan',
                             'Golf': 'Scotland',
                             'Sumo': 'Japan',
                             'Taekwondo': 'South Korea'})
cricket_loving_countries = pd.Series(['Australia',
                                      'Barbados',
                                      'Pakistan',
                                      'England'], 
                                   index=['Cricket',
                                          'Cricket',
                                          'Cricket',
                                          'Cricket'])
all_countries = original_sports.append(cricket_loving_countries)

# + _uuid="95fa6b69a0927865a896c7a518ba848c3b994cad"
original_sports

# + _uuid="bb617195f684747e2a79b87eb7307e832f2bfe50"
cricket_loving_countries

# + _uuid="5274031be8720d734db1866f20ad048c4c2ea7da"
all_countries

# + _uuid="33c9d9f54962decaa00a7328b124252ddcf2b661"
all_countries.loc['Cricket']

# + [markdown] _uuid="7653d84bad68f6370b1bdf484a2e9b6fb5982977"
# <a id="22"></a> <br>
# ## 4- Pandas:The DataFrame Data Structure
#  You'll hone your pandas skills by learning how to organize, reshape, and aggregate multiple data sets to answer your specific questions.
#  **Pandas**:
# Two-dimensional size-mutable, potentially heterogeneous tabular data structure with labeled axes (rows and columns). Arithmetic operations align on both row and column labels. Can be thought of as a dict-like container for Series objects. The primary pandas data structure.
#
# Pandas is capable of many tasks including:
#
# Reading/writing many different data formats
# Selecting subsets of data
# Calculating across rows and down columns
# Finding and filling missing data
# Applying operations to independent groups within the data
# Reshaping data into different forms
# Combing multiple datasets together
# Advanced time-series functionality
# Visualization through matplotlib and seaborn
# Although pandas is very capable, it does not provide functionality for the entire data science pipeline. Pandas is typically the intermediate tool used for data exploration and cleaning squashed between data capturing and storage, and data modeling and predicting.
# ###### [Go to top](#top)

# + _uuid="4e82246f590c37992f9190583cdb0035d93c0dcd"

purchase_1 = pd.Series({'Name': 'Chris',
                        'Item Purchased': 'Dog Food',
                        'Cost': 22.50})
purchase_2 = pd.Series({'Name': 'Kevyn',
                        'Item Purchased': 'Kitty Litter',
                        'Cost': 2.50})
purchase_3 = pd.Series({'Name': 'Vinod',
                        'Item Purchased': 'Bird Seed',
                        'Cost': 5.00})
df = pd.DataFrame([purchase_1, purchase_2, purchase_3], index=['Store 1', 'Store 1', 'Store 2'])
df.head()

# + _uuid="835c6bb2dba772d11345bb2d5e40a992999d31b6"
df.loc['Store 2']

# + _uuid="0b48c39ad30936af6ce6932597073451c6b2bac9"
type(df.loc['Store 2'])

# + _uuid="c929565dc25461fd914c756431df400e6cdf058b"
df.loc['Store 1']

# + _uuid="a11d22702f1be476a4d443cf811fe0a07a5dbbe4"
df.loc['Store 1', 'Cost']

# + _uuid="adb35d3ac7e00c00ced7236aa7c3eaab3c85e675"
df.T

# + _uuid="0e478aabe2d50b04f5bd9cca220150476adc8b1f"
df.T.loc['Cost']

# + _uuid="2ed1c53b43bb00bc07a95642071be8ef2b5aa779"
df['Cost']

# + _uuid="084454d6fcc47a738808b0c832d59aebc49be70c"
df.loc['Store 1']['Cost']

# + _uuid="fdbddb30cb59ed8462a07795aaadbcd9ad2b1aa4"
df.loc[:,['Name', 'Cost']]

# + _uuid="2541e9071816bd8f496ddf62b77cccb8fe325fbc"
df.drop('Store 1')

# + _uuid="5fa5a22b81b92bae274d3a9afd76283ded17b478"
df

# + _uuid="918e58d124508bd5edff0ed84ba6a4c252cdec3d"
copy_df = df.copy()
copy_df = copy_df.drop('Store 1')
copy_df

# + _uuid="0e3830ddb755492607166e3975ac6e18c9436422"
copy_df.drop

# + _uuid="946e48754fa8ed9914ef62b1c7049260861098db"
del copy_df['Name']
copy_df

# + _uuid="de93b5c96c8b546bfb01b57d75347f5045ea01d1"
df['Location'] = None
df

# + _uuid="42414bc5b478108d59aaf9b5dff463c95904097d"
costs = df['Cost']
costs

# + _uuid="76922dc612283caa9821f793abac91dad9328c75"
costs+=2
costs

# + _uuid="3ea1881948d05207d7dc2e1805c446adfa544959"
df

# + [markdown] _uuid="4d339cee9608b148762d7ad3068c362bbc9454f7"
# <a id="23"></a> <br>
# # 4-1 Pandas:Dataframe Indexing and Loading
#
# As a Data Scientist, you'll often find that the data you need is not in a single file. It may be spread across a number of text files, spreadsheets, or databases. You want to be able to import the data of interest as a collection of DataFrames and figure out how to combine them to answer your central questions.
# ###### [Go to top](#top)

# + _uuid="53fa2f4cb18784de6d077871a606dcf5b1511862"
df = pd.read_csv('train.csv')
df.head()

# + _uuid="01a8c7bc0a10635dc10dd56ba6edcbe595013772"
df.columns

# + _uuid="ffb1b09a5a8953e7dae65425890b652c214b1fb5"
# Querying a DataFrame

# + _uuid="258b2c7201efba77d84b92286bbe69a6af240ca8"
df['Survived'] > 0

# + _uuid="739f5037a2fcdd548abc5e68f5abcba3fcdb68e4"
only_Survived = df.where(df['Survived'] > 0)
only_Survived.head()

# + _uuid="e420645f2daa14d2bf12b3370438b5c1741f5c52"
only_Survived['Survived'].count()

# + _uuid="2c3561ac4d86a22f3984b11ebe1200100fc95417"
df['Survived'].count()

# + _uuid="86e547ab11dacd87ccfe4657f8eb11fd9fcf3fef"
only_Survived = only_Survived.dropna()
only_Survived.head()

# + _uuid="cf0829126fff5c075151fcc5418bbe9b945c14c9"
only_Survived = df[df['Survived'] > 0]
only_Survived.head()

# + _uuid="5b957eae6c82a982ff9672d321deadf637aa421c"
len(df[(df['Survived'] > 0) | (df['Survived'] > 0)])

# + _uuid="d076fe5f6dade0e49c2a35c7f9c64baeaf42a59d"
df[(df['Survived'] > 0) & (df['Survived'] == 0)]

# + _uuid="717b27412a1a852c84f820272d8bf94a45022aca"
# Indexing Dataframes

# + _uuid="822efde2bbb058575dea289d057368d1af7d1394"
df.head()

# + _uuid="b11ed5fbe0e8d35b303125afc78b04abf4dc0190"
df['PassengerId'] = df.index
df = df.set_index('Survived')
df.head()

# + _uuid="4848977e538e02e0444862e632101a9d6bc97742"

df = df.reset_index()
df.head()

# + _uuid="5da1a958ccd43f5f5427415dc8682ccbbd589b3d"
df = pd.read_csv('train.csv')
df.head()

# + _uuid="9c818adf02056d59d534e4cb790dd6ce74c2b861"
df['Age'].unique()

# + _uuid="c1adb2169bf24831daaa59655083e069d5fda4a5"
df=df[df['Age'] == 50]
df.head()

# + [markdown] _uuid="94d2eb99802e00e342e3a046f9b26a06a3c501a7"
# <a id="24"></a> <br>
# # 4-2 Pandas:Missing values
#

# + _uuid="6946487c3ba7a29af57472c6fe03cde0ababd341"
df = pd.read_csv('train.csv')
df

# + _uuid="30cc0a09aa17b60a69ddccebbc0b6ceaf6077bfb"
df.fillna

# + _uuid="be537539e67066ad45f9217988aa7ca7c23a370b"
df = df.set_index('PassengerId')
df = df.sort_index()
df

# + _uuid="d2681f382b87e0eb47c41745576c2d35a8f55f5b"
df = df.reset_index()
df = df.set_index(['PassengerId', 'Survived'])
df

# + _uuid="bea1dfdc973fe52315d701fecb6abb28edaecb81"
df = df.fillna(method='ffill')
df.head()

# + [markdown] _uuid="d79a17c1a5930de30ef9c238bf143cfc9962d24f"
# <a id="25"></a> <br>
# # 4-3 Pandas :Merging Dataframes
# pandas provides various facilities for easily combining together Series, DataFrame, and Panel objects with various kinds of set logic for the indexes and relational algebra functionality in the case of join / merge-type operations.
#

# + _uuid="4809bb7be74c5ef657c069446ecffb409937f952"
df = pd.DataFrame([{'Name': 'MJ', 'Item Purchased': 'Sponge', 'Cost': 22.50},
                   {'Name': 'Kevyn', 'Item Purchased': 'Kitty Litter', 'Cost': 2.50},
                   {'Name': 'Filip', 'Item Purchased': 'Spoon', 'Cost': 5.00}],
                  index=['Store 1', 'Store 1', 'Store 2'])
df

# + _uuid="f30d304abf7b7345a4e6e7c1105e190dd1a621d2"
df['Date'] = ['December 1', 'January 1', 'mid-May']
df

# + _uuid="fbedab0057046e0510dd1331f03ffc18c9ba520b"
df['Delivered'] = True
df

# + _uuid="8ed708570d219bad3637b3a907bb0a00be33b939"
df['Feedback'] = ['Positive', None, 'Negative']
df

# + _uuid="fc549de9e14ccf0504553ee8960442180ba895b0"
adf = df.reset_index()
adf['Date'] = pd.Series({0: 'December 1', 2: 'mid-May'})
adf

# + _uuid="80e056750b87aa3d692e6f3aa07ca4e40ce05512"
staff_df = pd.DataFrame([{'Name': 'Kelly', 'Role': 'Director of HR'},
                         {'Name': 'Sally', 'Role': 'Course liasion'},
                         {'Name': 'James', 'Role': 'Grader'}])
staff_df = staff_df.set_index('Name')
student_df = pd.DataFrame([{'Name': 'James', 'School': 'Business'},
                           {'Name': 'Mike', 'School': 'Law'},
                           {'Name': 'Sally', 'School': 'Engineering'}])
student_df = student_df.set_index('Name')
print(staff_df.head())
print()
print(student_df.head())

# + _uuid="c0e141f46ea59c406f9c75a501139d808720bea6"
pd.merge(staff_df, student_df, how='outer', left_index=True, right_index=True)

# + _uuid="117f0b5ad0687b45deead65bfd2cd2e2b42aec7a"
pd.merge(staff_df, student_df, how='inner', left_index=True, right_index=True)

# + _uuid="0fcc1d0780a0b786ffbb77e88a1e1bdc5f415a4a"
pd.merge(staff_df, student_df, how='left', left_index=True, right_index=True)

# + _uuid="d2492a6a8108c115d1f9c980a8b01242cc695a37"
pd.merge(staff_df, student_df, how='right', left_index=True, right_index=True)

# + _uuid="7876e8102392731c7d48123c8c5dced6693a32d2"
staff_df = staff_df.reset_index()
student_df = student_df.reset_index()
pd.merge(staff_df, student_df, how='left', left_on='Name', right_on='Name')

# + _uuid="08c724f0bf11154a1244924bd7ca0b195fff3a21"
staff_df = pd.DataFrame([{'Name': 'Kelly', 'Role': 'Director of HR', 'Location': 'State Street'},
                         {'Name': 'Sally', 'Role': 'Course liasion', 'Location': 'Washington Avenue'},
                         {'Name': 'James', 'Role': 'Grader', 'Location': 'Washington Avenue'}])
student_df = pd.DataFrame([{'Name': 'James', 'School': 'Business', 'Location': '1024 Billiard Avenue'},
                           {'Name': 'Mike', 'School': 'Law', 'Location': 'Fraternity House #22'},
                           {'Name': 'Sally', 'School': 'Engineering', 'Location': '512 Wilson Crescent'}])
pd.merge(staff_df, student_df, how='left', left_on='Name', right_on='Name')

# + _uuid="da1dc7a2a97543ef9d338dd45fc74b8e66f6221e"
staff_df = pd.DataFrame([{'First Name': 'Kelly', 'Last Name': 'Desjardins', 'Role': 'Director of HR'},
                         {'First Name': 'Sally', 'Last Name': 'Brooks', 'Role': 'Course liasion'},
                         {'First Name': 'James', 'Last Name': 'Wilde', 'Role': 'Grader'}])
student_df = pd.DataFrame([{'First Name': 'James', 'Last Name': 'Hammond', 'School': 'Business'},
                           {'First Name': 'Mike', 'Last Name': 'Smith', 'School': 'Law'},
                           {'First Name': 'Sally', 'Last Name': 'Brooks', 'School': 'Engineering'}])
staff_df
student_df
pd.merge(staff_df, student_df, how='inner', left_on=['First Name','Last Name'], right_on=['First Name','Last Name'])

# + [markdown] _uuid="c47476edc934d4d851254db98b156de91018a0c8"
# <a id="26"></a> <br>
# # 4-4 Idiomatic Pandas: Making Code Pandorable
#

# + _uuid="2cd537dc9b3bd93a924d808ef8d6377853dae984"

df = pd.read_csv('../input/train.csv')
df

# + _uuid="6ec456b0ba8db3f621911ad4c3a36008b04cfc7f"
df = df[df['Age']==50]
df.set_index(['PassengerId','Survived'], inplace=True)
df.rename(columns={'Pclass': 'pclass'})

# + [markdown] _uuid="ef246335b484cd563b29cafd6c178820c04a6f0f"
# <a id="27"></a> <br>
# ## 4-5 Pandas :Group by

# + _uuid="b45f3ee990bce0101774749b4b81e24b81911ad6"

df = pd.read_csv('../input/train.csv')
df = df[df['Age']==50]
df

# + _uuid="1c9afa54368039e6439950f57d39e5e0ae1faf7a"
df.head()

# + [markdown] _uuid="49994af42a822d8e0ad579866d12fdcd3a7b65ba"
# <a id="28"></a> <br>
# ## 4-6 Pandas:Scales
#

# + _uuid="ef0c2c453afcd5f43e37f27dd3dccd01aa7e33c0"
df = pd.DataFrame(['A+', 'A', 'A-', 'B+', 'B', 'B-', 'C+', 'C', 'C-', 'D+', 'D'],
                  index=['excellent', 'excellent', 'excellent', 'good', 'good', 'good', 'ok', 'ok', 'ok', 'poor', 'poor'])
df.rename(columns={0: 'Grades'}, inplace=True)
df

# + _uuid="cc490b8a1f851253430185eaab0d8a5ac1b843b8"
df['Grades'].astype('category').head()

# + _uuid="388f3454ed9fcc1b88c898e329c0c2d4b062df1f"
grades = df['Grades'].astype('category',
                             categories=['D', 'D+', 'C-', 'C', 'C+', 'B-', 'B', 'B+', 'A-', 'A', 'A+'],
                             ordered=True)
grades.head()

# + _uuid="51a0c5d18dbc6c395b900c34e9e590591d671676"
grades > 'C'

# + [markdown] _uuid="d205d889ce4d23e00f2ced7864c044e9d3d3ec84"
# <a id="30"></a> <br>
# ## 4-7 Pandas:Date Functionality
# ###### [Go to top](#top)

# + [markdown] _uuid="a466f75d44bdabf52ddb21d0c173d9421afce7d9"
# ### 4-7-1 Timestamp

# + _uuid="80a449e3cac139ac3b9697dba331363538a1a65f"
pd.Timestamp('9/1/2016 10:05AM')

# + [markdown] _uuid="c39545f2495d3f837c6b75dfd57b1a53c3d27d75"
# ### 4-7-2 Period

# + _uuid="3c160f45b74a5e4faecbf6661978e9c59e933e14"
pd.Period('1/2016')

# + _uuid="124c5a5ba7872ab55b3cd1fbbe18669747124eea"
pd.Period('3/5/2016')

# + [markdown] _uuid="f19b87e81ba7aa0d8adb850d75d9452cf3a73ddf"
# ### 4-7-3 DatetimeIndex

# + _uuid="68a99def5b8fbd9839cf35667a3481ed23c476c0"
t1 = pd.Series(list('abc'), [pd.Timestamp('2016-09-01'), pd.Timestamp('2016-09-02'), pd.Timestamp('2016-09-03')])
t1

# + _uuid="073eeeb51e53b50660a822eb7484d4e8b72a7dfa"
type(t1.index)

# + [markdown] _uuid="c891236e512838ed6088e02c88dd888029b226a3"
# ### 4-7-4 PeriodIndex

# + _uuid="5a19abe4e4a7324a8f7565c66f8270ab9eb3cae6"
t2 = pd.Series(list('def'), [pd.Period('2016-09'), pd.Period('2016-10'), pd.Period('2016-11')])
t2

# + _uuid="522200c3bbb47e10177f1c63a0ed3bfb49cbcf47"
type(t2.index)

# + [markdown] _uuid="50e147a4abff8fd3014f3fb2a105f516b4a5ea2f"
# ## 4-8 Pandas: Converting to Datetime

# + _uuid="862d2bc5c5b430ed2be1292b9d3b5efe8a3c9cc1"
d1 = ['2 June 2013', 'Aug 29, 2014', '2015-06-26', '7/12/16']
ts3 = pd.DataFrame(np.random.randint(10, 100, (4,2)), index=d1, columns=list('ab'))
ts3

# + _uuid="840963eeb8ca7bfe0a5d3167211e74d48446fe3a"
ts3.index = pd.to_datetime(ts3.index)
ts3

# + _uuid="e2482002fae0947549bdd81a3da93c8e0cde40fe"
pd.to_datetime('4.7.12', dayfirst=True)

# + _uuid="cae5847092e33cba7b657a90810a7ef35ae307e4"
pd.Timestamp('9/3/2016')-pd.Timestamp('9/1/2016')

# + [markdown] _uuid="c54a7d88507df86710fc585582b2074cb8d5aa5a"
# ### 4-8-1 Timedeltas

# + _uuid="fe333509538c1ec2bf81fda4613344cfa699b410"
pd.Timestamp('9/3/2016')-pd.Timestamp('9/1/2016')

# + _uuid="08e10cce4428eaa329eb84677c755b0307488bfa"
pd.Timestamp('9/2/2016 8:10AM') + pd.Timedelta('12D 3H')

# + [markdown] _uuid="a73485dcf7310c754e69ab5ed802d7e466684242"
# ### 4-8-2 Working with Dates in a Dataframe
#

# + _uuid="f5bcbc00ce23e50d346495588428e5d6f430a8df"
dates = pd.date_range('10-01-2016', periods=9, freq='2W-SUN')
dates

# + _uuid="41df9aeeb10a2d404f1eada62b4a4066e0f37af0"
df.index.ravel

# + [markdown] _uuid="2f7c5d5041dc630abeaff47ff5a96a0dd53db8e5"
# <a id="31"></a> <br>
# ## 4-9 Distributions in Pandas
# ###### [Go to top](#top)

# + _uuid="96be284ba6d63fd0b1db5641a21d75aacdfb7da4"
np.random.binomial(1, 0.5)

# + _uuid="4ae2c7ff2cf941bae62be23864a1685a196551d0"
np.random.binomial(1000, 0.5)/1000

# + _uuid="c5e89f6f0c7376c164f80dc4c5d582b0a639e254"
chance_of_tornado = 0.01/100
np.random.binomial(100000, chance_of_tornado)

# + _uuid="7804824638e0e6e97ebe5a252806d51a4e5cac2c"
chance_of_tornado = 0.01

tornado_events = np.random.binomial(1, chance_of_tornado, 1000000)
    
two_days_in_a_row = 0
for j in range(1,len(tornado_events)-1):
    if tornado_events[j]==1 and tornado_events[j-1]==1:
        two_days_in_a_row+=1

print('{} tornadoes back to back in {} years'.format(two_days_in_a_row, 1000000/365))

# + _uuid="4583c0a0ec05e914af1c01e2c901da41828ad653"
np.random.uniform(0, 1)

# + _uuid="365cfd6e6602f46bc98c13d10a47ad1d98af978e"
np.random.normal(0.75)

# + _uuid="8091adda4fbdccffb73424476c68a7aa0fb53c9a"
distribution = np.random.normal(0.75,size=1000)

np.sqrt(np.sum((np.mean(distribution)-distribution)**2)/len(distribution))

# + _uuid="9eed1ac016763a69de465e82736d30d7e5b1d028"
np.std(distribution)

# + _uuid="ab89e527fcd900577d879b50272d400ae0bdbaa0"

stats.kurtosis(distribution)

# + _uuid="c974e2563460f9b4942ab8c9b1d1783479779fa5"
stats.skew(distribution)

# + _uuid="de01fc7f82016eeb5f3b206788e7213ad373a441"
chi_squared_df2 = np.random.chisquare(2, size=10000)
stats.skew(chi_squared_df2)

# + _uuid="d8cce6278eaf4b4865efebf34dfb4ec2d9b684ca"
chi_squared_df5 = np.random.chisquare(5, size=10000)
stats.skew(chi_squared_df5)

# + _uuid="1c30298fc2dcd63dface93cb1626c5e7367d4698"
output = plt.hist([chi_squared_df2,chi_squared_df5], bins=50, histtype='step', 
                  label=['2 degrees of freedom','5 degrees of freedom'])
plt.legend(loc='upper right')


# + [markdown] _uuid="4f02c235284ef14f0007400125119ba36b81e325"
# <a id="33"></a> <br>
# ##  5- Matplotlib
#
# This Matplotlib tutorial takes you through the basics Python data visualization: the anatomy of a plot, pyplot and pylab, and much more
# ###### [Go to top](#top)

# + [markdown] _uuid="857d9a02e72211b4fde6f06debd3a57c8a0b1849"
# You can show matplotlib figures directly in the notebook by using the `%matplotlib notebook` and `%matplotlib inline` magic commands. 
#
# `%matplotlib notebook` provides an interactive environment.

# + _uuid="0d66eabe3f4f7288f12f8f0160b2f8e914d16ba9"
# because the default is the line style '-', 
# nothing will be shown if we only pass in one point (3,2)
plt.plot(3, 2)

# + _uuid="dad68ac5985444a4458259c448d1b6ffd9bf1710"
# we can pass in '.' to plt.plot to indicate that we want
# the point (3,2) to be indicated with a marker '.'
plt.plot(3, 2, '.')

# + [markdown] _uuid="18c4b411903becff1b3612730bc0287cb6ac14ad"
# Let's see how to make a plot without using the scripting layer.

# + _uuid="8f1bd139edb5cf1cc73f9f956d2d34844a726566"
# First let's set the backend without using mpl.use() from the scripting layer
from matplotlib.backends.backend_agg import FigureCanvasAgg
from matplotlib.figure import Figure

# create a new figure
fig = Figure()

# associate fig with the backend
canvas = FigureCanvasAgg(fig)

# add a subplot to the fig
ax = fig.add_subplot(111)

# plot the point (3,2)
ax.plot(3, 2, '.')

# save the figure to test.png
# you can see this figure in your Jupyter workspace afterwards by going to
# https://hub.coursera-notebooks.org/
canvas.print_png('test.png')

# + [markdown] _uuid="66c8f302e0afffb4652ff2f80212610fd37ae126"
# We can use html cell magic to display the image.

# + _uuid="d43abba30465cee9dc1364fd2a27334c70da8598" language="html"
# <img src='test.png' />

# + _uuid="2c76e50e3ecda1fe2144f16b9c9fdaa6bca37c61"
# create a new figure
plt.figure()

# plot the point (3,2) using the circle marker
plt.plot(3, 2, 'o')

# get the current axes
ax = plt.gca()

# Set axis properties [xmin, xmax, ymin, ymax]
ax.axis([0,6,0,10])

# + _uuid="0cc553845bb24ead8e57b5ccc59c6753a847f136"
# create a new figure
plt.figure()

# plot the point (1.5, 1.5) using the circle marker
plt.plot(1.5, 1.5, 'o')
# plot the point (2, 2) using the circle marker
plt.plot(2, 2, 'o')
# plot the point (2.5, 2.5) using the circle marker
plt.plot(2.5, 2.5, 'o')

# + _uuid="41cc8f299ccc6825cf1ac49a8bc2039496bcc9a9"
# get current axes
ax = plt.gca()
# get all the child objects the axes contains
ax.get_children()

# + _uuid="add973c3905be06ebae9f033070ee779f58dd785"

plt.plot([1, 2, 3, 4], [10, 20, 25, 30], color='lightblue', linewidth=3)
plt.scatter([0.3, 3.8, 1.2, 2.5], [11, 25, 9, 26], color='darkgreen', marker='^')
plt.xlim(0.5, 4.5)
plt.show()

# + [markdown] _uuid="0f4decd6f93319534a203748fd8a2b9bcc154ddb"
# <a id="34"></a> <br>
# ## 5-1 Scatterplots

# + _uuid="bf892e30bb09f0dc5fdf039d827ac55560b6ec06"


x = np.array([1,2,3,4,5,6,7,8])
y = x

plt.figure()
plt.scatter(x, y) # similar to plt.plot(x, y, '.'), but the underlying child objects in the axes are not Line2D

# + _uuid="9d4e979c8b366aa6e511a6d7316f63586346c649"


x = np.array([1,2,3,4,5,6,7,8])
y = x

# create a list of colors for each point to have
# ['green', 'green', 'green', 'green', 'green', 'green', 'green', 'red']
colors = ['green']*(len(x)-1)
colors.append('red')

plt.figure()

# plot the point with size 100 and chosen colors
plt.scatter(x, y, s=100, c=colors)

# + _uuid="935be30b9a6a7c78b58fe1a9c6e4690c494fb697"
# convert the two lists into a list of pairwise tuples
zip_generator = zip([1,2,3,4,5], [6,7,8,9,10])

print(list(zip_generator))
# the above prints:
# [(1, 6), (2, 7), (3, 8), (4, 9), (5, 10)]

zip_generator = zip([1,2,3,4,5], [6,7,8,9,10])
# The single star * unpacks a collection into positional arguments
print(*zip_generator)
# the above prints:
# (1, 6) (2, 7) (3, 8) (4, 9) (5, 10)

# + _uuid="d6f848fc8d0879c658a6a4e90ca533bddf3be6e3"
# use zip to convert 5 tuples with 2 elements each to 2 tuples with 5 elements each
print(list(zip((1, 6), (2, 7), (3, 8), (4, 9), (5, 10))))
# the above prints:
# [(1, 2, 3, 4, 5), (6, 7, 8, 9, 10)]


zip_generator = zip([1,2,3,4,5], [6,7,8,9,10])
# let's turn the data back into 2 lists
x, y = zip(*zip_generator) # This is like calling zip((1, 6), (2, 7), (3, 8), (4, 9), (5, 10))
print(x)
print(y)
# the above prints:
# (1, 2, 3, 4, 5)
# (6, 7, 8, 9, 10)

# + _uuid="b9c79e1558adbb3b19de2564376a24191766ab1f"
plt.figure()
# plot a data series 'Tall students' in red using the first two elements of x and y
plt.scatter(x[:2], y[:2], s=100, c='red', label='Tall students')
# plot a second data series 'Short students' in blue using the last three elements of x and y 
plt.scatter(x[2:], y[2:], s=100, c='blue', label='Short students')

# + _uuid="70731835527ce5a16e4a8bd2a0017e3bbe8065ee"
# add a label to the x axis
plt.xlabel('The number of times the child kicked a ball')
# add a label to the y axis
plt.ylabel('The grade of the student')
# add a title
plt.title('Relationship between ball kicking and grades')

# + _uuid="691db571df12646a4fe2740e55a43942ea779063"
# add a legend (uses the labels from plt.scatter)
plt.legend()

# + _uuid="0908d7c94c14feb08f953e5cd10f31cb0081f5d3"
# add the legend to loc=4 (the lower right hand corner), also gets rid of the frame and adds a title
plt.legend(loc=4, frameon=False, title='Legend')

# + _uuid="0eb79626b7b55559e40ca71ba5c3542466b629c9"
# get children from current axes (the legend is the second to last item in this list)
plt.gca().get_children()

# + _uuid="f2b5d97457e3ebd63311aef5b92fc137bd1701f5"
# get the legend from the current axes
legend = plt.gca().get_children()[-2]

# + _uuid="5334176f4177a64cb1726accb26502beb9b36626"
x = np.random.randint(low=1, high=11, size=50)
y = x + np.random.randint(1, 5, size=x.size)
data = np.column_stack((x, y))

fig, (ax1, ax2) = plt.subplots(nrows=1, ncols=2,
                               figsize=(8, 4))

ax1.scatter(x=x, y=y, marker='o', c='r', edgecolor='b')
ax1.set_title('Scatter: $x$ versus $y$')
ax1.set_xlabel('$x$')
ax1.set_ylabel('$y$')

ax2.hist(data, bins=np.arange(data.min(), data.max()),
         label=('x', 'y'))
ax2.legend(loc=(0.65, 0.8))
ax2.set_title('Frequencies of $x$ and $y$')
ax2.yaxis.tick_right()

# + [markdown] _uuid="c372afa727e992b9ec097c1f172becfd06bff4ef"
# <a id="35"></a> <br>
# ## 5-2 Line Plots

# + _uuid="163942ca710e3c67e8479cd02518ca37eae8ead7"


linear_data = np.array([1,2,3,4,5,6,7,8])
exponential_data = linear_data**2

plt.figure()
# plot the linear data and the exponential data
plt.plot(linear_data, '-o', exponential_data, '-o')

# + _uuid="7a4c6677b81fb82e1e7a1164ced70c4168080dbf"
# plot another series with a dashed red line
plt.plot([22,44,55], '--r')

# + [markdown] _uuid="517e72eb500a3ba4db76467919f59ded9d3683cb"
# <a id="36"></a> <br>
# ## 5-3 Bar Charts

# + _uuid="8a5ee722c74c9228af2bcece24831e1539fe97b2"
plt.figure()
xvals = range(len(linear_data))
plt.bar(xvals, linear_data, width = 0.3)

# + _uuid="11a85a29ca9b1574536bfee91cc63122148d996e"
new_xvals = []

# plot another set of bars, adjusting the new xvals to make up for the first set of bars plotted
for item in xvals:
    new_xvals.append(item+0.3)

plt.bar(new_xvals, exponential_data, width = 0.3 ,color='red')

# + _uuid="f6597010276961362ce22b84d49a9ef5f46521f6"
from random import randint
linear_err = [randint(0,15) for x in range(len(linear_data))] 

# This will plot a new set of bars with errorbars using the list of random error values
plt.bar(xvals, linear_data, width = 0.3, yerr=linear_err)

# + _uuid="8db2b0bb5d4160f7954bbcd7d9e53d29c68ef342"
# stacked bar charts are also possible
plt.figure()
xvals = range(len(linear_data))
plt.bar(xvals, linear_data, width = 0.3, color='b')
plt.bar(xvals, exponential_data, width = 0.3, bottom=linear_data, color='r')

# + _uuid="ee208f2f850122b6e01ff1236b2915914fe59466"
# or use barh for horizontal bar charts
plt.figure()
xvals = range(len(linear_data))
plt.barh(xvals, linear_data, height = 0.3, color='b')
plt.barh(xvals, exponential_data, height = 0.3, left=linear_data, color='r')

# + _uuid="acd3cfbdedddae19d7a2bbc9b02fefba21f595de"


# Initialize the plot
fig = plt.figure(figsize=(20,10))
ax1 = fig.add_subplot(121)
ax2 = fig.add_subplot(122)

# or replace the three lines of code above by the following line: 
#fig, (ax1, ax2) = plt.subplots(1,2, figsize=(20,10))

# Plot the data
ax1.bar([1,2,3],[3,4,5])
ax2.barh([0.5,1,2.5],[0,1,2])

# Show the plot
plt.show()

# + _uuid="db3db00430b8e5ddb633f3c7c82a6bf63ccb0cbc"
plt.figure()
# subplot with 1 row, 2 columns, and current axis is 1st subplot axes
plt.subplot(1, 2, 1)

linear_data = np.array([1,2,3,4,5,6,7,8])

plt.plot(linear_data, '-o')

# + _uuid="ce5372a8e09fd5379b1b33c7b451ec1b983f8616"
exponential_data = linear_data**2 

# subplot with 1 row, 2 columns, and current axis is 2nd subplot axes
plt.subplot(1, 2, 2)
plt.plot(exponential_data, '-o')

# + _uuid="cc4de0125429bdcd3227e894d083c0425934b228"
# plot exponential data on 1st subplot axes
plt.subplot(1, 2, 1)
plt.plot(exponential_data, '-x')

# + _uuid="0b565e5356d99d5875de88a853710ee2dd3c4a53"
plt.figure()
ax1 = plt.subplot(1, 2, 1)
plt.plot(linear_data, '-o')
# pass sharey=ax1 to ensure the two subplots share the same y axis
ax2 = plt.subplot(1, 2, 2, sharey=ax1)
plt.plot(exponential_data, '-x')

# + _uuid="ec35c2bab0a1debaa6420f15b929d127d0d5f8cd"
plt.figure()
# the right hand side is equivalent shorthand syntax
plt.subplot(1,2,1) == plt.subplot(121)

# + _uuid="43bb9cbc7acd94927be086a66ad913bd0a796fc5"
# create a 3x3 grid of subplots
fig, ((ax1,ax2,ax3), (ax4,ax5,ax6), (ax7,ax8,ax9)) = plt.subplots(3, 3, sharex=True, sharey=True)
# plot the linear_data on the 5th subplot axes 
ax5.plot(linear_data, '-')

# + _uuid="ec95c476865ba15ca11e2ef4a1ba466a9212f9b9"
# set inside tick labels to visible
for ax in plt.gcf().get_axes():
    for label in ax.get_xticklabels() + ax.get_yticklabels():
        label.set_visible(True)
plt.show()

# + _uuid="161be784e0c413060709d21a160985ad798b40f8"
# necessary on some systems to update the plot
plt.gcf().canvas.draw()
plt.show()

# + [markdown] _uuid="487fb5e77983c55ae891c3a99537d6cef0450b74"
# <a id="37"></a> <br>
# ## 5-4 Histograms

# + _uuid="42551850b478e274b9f78d4f6ef717c636242616"
# create 2x2 grid of axis subplots
fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, sharex=True)
axs = [ax1,ax2,ax3,ax4]

# draw n = 10, 100, 1000, and 10000 samples from the normal distribution and plot corresponding histograms
for n in range(0,len(axs)):
    sample_size = 10**(n+1)
    sample = np.random.normal(loc=0.0, scale=1.0, size=sample_size)
    axs[n].hist(sample)
    axs[n].set_title('n={}'.format(sample_size))

# + _uuid="39b84012b223069dd4cc6f1441d2ad0f585218bf"
# repeat with number of bins set to 100
fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, sharex=True)
axs = [ax1,ax2,ax3,ax4]

for n in range(0,len(axs)):
    sample_size = 10**(n+1)
    sample = np.random.normal(loc=0.0, scale=1.0, size=sample_size)
    axs[n].hist(sample, bins=100)
    axs[n].set_title('n={}'.format(sample_size))

# + _uuid="4def4012769663b5665f7b70a077cf839b7793f1"
plt.figure()
Y = np.random.normal(loc=0.0, scale=1.0, size=10000)
X = np.random.random(size=10000)
plt.scatter(X,Y)

# + _uuid="f6304a7d71b6edf6894e93bc62ad9ee4ffb0cce1"
# use gridspec to partition the figure into subplots
import matplotlib.gridspec as gridspec

plt.figure()
gspec = gridspec.GridSpec(3, 3)

top_histogram = plt.subplot(gspec[0, 1:])
side_histogram = plt.subplot(gspec[1:, 0])
lower_right = plt.subplot(gspec[1:, 1:])

# + _uuid="2a8f66c5169fcabafafa104e87897f175624245e"
Y = np.random.normal(loc=0.0, scale=1.0, size=10000)
X = np.random.random(size=10000)
lower_right.scatter(X, Y)
top_histogram.hist(X, bins=100)
s = side_histogram.hist(Y, bins=100, orientation='horizontal')

# + _uuid="97062a938d44626006bd9e24b5893fdf9d98155d"
# clear the histograms and plot normed histograms
top_histogram.clear()
top_histogram.hist(X, bins=100, normed=True)
side_histogram.clear()
side_histogram.hist(Y, bins=100, orientation='horizontal', normed=True)
# flip the side histogram's x axis
side_histogram.invert_xaxis()

# + _uuid="e6b1208bb887180b8f0363c9e82cc6fcbf387698"
# change axes limits
for ax in [top_histogram, lower_right]:
    ax.set_xlim(0, 1)
for ax in [side_histogram, lower_right]:
    ax.set_ylim(-5, 5)

# + [markdown] _uuid="2bc0c6fc0bb19748e9e87603e3207f75ffa9b565"
# <a id="38"></a> <br>
# ## 5-5 Box and Whisker Plots

# + _uuid="94dad21ec08e2633dacb64a89e5c807145042994"

normal_sample = np.random.normal(loc=0.0, scale=1.0, size=10000)
random_sample = np.random.random(size=10000)
gamma_sample = np.random.gamma(2, size=10000)

df = pd.DataFrame({'normal': normal_sample, 
                   'random': random_sample, 
                   'gamma': gamma_sample})

# + _uuid="87c08a9f914647f1735cb4b835b80f645685ef1e"
df.describe()

# + _uuid="9f4b288fe4b8ab78e6ad788e4bcfb5931920fcf2"
plt.figure()
# create a boxplot of the normal data, assign the output to a variable to supress output
_ = plt.boxplot(df['normal'], whis='range')

# + _uuid="9dc56a6415be6584fba51630ced26b0aaa486a09"
# clear the current figure
plt.clf()
# plot boxplots for all three of df's columns
_ = plt.boxplot([ df['normal'], df['random'], df['gamma'] ], whis='range')

# + _uuid="0f44453d7022928d2aeed3c3e0126cbd7118cdd9"
plt.figure()
_ = plt.hist(df['gamma'], bins=100)

# + _uuid="4dba4f705171f002fc94429ec80f7b6a2fe67ff3"
import mpl_toolkits.axes_grid1.inset_locator as mpl_il

plt.figure()
plt.boxplot([ df['normal'], df['random'], df['gamma'] ], whis='range')
# overlay axis on top of another 
ax2 = mpl_il.inset_axes(plt.gca(), width='60%', height='40%', loc=2)
ax2.hist(df['gamma'], bins=100)
ax2.margins(x=0.5)

# + _uuid="90b1e8ffe23e39ec54414c1fd63b5d5c4e72be6f"
# switch the y axis ticks for ax2 to the right side
ax2.yaxis.tick_right()

# + _uuid="cb3652a440484d391d27122878456a642c58d804"
# if `whis` argument isn't passed, boxplot defaults to showing 1.5*interquartile (IQR) whiskers with outliers
plt.figure()
_ = plt.boxplot([ df['normal'], df['random'], df['gamma'] ] )

# + [markdown] _uuid="fbf8bf0a67f6c49d78911c5f37be531ebbcd9edb"
# <a id="39"></a> <br>
# ## 5-6 Heatmaps

# + _uuid="ebfc0dcb8e85aa540f6568fa96431a4e9707f3c1"
plt.figure()

Y = np.random.normal(loc=0.0, scale=1.0, size=10000)
X = np.random.random(size=10000)
_ = plt.hist2d(X, Y, bins=25)

# + _uuid="fdbcf35950f94a4d0f1ce10efee6a4502f6ecfc8"
plt.figure()
_ = plt.hist2d(X, Y, bins=100)

# + [markdown] _uuid="139f44a4deb043128c8c7254eb60c33e0fc26e68"
# <a id="40"></a> <br>
# ## 5-7 Animations

# + _uuid="b3676970195153dc6056600f024f55c1b6f0ba12"
import matplotlib.animation as animation

n = 100
x = np.random.randn(n)


# + _uuid="ea3eb5835e4acd53d43da483e17d79c32228cad6"
# create the function that will do the plotting, where curr is the current frame
def update(curr):
    # check if animation is at the last frame, and if so, stop the animation a
    if curr == n: 
        a.event_source.stop()
    plt.cla()
    bins = np.arange(-4, 4, 0.5)
    plt.hist(x[:curr], bins=bins)
    plt.axis([-4,4,0,30])
    plt.gca().set_title('Sampling the Normal Distribution')
    plt.gca().set_ylabel('Frequency')
    plt.gca().set_xlabel('Value')
    plt.annotate('n = {}'.format(curr), [3,27])


# + _uuid="fb2314b3b1735c5e191c8427c5abe6429e4ff767"
fig = plt.figure()
a = animation.FuncAnimation(fig, update, interval=100)

# + [markdown] _uuid="7e702e5e0a876f9fa0b2e4fe497a56b91e00a95d"
# <a id="41"></a> <br>
# ## 5-8 Interactivity

# + _uuid="51fefb947daf8ca558cbc153e2ddbf39bcb7d4b2"
plt.figure()
data = np.random.rand(10)
plt.plot(data)

def onclick(event):
    plt.cla()
    plt.plot(data)
    plt.gca().set_title('Event at pixels {},{} \nand data {},{}'.format(event.x, event.y, event.xdata, event.ydata))

# tell mpl_connect we want to pass a 'button_press_event' into onclick when the event is detected
plt.gcf().canvas.mpl_connect('button_press_event', onclick)

# + _uuid="1d64a25dc386a30bd895ca8f58ca86d632f05d74"
from random import shuffle
origins = ['China', 'Brazil', 'India', 'USA', 'Canada', 'UK', 'Germany', 'Iraq', 'Chile', 'Mexico']

shuffle(origins)

df = pd.DataFrame({'height': np.random.rand(10),
                   'weight': np.random.rand(10),
                   'origin': origins})
df

# + _uuid="e13dd7e002938af1a52d7520004d839a3d7d2011"
plt.figure()
# picker=5 means the mouse doesn't have to click directly on an event, but can be up to 5 pixels away
plt.scatter(df['height'], df['weight'], picker=5)
plt.gca().set_ylabel('Weight')
plt.gca().set_xlabel('Height')


# + _uuid="e926ab1d2dc2098a6af48526e9f980bf594c79cd"
def onpick(event):
    origin = df.iloc[event.ind[0]]['origin']
    plt.gca().set_title('Selected item came from {}'.format(origin))

# tell mpl_connect we want to pass a 'pick_event' into onpick when the event is detected
plt.gcf().canvas.mpl_connect('pick_event', onpick)

# + _uuid="bb968584c16b9acc6466a3897c9415c57f3a7404"
# use the 'seaborn-colorblind' style
plt.style.use('seaborn-colorblind')

# + [markdown] _uuid="84742816a16280f7bca7c43879d6762e10e0a440"
# <a id="42"></a> <br>
# ## 5-9 DataFrame.plot

# + _uuid="97a3554f9640a2b77e07c861b5a5b6c814a3b276"
np.random.seed(123)

df = pd.DataFrame({'A': np.random.randn(365).cumsum(0), 
                   'B': np.random.randn(365).cumsum(0) + 20,
                   'C': np.random.randn(365).cumsum(0) - 20}, 
                  index=pd.date_range('1/1/2017', periods=365))
df.head()

# + _uuid="4d64fa4bc8b62fe1f4d2c2de0869bb49c8f7fc3d"
df.plot('A','B', kind = 'scatter');

# + [markdown] _uuid="857cecae1e2c9eb59c1a9d136ef1c5422d86d5ba"
# You can also choose the plot kind by using the `DataFrame.plot.kind` methods instead of providing the `kind` keyword argument.
#
# `kind` :
# - `'line'` : line plot (default)
# - `'bar'` : vertical bar plot
# - `'barh'` : horizontal bar plot
# - `'hist'` : histogram
# - `'box'` : boxplot
# - `'kde'` : Kernel Density Estimation plot
# - `'density'` : same as 'kde'
# - `'area'` : area plot
# - `'pie'` : pie plot
# - `'scatter'` : scatter plot
# - `'hexbin'` : hexbin plot
# ###### [Go to top](#top)

# + _uuid="74997530957394a96f0aed15c21e65f54911159c"
# create a scatter plot of columns 'A' and 'C', with changing color (c) and size (s) based on column 'B'
df.plot.scatter('A', 'C', c='B', s=df['B'], colormap='viridis')

# + _uuid="6299f8dddb909c7850620499edc49afdfd909f75"
ax = df.plot.scatter('A', 'C', c='B', s=df['B'], colormap='viridis')
ax.set_aspect('equal')

# + _uuid="91a25480397c8759047100da9ebc6c0264d8a918"
df.plot.box();

# + _uuid="b7bfa0ce17ea260d75eb97a0161af3dbd700f780"
df.plot.hist(alpha=0.7);

# + [markdown] _uuid="21a68cb3d0111753d29df2b402011daff81c5ff4"
# [Kernel density estimation plots](https://en.wikipedia.org/wiki/Kernel_density_estimation) are useful for deriving a smooth continuous function from a given sample.

# + _uuid="7b4d0f65af26e55acaf9a06da13dc71eb21a408b"
df.plot.kde();


# + [markdown] _uuid="68f7c4b8c85e6e28d978f11adbd9b54223d58538"
# <a id="43"></a> <br>
# # 6- Seaborn
#
# As you have just read, **Seaborn** is complimentary to Matplotlib and it specifically targets statistical data visualization. But it goes even further than that: Seaborn extends Matplotlib and that’s why it can address the two biggest frustrations of working with Matplotlib. Or, as <NAME> says in the “introduction to Seaborn”: “If matplotlib “tries to make easy things easy and hard things possible”, seaborn tries to make a well-defined set of hard things easy too.”
#
# One of these hard things or frustrations had to do with the default Matplotlib parameters. Seaborn works with different parameters, which undoubtedly speaks to those users that don’t use the default looks of the Matplotlib plots
# Seaborn is a library for making statistical graphics in Python. It is built on top of matplotlib and closely integrated with pandas data structures.
#
# Here is some of the functionality that seaborn offers:
#
# A dataset-oriented API for examining relationships between multiple variables
# Specialized support for using categorical variables to show observations or aggregate statistics
# Options for visualizing univariate or bivariate distributions and for comparing them between subsets of data
# Automatic estimation and plotting of linear regression models for different kinds dependent variables
# Convenient views onto the overall structure of complex datasets
# High-level abstractions for structuring multi-plot grids that let you easily build complex visualizations
# Concise control over matplotlib figure styling with several built-in themes
# Tools for choosing color palettes that faithfully reveal patterns in your data
# Seaborn aims to make visualization a central part of exploring and understanding data. Its dataset-oriented plotting functions operate on dataframes and arrays containing whole datasets and internally perform the necessary semantic mapping and statistical aggregation to produce informative plots.
#
# Here’s an example of what this means:

# + [markdown] _uuid="ad6fd29ef151c7625da19a38a8ef9b24b353427f"
# <a id="43"></a> <br>
# ## 6-1 Seaborn Vs Matplotlib
#
# It is summarized that if Matplotlib “tries to make easy things easy and hard things possible”, Seaborn tries to make a well defined set of hard things easy too.”
#
# Seaborn helps resolve the two major problems faced by Matplotlib; the problems are
#
# * Default Matplotlib parameters
# * Working with data frames
#
# As Seaborn compliments and extends Matplotlib, the learning curve is quite gradual. If you know Matplotlib, you are already half way through Seaborn.
#
# Important Features of Seaborn
# Seaborn is built on top of Python’s core visualization library Matplotlib. It is meant to serve as a complement, and not a replacement. However, Seaborn comes with some very important features. Let us see a few of them here. The features help in −
#
# * Built in themes for styling matplotlib graphics
# * Visualizing univariate and bivariate data
# * Fitting in and visualizing linear regression models
# * Plotting statistical time series data
# * Seaborn works well with NumPy and Pandas data structures
# * It comes with built in themes for styling Matplotlib graphics
#
# In most cases, you will still use Matplotlib for simple plotting. The knowledge of Matplotlib is recommended to tweak Seaborn’s default plots.

# + _uuid="43b3d518ca544a9eac820327635535b04fba563f"
def sinplot(flip = 1):
   x = np.linspace(0, 14, 100)
   for i in range(1, 5): 
      plt.plot(x, np.sin(x + i * .5) * (7 - i) * flip)
sinplot()
plt.show()


# + _uuid="3b82e364e3599e9979d62bc70c8058d4d60a3fc8"
def sinplot(flip = 1):
   x = np.linspace(0, 14, 100)
   for i in range(1, 5):
      plt.plot(x, np.sin(x + i * .5) * (7 - i) * flip)
 
sns.set()
sinplot()
plt.show()

# + _uuid="a78ee6a9554ff6a42f4d584d00f700c63b3945cb"
np.random.seed(1234)

v1 = pd.Series(np.random.normal(0,10,1000), name='v1')
v2 = pd.Series(2*v1 + np.random.normal(60,15,1000), name='v2')

# + _uuid="ee4e031c263014b7d16c96351bbec7503ea16fbf"
plt.figure()
plt.hist(v1, alpha=0.7, bins=np.arange(-50,150,5), label='v1');
plt.hist(v2, alpha=0.7, bins=np.arange(-50,150,5), label='v2');
plt.legend();

# + _uuid="67dcace37ccbc4ebbb58297ad5b09c4edcbc32e8"
plt.figure()
# we can pass keyword arguments for each individual component of the plot
sns.distplot(v2, hist_kws={'color': 'Teal'}, kde_kws={'color': 'Navy'});

# + _uuid="7bc165c0ea4d4f20a5975bd71a81e203afae0feb"
sns.jointplot(v1, v2, alpha=0.4);

# + _uuid="ee8e1e256786a02ada1f55233695ac507c05558e"
grid = sns.jointplot(v1, v2, alpha=0.4);
grid.ax_joint.set_aspect('equal')

# + _uuid="b25a62cb3ddf0be742e2465d20c9f69bc8062a95"
sns.jointplot(v1, v2, kind='hex');

# + _uuid="28893e60d141b295029b9414d6f4ac3fece7c43a"
# set the seaborn style for all the following plots
sns.set_style('white')

sns.jointplot(v1, v2, kind='kde', space=0);

# + _uuid="9eef5ff46fbf313f209a35ea44bce3b0d85e5966"
train = pd.read_csv('../input/train.csv')
train.head()

# + [markdown] _uuid="5587ea1eec62478c9970831d2886c4cca9237257"
# ## 6-2  10 Useful Python Data Visualization Libraries

# + [markdown] _uuid="c1c287a60f1c634dc68b18165ca6ee4df415320c"
# I am giving an overview of 10 interdisciplinary Python data visualization libraries, from the well-known to the obscure.
#
# * 1- matplotlib
#
# matplotlib is the O.G. of Python data visualization libraries. Despite being over a decade old, it’s still the most widely used library for plotting in the Python community. It was designed to closely resemble MATLAB, a proprietary programming language developed in the 1980s.
#
# * 2- Seaborn
#
# Seaborn harnesses the power of matplotlib to create beautiful charts in a few lines of code. The key difference is Seaborn’s default styles and color palettes, which are designed to be more aesthetically pleasing and modern. Since Seaborn is built on top of matplotlib, you’ll need to know matplotlib to tweak Seaborn’s defaults.
#
# * 3- ggplot
#
# ggplot is based on ggplot2, an R plotting system, and concepts from The Grammar of Graphics. ggplot operates differently than matplotlib: it lets you layer components to create a complete plot. For instance, you can start with axes, then add points, then a line, a trendline, etc. Although The Grammar of Graphics has been praised as an “intuitive” method for plotting, seasoned matplotlib users might need time to adjust to this new mindset.
#
#
# * 4- Bokeh
#
# Like ggplot, Bokeh is based on The Grammar of Graphics, but unlike ggplot, it’s native to Python, not ported over from R. Its strength lies in the ability to create interactive, web-ready plots, which can be easily outputted as JSON objects, HTML documents, or interactive web applications. Bokeh also supports streaming and real-time data.
#
#
# * 5- pygal
#
# Like Bokeh and Plotly, pygal offers interactive plots that can be embedded in the web browser. Its prime differentiator is the ability to output charts as SVGs. As long as you’re working with smaller datasets, SVGs will do you just fine. But if you’re making charts with hundreds of thousands of data points, they’ll have trouble rendering and become sluggish.
#
# * 6- Plotly
#
# You might know Plotly as an online platform for data visualization, but did you also know you can access its capabilities from a Python notebook? Like Bokeh, Plotly’s forte is making interactive plots, but it offers some charts you won’t find in most libraries, like contour plots, dendograms, and 3D charts.
#
# * 7- geoplotlib
#
# geoplotlib is a toolbox for creating maps and plotting geographical data. You can use it to create a variety of map-types, like choropleths, heatmaps, and dot density maps. You must have Pyglet (an object-oriented programming interface) installed to use geoplotlib. Nonetheless, since most Python data visualization libraries don’t offer maps, it’s nice to have a library dedicated solely to them.
#
# * 8- Gleam
#
# Gleam is inspired by R’s Shiny package. It allows you to turn analyses into interactive web apps using only Python scripts, so you don’t have to know any other languages like HTML, CSS, or JavaScript. Gleam works with any Python data visualization library. Once you’ve created a plot, you can build fields on top of it so users can filter and sort data.
#
#
# * 9- missingno
#
# Dealing with missing data is a pain. missingno allows you to quickly gauge the completeness of a dataset with a visual summary, instead of trudging through a table. You can filter and sort data based on completion or spot correlations with a heatmap or a dendrogram.
#
#
# * 10- Leather
#
# Leather’s creator, <NAME>, puts it best: “Leather is the Python charting library for those who need charts now and don’t care if they’re perfect.” It’s designed to work with all data types and produces charts as SVGs, so you can scale them without losing image quality. Since this library is relatively new, some of the documentation is still in progress. The charts you can make are pretty basic—but that’s the intention.
#
# At the end, nice cheatsheet on how to best visualize your data. I think I will print it out as a good reminder of "best practices". Check out the link for the complete cheatsheet, also as a PDF. hashtag#data hashtag#visualization hashtag#datascience
#
# Link: https://github.com/mjbahmani/Machine-Learning-Workflow-with-Python
# ![cheatsheet ][1]
# [Reference][2]
#
#
#   [1]: http://s8.picofile.com/file/8340669884/53f6a826_d7df_4b55_81e6_7c23b3fff0a3_original.png
#   [2]: https://blog.modeanalytics.com/python-data-visualization-libraries/

# + [markdown] _uuid="4524fb4d3bdeca34f1a526bbf2fc2caa9bfbf51a"
# <a id="44"></a> <br>
# ## 7- SKlearn
#
# - The __open source__ Python ecosystem provides __a standalone, versatile and powerful scientific working environment__, including: [NumPy](http://numpy.org), [SciPy](http://scipy.org), [IPython](http://ipython.org), [Matplotlib](http://matplotlib.org), [Pandas](http://pandas.pydata.org/), _and many others..._
#

# + [markdown] _uuid="ec7344e7f2a1bafa9a44a518722fcd8ec47c374b"
# <a id="45"></a> <br>
# ## 7-1 Introduction
#
#
#
# - Scikit-Learn builds upon NumPy and SciPy and __complements__ this scientific environment with machine learning algorithms;
# - By design, Scikit-Learn is __non-intrusive__, easy to use and easy to combine with other libraries;
# - Core algorithms are implemented in low-level languages.

# + [markdown] _uuid="6e80040de557789b0dff267ce45ba3e494885fee"
# <a id="46"></a> <br>
# ## 7-2 Algorithms 

# + [markdown] _uuid="666c206f83175114a513b37fb9ae322b5cd8543e"
# __Supervised learning:__
#
# * Linear models (Ridge, Lasso, Elastic Net, ...)
# * Support Vector Machines
# * Tree-based methods (Random Forests, Bagging, GBRT, ...)
# * Nearest neighbors 
# * Neural networks (basics)
# * Gaussian Processes
# * Feature selection

# + [markdown] _uuid="44eef8d741beebe15555c5166360b2ce77f5d5b1"
# __Unsupervised learning:__
#
# * Clustering (KMeans, Ward, ...)
# * Matrix decomposition (PCA, ICA, ...)
# * Density estimation
# * Outlier detection

# + [markdown] _uuid="8da2cc5428b697a7b5f21d34038d343bb8b094bb"
# __Model selection and evaluation:__
#
# * Cross-validation
# * Grid-search
# * Lots of metrics
#
# _... and many more!_ (See our [Reference](http://scikit-learn.org/dev/modules/classes.html))

# + [markdown] _uuid="e8a877d51d20c1ad31bb635cffc89175426eb77c"
# <a id="47"></a> <br>
# ## 7-3 Framework
#
# Data comes as a finite learning set ${\cal L} = (X, y)$ where
# * Input samples are given as an array $X$ of shape `n_samples` $\times$ `n_features`, taking their values in ${\cal X}$;
# * Output values are given as an array $y$, taking _symbolic_ values in ${\cal Y}$.

# + [markdown] _uuid="bafb45df9ecfe90563f2f9a1be8a327823cf6d35"
# The goal of supervised classification is to build an estimator $\varphi: {\cal X} \mapsto {\cal Y}$ minimizing
#
# $$
# Err(\varphi) = \mathbb{E}_{X,Y}\{ \ell(Y, \varphi(X)) \}
# $$
#
# where $\ell$ is a loss function, e.g., the zero-one loss for classification $\ell_{01}(Y,\hat{Y}) = 1(Y \neq \hat{Y})$.

# + [markdown] _uuid="7efef8f514caf78e7bc2a60b4d5c0e7fa6d160ac"
# <a id="48"></a> <br>
# ## 7-4 Applications
#
# - Classifying signal from background events; 
# - Diagnosing disease from symptoms;
# - Recognising cats in pictures;
# - Identifying body parts with Kinect cameras;
# - ...
#   

# + [markdown] _uuid="7cc13baab79cbc6446763e4ebe8feba2c95e74c9"
# <a id="49"></a> <br>
# ## 7-5 Data 
#
# - Input data = Numpy arrays or Scipy sparse matrices ;
# - Algorithms are expressed using high-level operations defined on matrices or vectors (similar to MATLAB) ;
#     - Leverage efficient low-leverage implementations ;
#     - Keep code short and readable. 

# + _uuid="da71475a60440648988cf0624a200439759112f7"
from sklearn import datasets
iris = datasets.load_iris()
X_iris = iris.data
y_iris = iris.target


# + [markdown] _uuid="9c1cb2c59e225e0bafa33bae8e77db53c1f34572"
# The dataset includes 150 instances, with 4 attributes each. For each instance, we will also have a target class (in our case, the species). This class is a special attribute which we will aim to predict for new, previously unseen instances, given the remaining (known) attributes.

# + _uuid="d60e800532bb74141e10d0ca57761ab8886831a9"
print (X_iris.shape, y_iris.shape)
print ('Feature names:{0}'.format(iris.feature_names))
print ('Target classes:{0}'.format(iris.target_names))
print ('First instance features:{0}'.format(X_iris[0]))



# + [markdown] _uuid="0699a589dbf3d8de81b360a3d1aee84019a64e3e"
# Let us display each instance in a 2d-scatter plot, using first sepal measures, and then petal measures.

# + _uuid="20a39dd47768e6ff4944573955c2b65322386efd"
plt.figure('sepal')
colormarkers = [ ['red','s'], ['greenyellow','o'], ['blue','x']]
for i in range(len(colormarkers)):
    px = X_iris[:, 0][y_iris == i]
    py = X_iris[:, 1][y_iris == i]
    plt.scatter(px, py, c=colormarkers[i][0], marker=colormarkers[i][1])

plt.title('Iris Dataset: Sepal width vs sepal length')
plt.legend(iris.target_names)
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')
plt.figure('petal')

for i in range(len(colormarkers)):
    px = X_iris[:, 2][y_iris == i]
    py = X_iris[:, 3][y_iris == i]
    plt.scatter(px, py, c=colormarkers[i][0], marker=colormarkers[i][1])

plt.title('Iris Dataset: petal width vs petal length')
plt.legend(iris.target_names)
plt.xlabel('Petal length')
plt.ylabel('Petal width')
plt.show()


# + [markdown] _uuid="90e0f3c3f49ae20dfd3b0648bf1fb4e9131455ef"
# <a id="50"></a> <br>
# ## 7-6 Supervised Learning: Classification

# + [markdown] _uuid="d41c9b43569995b19a096eaf1435ff05aa4010f4"
# In 1936 <NAME> introduced the Iris dataset to the statistics world, using it to develop a _linear discriminant model_. What he did was to build a linear combination of the attributes that separates a species from the rest, that is, find a straight line similar to the one we suggested in the previous section.
#
# Our first task will be to predict the specie of an Iris flower given the four sepal and length measures. For the moment, we will start using only two attributes, its sepal width and length. We will do this to ease visualization, but later we will use the four attributes, and see if performance improves. This is an instance of a **classification problem**, where we want to assign a label taken from a discrete set to an item according to its features.
#
# The typical classification process roughly involves the following steps: 
# - select your attributes, 
# - build a model based on available data, and 
# - evaluate your model’s performance on previously unseen data. 
#
# To do this, before building our model we should separate training and testing data. Training data will be used to build the model, and testing data will be used to evaluate its performance.
#

# + [markdown] _uuid="4f4c054a1e478dbf9745c1c9128ebed228a369dd"
# <a id="51"></a> <br>
# ## 7-7 Separate training and testing sets

# + [markdown] _uuid="29d9d5fb317893fd18c495bc6b93db45c8825f97"
# Our first step will be to separate the dataset into to separate sets, using 75% of the instances for training our classifier, and the remaining 25% for evaluating it (and, in this case, taking only two features, sepal width and length). We will also perform _feature scaling_: for each feature, calculate the average, subtract the mean value from the feature value, and divide the result by their standard deviation. After scaling, each feature will have a zero average, with a standard deviation of one. This standardization of values (which does not change their distribution, as you could verify by plotting the X values before and after scaling) is a common requirement of machine learning methods, to avoid that features with large values may weight too much on the final results.

# + _uuid="a0ba1691f3394e53a8939c709b2f874d34276e55"
from sklearn.model_selection import train_test_split
from sklearn import preprocessing

# Create dataset with only the first two attributes
X, y = X_iris[:, [0,1]], y_iris
# Test set will be the 25% taken randomly
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=33)
    
# Standarize the features
scaler = preprocessing.StandardScaler().fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)



# + [markdown] _uuid="d3d494e1c138f0ceccf7e916a7a0466053fb72e7"
# Check that, after scaling, the mean is 0 and the standard deviation is 1 (this should be exact in the training set, but only approximated in the testing set, because we used the training set media and standard deviation):

# + _uuid="50971f3878324a26664603f242e743c0005eaa47"
print ('Training set mean:{:.2f} and standard deviation:{:.2f}'.format(np.average(X_train),np.std(X_train)))
print ('Testing set mean:{:.2f} and standard deviation:{:.2f}'.format(np.average(X_test),np.std(X_test)))


# + [markdown] _uuid="f0f7e10a6a4435eae2734323f23fc3a5aa3f364b"
# Display the training data, after scaling.

# + _uuid="bf6fe6e16343a274dde88a9668521bad8d745617"
colormarkers = [ ['red','s'], ['greenyellow','o'], ['blue','x']]
plt.figure('Training Data')
for i in range(len(colormarkers)):
    xs = X_train[:, 0][y_train == i]
    ys = X_train[:, 1][y_train == i]
    plt.scatter(xs, ys, c=colormarkers[i][0], marker=colormarkers[i][1])

plt.title('Training instances, after scaling')
plt.legend(iris.target_names)
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')
plt.show()


# + [markdown] _uuid="4c6d378e8aa4a86c374cee25410178c06e880420"
# <a id="52"></a> <br>
# ## 7-8 A linear, binary classifier

# + [markdown] _uuid="a1220ea7fdc23fe8831310212c52bdd4d5a2dd09"
# To start, let's transform the problem to a binary classification task: we will only want to distinguish setosa flowers from the rest (it seems easy, according to the plot). To do this, we will just collapse all non-setosa targets into the same clas (later we will come back to the three-class original problem)

# + _uuid="5f762b1f6b88ea60a54a1bacc44bec42bc403bf7"
import copy 
y_train_setosa = copy.copy(y_train) 
# Every 1 and 2 classes in the training set will became just 1
y_train_setosa[y_train_setosa > 0]=1
y_test_setosa = copy.copy(y_test)
y_test_setosa[y_test_setosa > 0]=1

print ('New training target classes:\n{0}'.format(y_train_setosa))


# + [markdown] _uuid="753b8c2d1e1d6320cdb4b23818aa9afac5ca1099"
# Our first classifier will be a linear one. 
#
# Linear classification models have been very well studied through many years, and the are a lot of different methods with actually very different approaches for building the separating hyperplane.  We will use the `SGDClassifier` from scikit-learn to implement a linear model, including regularization.  The classifier (actually, a family of classifiers, as we will see) receives its name from using Stochastic Gradient Descent, a very effective numerical procedure to find the local minimum of a function. 
#
# Gradient Descent was introduced by <NAME> in 1847, to solve a system of linear equations. The idea is based on the observation that a multivariable function decreases fastest in the direction of its negative gradient (you can think of the gradient as a generalization of the derivative for several dimensions). If we want to find its minimum (at least a local one) we could move in the direction of its negative gradient. This is exactly what gradient descent does.
#

# + [markdown] _uuid="077958a4afee1ca86cb244f34b7926ab7ae7e856"
# Every classifier in scikit-learn is created the same way: calling a method with the classifier's configurable hyperparameters to create an instance of the classifier. In this case, we will use `linear_model.SGDClassifier`, telling scikit-learn to use a _log_ loss function. 

# + _uuid="624efe224ad09deb106bbd41098387241686e8e3"
from sklearn import linear_model 
clf = linear_model.SGDClassifier(loss='log', random_state=42)
print (clf)

# + [markdown] _uuid="9fade050090fe34bef437d28a35b918e399c304f"
# Note that the classifier includes several parameteres. Usually, scikit-learn specifies default values for every parameter. But be aware that it is not a good idea to keep it with their default values. Later (or in future notebooks, I do not know yet), we will talk about _model selection_, the process of selecting the best parameters.
#
# Now, we just call the `fit` method to train the classifier (i.e., build a model we will later use), based on the available training data. In our case, the trainig setosa set.
#

# + _uuid="79b66e76e757b42f4ed7fd49e3c62b83e861b625"
clf.fit(X_train, y_train_setosa)

    

# + [markdown] _uuid="a15f78bd1067a7e1614127a2f32a5971966c89a2"
# How does our model look? Well, since we are building a linear classifier, our model is a... line. We can show its coefficients:

# + _uuid="d087bff89ccc12837db145767a2213e19a83e095"
print (clf.coef_,clf.intercept_)


# + [markdown] _uuid="3c72318eb1be6a0670405774623093466716b386"
# ... and we can draw the decision boundary using pyplot:

# + _uuid="76d30025f191ad3f6d6e3572fdc24b37411de3c6"
x_min, x_max = X_train[:, 0].min() - .5, X_train[:, 0].max() + .5
y_min, y_max = X_train[:, 1].min() - .5, X_train[:, 1].max() + .5
xs = np.arange(x_min, x_max, 0.5)
fig,axes = plt.subplots()
axes.set_aspect('equal')
axes.set_title('Setosa classification')
axes.set_xlabel('Sepal length')
axes.set_ylabel('Sepal width')
axes.set_xlim(x_min, x_max)
axes.set_ylim(y_min, y_max)
plt.sca(axes)
plt.scatter(X_train[:, 0][y_train == 0], X_train[:, 1][y_train == 0], c='red', marker='s')
plt.scatter(X_train[:, 0][y_train == 1], X_train[:, 1][y_train == 1], c='black', marker='x')
ys = (-clf.intercept_[0]- xs * clf.coef_[0, 0]) / clf.coef_[0, 1]
plt.plot(xs, ys, hold=True)
plt.show()

# + [markdown] _uuid="5b44914dc28906e9b62bdde03b8dc6bc695421ce"
# The blue line is our decision boundary. Every time $30.97 \times sepal\_length - 17.82 \times sepal\_width - 17.34$ is greater than zero we will have an iris setosa (class 0). 

# + [markdown] _uuid="67295f4a942c50d80cb1e9ae2ca8ad59a6d2fb74"
# <a id="53"></a> <br>
# ## 7-9 Prediction

# + [markdown] _uuid="ea3f08d3e7d80014d39a580fee23a5b901836278"
# Now, the really useful part: when we have a new flower, we just have to get its petal width and length and call the `predict` method of the classifier on the new instance. _This works the same way no matter the classifier we are using or the method we used to build it_

# + _uuid="adc12ede8d795ab655398672c326947bfaf72352"
print ('If the flower has 4.7 petal width and 3.1 petal length is a {}'.format(
        iris.target_names[clf.predict(scaler.transform([[4.7, 3.1]]))]))

# + [markdown] _uuid="d1176b1981567568ad2a6fdf7df660971c60abbe"
# Note that we first scaled the new instance, then applyied the `predict` method, and used the result to lookup into the iris target names arrays. 

# + [markdown] _uuid="86353db47b33bc43025f024071c6cafd09cd3990"
# <a id="54"></a> <br>
# ## 7-10 Back to the original three-class problem

# + [markdown] _uuid="f388c210be1b2158117b65eb8592bfa7bc1386ca"
# Now, do the training using the three original classes. Using scikit-learn this is simple: we do exactly the same procedure, using the original three target classes:

# + _uuid="dff8a43a1c8d68407e90d1b83d1e8dbea87e958f"
clf2 = linear_model.SGDClassifier(loss='log', random_state=33)
clf2.fit(X_train, y_train) 
print (len(clf2.coef_))

# + [markdown] _uuid="8c986abaa3874a8cd0b65a272528a70a345b91cd"
# We have know _three_ decision curves... scikit-learn has simply converted the problem into three one-versus-all binary classifiers. Note that Class 0 is linearly separable, while Class 1 and Class 2 are not

# + _uuid="0b5cd2ffc95fb39d6b136cb6c9966dbbbd1f0500"
x_min, x_max = X_train[:, 0].min() - .5, X_train[:, 0].max() + .5
y_min, y_max = X_train[:, 1].min() - .5, X_train[:, 1].max() + .5
xs = np.arange(x_min,x_max,0.5)
fig, axes = plt.subplots(1,3)
fig.set_size_inches(10,6)
for i in [0,1,2]:
    axes[i].set_aspect('equal')
    axes[i].set_title('Class '+ iris.target_names[i] + ' versus the rest')
    axes[i].set_xlabel('Sepal length')
    axes[i].set_ylabel('Sepal width')
    axes[i].set_xlim(x_min, x_max)
    axes[i].set_ylim(y_min, y_max)
    plt.sca(axes[i])
    ys=(-clf2.intercept_[i]-xs*clf2.coef_[i,0])/clf2.coef_[i,1]
    plt.plot(xs,ys,hold=True)    
    for j in [0,1,2]:
        px = X_train[:, 0][y_train == j]
        py = X_train[:, 1][y_train == j]
        color = colormarkers[j][0] if j==i else 'black'
        marker = 'o' if j==i else 'x'
        plt.scatter(px, py, c=color, marker=marker)     

plt.show()


# + [markdown] _uuid="fa09989724671462a28e9dc963260c585441f48a"
# Let us evaluate on the previous instance to find the three-class prediction. Scikit-learn tries the three classifiers. 

# + _uuid="4a4819672b9113bf4cc77bd32d343eaa2defdfc0"
scaler.transform([[4.7, 3.1]])
print(clf2.decision_function(scaler.transform([[4.7, 3.1]])))
clf2.predict(scaler.transform([[4.7, 3.1]]))


# + [markdown] _uuid="3a67576ba3e26fc8b1a01ae245efee25fb28732c"
# The `decision_function` method tell us the classifier scores (in our case, the left side of the decision boundary inequality). In our example, the first classifier says the flower is a setosa (we have a score greater than zero), and it is not a versicolor nor a virginica. Easy. What if we had two positive values? In our case, the greatest score will be the point which is further away from the decision line. 

# + [markdown] _uuid="447f9507fe540dda42582d97b3958847c6a1c322"
# <a id="55"></a> <br>
# ## 7-11 Evaluating the classifier

# + [markdown] _uuid="d42f5fa8d221da382fa01028d2492a908059e840"
# The performance of an estimator is a measure of its effectiveness. The most obvious performance measure is called _accuracy_: given a classifier and a set of instances, it simply measures the proportion of instances correctly classified by the classifier. We can, for example, use the instances in the training set and calculate the accuracy of our classifier when predicting their target classes. Scikit-learn includes a `metrics` module that implements this (and many others) performance metric.

# + _uuid="b47d40cd9bec34549456e928de1fb3fa059fbdbc"
from sklearn import metrics
y_train_pred = clf2.predict(X_train)
print ('Accuracy on the training set:{:.2f}'.format(metrics.accuracy_score(y_train, y_train_pred)))


# + [markdown] _uuid="bd340e2bd542d74bf2507fb1d4714e21c1229749"
# This means that our classifier correctly predicts 83\% of the instances in the training set. But this is actually a bad idea.  The problem with the evaluating on the training set is that you have built your model using this data, and it is possible that your model adjusts actually very well to them, but performs poorly in previously unseen data (which is its ultimate purpose). This phenomenon is called overfitting, and you will see it once and again while you read this book. If you measure on your training data, you will never detect overfitting. So, _never ever_ measure on your training data. 
#
# Remember we separated a portion of the training set? Now it is time to use it: since it was not used for training, we expect it to give us and idead of how well our classifier performs on previously unseen data.

# + _uuid="70dd8452466af9966724f4b5af586bb68d81f27e"
y_pred = clf2.predict(X_test)
print ('Accuracy on the training set:{:.2f}'.format(metrics.accuracy_score(y_test, y_pred)))


# + [markdown] _uuid="760b5655dff3c9a89c4f1b3be08f5644a12de85b"
# Generally, accuracy on the testing set is lower than the accuracy on the training set, since the model is actually modeling the training set, not the testing set.
#
# One of the problems with accuracy is that does not reflect well how our model performs on each different target class. For example, we know that our classifier works very well identifying setosa species, but will probably fail when separating the other two species. If we could measure this, we could get hints for improving performance, changing the method or the features. 
#
# A very useful tool when facing multi-class problems is the confusion matrix. This matrix includes, in row i and column _j_ the number of instances of class _i_ that were predicted to be in class _j_. A good classifier will accumulate the values on the confusion matrix diagonal, where correctly classified instances belong. Having the original and predicted classes, we can easily print the confusion matrix:

# + _uuid="58e838b89ffd8b8d5b883897b531930e174fee3a"
print (metrics.confusion_matrix(y_test, y_pred))


# + [markdown] _uuid="6442fc90fe99617f6238ad264c140989c0bff47d"
# To read the classification matrix, just remember the definition: the “8” on row 2, column 3, means that eight instances if class 1 where predicted to be in class 2. Our classifier is never wrong in our evaluation set when it classifies class zero (setosa) flowers.  However, when it faces classes one and two (versicolor and virginica), it confuses them. The confusion matrix gives us useful information to know what kind of errors the classifier is making.

# + [markdown] _uuid="0bc68e8d5a2ef4423d70fbb7eb4ea48e6bee9594"
# Accuracy on the test set is a good performance measure when the number of instances of each class is similar, i.e., we have a uniform distribution of classes. However, consider that 99 percent of your instances belong to just one class (you have a skewed): a classifier that always predicts this  majority class will have an excellent performance in terms of accuracy, despite the fact that it is an extremely naive method (and that it will surely fail in the “difficult” 1% cases).
#
# Within scikit-learn, there are several evaluation functions; we will show three popular ones: precision, recall, and F1-score (or f-measure).

# + _uuid="9ca68dac14ab26a2392b8ac302db3ca16278c276"
print (metrics.classification_report(y_test, y_pred, target_names=iris.target_names))

# + [markdown] _uuid="6cb4f31974f9b2ea0c1ac9856daf7841e16fb344"
# - Precision computes the proportion of instances predicted as positives that were correctly evaluated (it measures how right is our classifier when it says that an instance is positive).
# - Recall counts the proportion of positive instances that were correctly evaluated (measuring how right our classifier is when faced with a positive instance).
# - F1-score is the harmonic mean of precision and recall, and tries to combine both in a single number.

# + [markdown] _uuid="790eff98fb2a7f1f5b469fd55b84f9565329a24a"
# <a id="56"></a> <br>
# ## 7-12 Using the four flower attributes

# + [markdown] _uuid="77229bf2110999ada713fbb0ae55c2ec725e7526"
# To end with this classification section, we will repeat the whole process, this time using the four original attributes, and check if performance improves.

# + _uuid="0b4214c627c2d3e64519b58aa529477f604db287"
# Test set will be the 25% taken randomly
X_train4, X_test4, y_train4, y_test4 = train_test_split(X_iris, y_iris, test_size=0.25, random_state=33)
    
# Standarize the features
scaler = preprocessing.StandardScaler().fit(X_train4)
X_train4 = scaler.transform(X_train4)
X_test4 = scaler.transform(X_test4)

# Build the classifier
clf3 = linear_model.SGDClassifier(loss='log', random_state=33)
clf3.fit(X_train4, y_train4) 

# Evaluate the classifier on the evaluation set
y_pred4 = clf3.predict(X_test4)
print (metrics.classification_report(y_test4, y_pred4, target_names=iris.target_names))


# + [markdown] _uuid="0ec1ebbf687f017f402f9fd66fc46e186d204ec6"
# <a id="57"></a> <br>
# ## 7-13 Unsupervised Learning: Clustering

# + [markdown] _uuid="af8b8dcd0dfc9fd8e224538b7e2c3cc0771100d5"
# Sometimes, is possible to take an unlabeled training set and try to find a hidden structure or patterns in the data: there is no given target class to predict or to evaluate the resulting model. We call these class of machine learning tasks _unsupervised learning_. For instance, _clustering_ methods try to group instances into subsets (called clusters): an instance should be similar to another in the same subset and different from those belonging to another subset.
#
# In this section, we will perform clustering of the Iris data set, to see if we could group instances using their petal and sepal width and length. The trainig set is the same we used we used for our last example on supervised classification.
#

# + [markdown] _uuid="0aab6518f743c93eb3708492b88e8f503589606f"
# K-means is probably the most popular clustering algorithm, because it is very simple and easy to implement, and it has shown good performance on different tasks. It belongs to the class of partition algorithms that simultaneously partition data points into distinct groups, called clusters. We will apply k-means to the training data, using only sepal dimensions, building 3 clusters (note that we could have selected a different number of cluster to group data into)

# + _uuid="da0b9bbfed649b023ea14910a4cb2fc6947df419"
from sklearn import cluster
clf_sepal = cluster.KMeans(init='k-means++', n_clusters=3, random_state=33)
clf_sepal.fit(X_train4[:,0:2])


# + [markdown] _uuid="8c5283b63443298ab77ce3c99ccef438b4ecfa32"
# We can show the label assigned for each instance (note that this label is a cluster name, it has nothing to do with our original target classes... actually, when you are doing clustering you have no target class!).

# + _uuid="dd66f6da3cdcf0b0df45033f8af80d8a913ebd3d"
print (clf_sepal.labels_)

# + [markdown] _uuid="fb2cf032fd214f500bc1b5c1e5cb2e91181e1d9c"
# Using NumPy's indexing capabilities, we can display the actual target classes for each cluster, just to compare the built clusters with our flower type classes...

# + _uuid="670b352cc30adc61caad1dfe64376c4d321ebecd"
print (y_train4[clf_sepal.labels_==0])

# + _uuid="1cde150dd604e7a62003b96f2897ed3882a85281"
print (y_train4[clf_sepal.labels_==1])

# + _uuid="0659d1c3fc12c5303065e2987816ad2dda76d455"
print (y_train4[clf_sepal.labels_==2])

# + [markdown] _uuid="2caa2d5609a8af2a2703a69c5395c8de9e6ebd10"
# As usually, is a good idea to display our instances and the clusters they belong to, to have a first approximation to how well our algorithm is behaving on our data: 

# + _uuid="6bc2b0a1cd98f8d58658f941799782e37f1fc381"
colormarkers = [ ['red','s'], ['greenyellow','o'], ['blue','x']]
step = .01 
margin = .1   
sl_min, sl_max = X_train4[:, 0].min()-margin, X_train4[:, 0].max() + margin
sw_min, sw_max = X_train4[:, 1].min()-margin, X_train4[:, 1].max() + margin
sl, sw  = np.meshgrid(
    np.arange(sl_min, sl_max, step),
np.arange(sw_min, sw_max, step)
    )
Zs = clf_sepal.predict(np.c_[sl.ravel(), sw.ravel()]).reshape(sl.shape)
centroids_s = clf_sepal.cluster_centers_


# + [markdown] _uuid="7cedf51a164ebbfae5cabb2cd24a94e74b243502"
# Display the data points and the calculated regions

# + _uuid="c756314b2724adffb276b09fafff446c4a29a044"
plt.figure(1)
plt.clf()
plt.imshow(Zs, interpolation='nearest', extent=(sl.min(), sl.max(), sw.min(), sw.max()), cmap= plt.cm.Pastel1, aspect='auto', origin='lower')
for j in [0,1,2]:
    px = X_train4[:, 0][y_train == j]
    py = X_train4[:, 1][y_train == j]
    plt.scatter(px, py, c=colormarkers[j][0], marker= colormarkers[j][1])
plt.scatter(centroids_s[:, 0], centroids_s[:, 1],marker='*',linewidths=3, color='black', zorder=10)
plt.title('K-means clustering on the Iris dataset using Sepal dimensions\nCentroids are marked with stars')
plt.xlim(sl_min, sl_max)
plt.ylim(sw_min, sw_max)
plt.xlabel("Sepal length")
plt.ylabel("Sepal width")
plt.show()

# + [markdown] _uuid="c6fa87cdcda44eeacba51b645c35dc7367e5c636"
# Repeat the experiment, using petal dimensions

# + _uuid="11e8ba67a6433ddac4efac055f86bb9b45e77252"
clf_petal = cluster.KMeans(init='k-means++', n_clusters=3, random_state=33)
clf_petal.fit(X_train4[:,2:4])


# + _uuid="8830e184fccb6094195cfc008a7bf5a479203167"
print (y_train4[clf_petal.labels_==0])

# + _uuid="f9d66355b903a618cabfef234f2eacb32b4eba4b"
print (y_train4[clf_petal.labels_==1])

# + _uuid="5f4ae459d0272d74281ab4ae2845ce4f74bcbb14"
print (y_train4[clf_petal.labels_==2])

# + [markdown] _uuid="e3a17a8fba39e8d49358579cae1ef6c3ab6e3c3c"
# Plot the clusters 

# + _uuid="5647c6daddf59a1bdcd8d5f2a58fbac147ba9f56"
colormarkers = [ ['red','s'], ['greenyellow','o'], ['blue','x']]
step = .01 
margin = .1
sl_min, sl_max = X_train4[:, 2].min()-margin, X_train4[:, 2].max() + margin
sw_min, sw_max = X_train4[:, 3].min()-margin, X_train4[:, 3].max() + margin
sl, sw  = np.meshgrid(
    np.arange(sl_min, sl_max, step),
    np.arange(sw_min, sw_max, step), 
    )
Zs = clf_petal.predict(np.c_[sl.ravel(), sw.ravel()]).reshape(sl.shape)
centroids_s = clf_petal.cluster_centers_
plt.figure(1)
plt.clf()
plt.imshow(Zs, interpolation='nearest', extent=(sl.min(), sl.max(), sw.min(), sw.max()), cmap= plt.cm.Pastel1, aspect='auto', origin='lower')
for j in [0,1,2]:
    px = X_train4[:, 2][y_train4 == j]
    py = X_train4[:, 3][y_train4 == j]
    plt.scatter(px, py, c=colormarkers[j][0], marker= colormarkers[j][1])
plt.scatter(centroids_s[:, 0], centroids_s[:, 1],marker='*',linewidths=3, color='black', zorder=10)
plt.title('K-means clustering on the Iris dataset using Petal dimensions\nCentroids are marked with stars')
plt.xlim(sl_min, sl_max)
plt.ylim(sw_min, sw_max)
plt.xlabel("Petal length")
plt.ylabel("Petal width")
plt.show()

# + [markdown] _uuid="409c72a611b055e4cc2e3dd523d88f7f46b10d16"
# Now, calculate the clusters, using the four attributes

# + _uuid="56668e53ec70e5e26d9994b0f172702011ed3e41"
clf = cluster.KMeans(init='k-means++', n_clusters=3, random_state=33)
clf.fit(X_train4)


# + _uuid="5a89293931b142460fd2912f1802a67712110bfb"
print (y_train[clf.labels_==0])

# + _uuid="27b83ae9ed49f4a517a409e72bb5416a3ea6dd8a"
print (y_train[clf.labels_==1])

# + _uuid="b7fae1d9f328ead4f338581999b41deaf29e20b7"
print (y_train[clf.labels_==2])

# + [markdown] _uuid="2ca2c5b2699e71c03c2ac1d9b47638adaddfb359"
# Measure precision & recall in the testing set, using all attributes, and using only petal measures

# + _uuid="f45af067d0c87aec50dcb0dffe93eba15257f741"
y_pred=clf.predict(X_test4)
print (metrics.classification_report(y_test, y_pred, target_names=['setosa','versicolor','virginica']))

# + _uuid="83c4f44e777257689fcb04190ae2ae2bad083f91"
y_pred_petal=clf_petal.predict(X_test4[:,2:4])
print (metrics.classification_report(y_test, y_pred_petal, target_names=['setosa','versicolor','virginica']))

# + [markdown] _uuid="b4da546786d8990d5b0ec6bc5488e11590bce7de"
# Wait, every performance measure is better using just two attributes. It is possible that less features give better results?  Although at a first glance this seems contradictory, we will see in future notebooks that selecting the right subset of features, a process called feature selection, could actually improve the performance of our algorithms.

# + [markdown] _uuid="341f79e2997c5dcedcc13434021135e5424006a6"
# <a id="58"></a> <br>
# ## 7-14 Supervised Learning: Regression

# + [markdown] _uuid="e7856f5680378e28373a93620f776fbec0a86847"
# In every example we have seen so far the output we aimed at predicting belonged to a discrete set. For classification, the set was the target class, while for the clustering algorithm the set included different calculated clusters. What if we want to predict a value extracted from the real line?. In this case, we are trying to solve a regression problem. 
#
# To show how regression works in scikit-learn, we will apply to a (very) simple and well-know problem: trying to predict the price of a house given some of its. As the dataset, we will use the Boston house-prices dataset (find the dataset description and attributes [here](https://github.com/mjbahmani).

# + _uuid="13728a32bdb3e5f5bb5355e95b43551054355b66"
from sklearn.datasets import load_boston
boston = load_boston()
print ('Boston dataset shape:{}'.format(boston.data.shape))


# + _uuid="d4610811ae4bfebb057e2054426cd2029c3f3226"
print (boston.feature_names)


# + [markdown] _uuid="e051b322579ebaec46e3de34b00b2e2186bdf210"
# Create training and testing sets, and scale values, as usual

# + [markdown] _uuid="43c8934779f1d7d266606adf95287cce83876068"
# Create a method for training and evaluating a model. This time, to evaluate our model we will a different approach: instead of separating the training set, we will use _cross-validation_. 
#
# Cross-validation usually involves the following steps:
# 1.	Partition the dataset into k different subsets.
# 2.	Create k different models by training on k-1 subsets and testing on the remaining one.
# 3.	Measure the performance of each of the k models and use the average value as you performance value.
#

# + [markdown] _uuid="91acd3dadd07de1cbfcdde2c3309c241429e9174"
# <a id="59"></a> <br>
# ## 8- Plotly
# How to use **Plotly** offline inside IPython notebooks.
#
#

# + [markdown] _uuid="f9c6140fcc2890c1da1ad4de3ad2d68c90acbb9f"
# <a id="5"></a> <br>
# ## 8-1 New to Plotly?
# Plotly, also known by its URL, Plot.ly, is a technical computing company headquartered in Montreal, Quebec, that develops online data analytics and visualization tools. Plotly provides online graphing, analytics, and statistics tools for individuals and collaboration, as well as scientific graphing libraries for Python, R, MATLAB, Perl, Julia, Arduino, and REST.

# + _uuid="a763c7f03b88cd1679931bc151c4f8f98006b3cb"
# example for plotly
import plotly.offline as py
import plotly.graph_objs as go
py.init_notebook_mode(connected=True)
from plotly import tools
import plotly.figure_factory as ff
iris = datasets.load_iris()
X = iris.data[:, :2]  # we only take the first two features.
Y = iris.target

x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
trace = go.Scatter(x=X[:, 0],
                   y=X[:, 1],
                   mode='markers',
                   marker=dict(color=np.random.randn(150),
                               size=10,
                               colorscale='Viridis',
                               showscale=False))

layout = go.Layout(title='Training Points',
                   xaxis=dict(title='Sepal length',
                            showgrid=False),
                   yaxis=dict(title='Sepal width',
                            showgrid=False),
                  )
 
fig = go.Figure(data=[trace], layout=layout)

# + _uuid="e2031ce099799a53d125b1cc0986be86bf51e0e1"
py.iplot(fig)

# + [markdown] _uuid="7f180dcbf1ac66e1e40aa21551467d0cfeb2f974"
# <a id="5"></a> <br>
# ## 8-2 Plotly Offline from Command Line
# You can plot your graphs from a python script from command line. On executing the script, it will open a web browser with your Plotly Graph drawn.

# + _uuid="cd95c77032e07526708e2339b0c4b862419286bb"
import plotly.graph_objs as go

plot([go.Scatter(x=[1, 2, 3], y=[3, 1, 6])])

# + [markdown] _uuid="77a55889cd7f12ec03adc2f8328548213aabfa14"
# <a id="5"></a> <br>
# ## 8-3 Generating Offline Graphs within Jupyter Notebook
# You can also plot your graphs offline inside a Jupyter Notebook Environment. First you need to initiate the Plotly Notebook mode as below:

# + _uuid="7eaf6d6cc7809ed749c29f11aa153a7767fff21c"
init_notebook_mode(connected=True)


# + [markdown] _uuid="3f536d2afd2f215c927bd2b6adbac28373fc9106"
# Run at the start of every ipython notebook to use plotly.offline. This injects the plotly.js source files into the notebook.
#
#

# + _uuid="2f19a9f0852aa105d632d5d72dd61cb8f6ec7bc7"
iplot([{"x": [1, 2, 3], "y": [3, 1, 6]}])


# + _uuid="242f7a3ed1b231373ef5a2040e8c865e701e4879"
import plotly.graph_objs as go

import numpy as np

x = np.random.randn(2000)
y = np.random.randn(2000)
iplot([go.Histogram2dContour(x=x, y=y, contours=dict(coloring='heatmap')),
       go.Scatter(x=x, y=y, mode='markers', marker=dict(color='white', size=3, opacity=0.3))], show_link=False)

# + [markdown] _uuid="beae557b399fbe7368ab4536ca1a5e074fe74741"
# ## 8-4 Plotting Offline with Cufflinks
#

# + _uuid="d43e390b979b5075e63e6836b26f150e6f9ed910"
import cufflinks as cf
iplot(cf.datagen.lines().iplot(asFigure=True,
                               kind='scatter',xTitle='Dates',yTitle='Returns',title='Returns'))

# + [markdown] _uuid="485e0412b2708e99457d9423449363a647e5e369"
# <a id="60"></a> <br>
# ## 9- Resource
# In the end, I have prepared some resource that can be useful for you.

# + [markdown] _uuid="1cddfa2ed52693fae8532576cc1064c05777354c"
# <a id="61"></a> <br>
# ## 9-1 Courses
# There are a lot of online courses that can help you develop your knowledge, here I have just  listed some of them:
#
# 1. [Machine Learning Certification by Stanford University (Coursera)](https://www.coursera.org/learn/machine-learning/)
#
# 2. [Machine Learning A-Z™: Hands-On Python & R In Data Science (Udemy)](https://www.udemy.com/machinelearning/)
#
# 3. [Deep Learning Certification by <NAME> from deeplearning.ai (Coursera)](https://www.coursera.org/specializations/deep-learning)
#
# 4. [Python for Data Science and Machine Learning Bootcamp (Udemy)](Python for Data Science and Machine Learning Bootcamp (Udemy))
#
# 5. [Mathematics for Machine Learning by Imperial College London](https://www.coursera.org/specializations/mathematics-machine-learning)
#
# 6. [Deep Learning A-Z™: Hands-On Artificial Neural Networks](https://www.udemy.com/deeplearning/)
#
# 7. [Complete Guide to TensorFlow for Deep Learning Tutorial with Python](https://www.udemy.com/complete-guide-to-tensorflow-for-deep-learning-with-python/)
#
# 8. [Data Science and Machine Learning Tutorial with Python – Hands On](https://www.udemy.com/data-science-and-machine-learning-with-python-hands-on/)
#
# 9. [Machine Learning Certification by University of Washington](https://www.coursera.org/specializations/machine-learning)
#
# 10. [Data Science and Machine Learning Bootcamp with R](https://www.udemy.com/data-science-and-machine-learning-bootcamp-with-r/)
# 11. [Creative Applications of Deep Learning with TensorFlow](https://www.class-central.com/course/kadenze-creative-applications-of-deep-learning-with-tensorflow-6679)
# 12. [Neural Networks for Machine Learning](https://www.class-central.com/mooc/398/coursera-neural-networks-for-machine-learning)
# 13. [Practical Deep Learning For Coders, Part 1](https://www.class-central.com/mooc/7887/practical-deep-learning-for-coders-part-1)
# 14. [Machine Learning](https://www.cs.ox.ac.uk/teaching/courses/2014-2015/ml/index.html)
#  <a id="61"></a> <br>
# ## 9-2 Ebooks
# If you love reading , here is **10 free machine learning books**
# 1. [Probability and Statistics for Programmers](http://www.greenteapress.com/thinkstats/)
# 2. [Bayesian Reasoning and Machine Learning](http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/091117.pdf)
# 2. [An Introduction to Statistical Learning](http://www-bcf.usc.edu/~gareth/ISL/)
# 2. [Understanding Machine Learning](http://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/index.html)
# 2. [A Programmer’s Guide to Data Mining](http://guidetodatamining.com/)
# 2. [Mining of Massive Datasets](http://infolab.stanford.edu/~ullman/mmds/book.pdf)
# 2. [A Brief Introduction to Neural Networks](http://www.dkriesel.com/_media/science/neuronalenetze-en-zeta2-2col-dkrieselcom.pdf)
# 2. [Deep Learning](http://www.deeplearningbook.org/)
# 2. [Natural Language Processing with Python](https://www.researchgate.net/publication/220691633_Natural_Language_Processing_with_Python)
# 2. [Machine Learning Yearning](http://www.mlyearning.org/)
#  <a id="1"></a> <br>
# ## 9-3 Cheat Sheets
# Data Science is an ever-growing field, there are numerous tools & techniques to remember. It is not possible for anyone to remember all the functions, operations and formulas of each concept. That’s why we have cheat sheets. But there are a plethora of cheat sheets available out there, choosing the right cheat sheet is a tough task. So, I decided to collect them here
#
# Here I have selected the cheat sheets on the following criteria: comprehensiveness, clarity, and content [5]:
# 1. [Quick Guide to learn Python for Data Science ](https://github.com/mjbahmani/10-steps-to-become-a-data-scientist/blob/master/cheatsheets/Data-Science-in-Python.pdf)
# 1. [Python for Data Science Cheat sheet ](https://github.com/mjbahmani/10-steps-to-become-a-data-scientist/blob/master/cheatsheets/beginners_python_cheat_sheet.pdf)
# 1. [Python For Data Science Cheat Sheet NumPy](https://github.com/mjbahmani/10-steps-to-become-a-data-scientist/blob/master/cheatsheets/Numpy_Python_Cheat_Sheet.pdf)
# 1. [Exploratory Data Analysis in Python]()
# 1. [Data Exploration using Pandas in Python](https://github.com/mjbahmani/10-steps-to-become-a-data-scientist/blob/master/cheatsheets/Data-Exploration-in-Python.pdf)
# 1. [Data Visualisation in Python](https://github.com/mjbahmani/10-steps-to-become-a-data-scientist/blob/master/cheatsheets/data-visualisation-infographics1.jpg)
# 1. [Python For Data Science Cheat Sheet Bokeh](https://github.com/mjbahmani/10-steps-to-become-a-data-scientist/blob/master/cheatsheets/Python_Bokeh_Cheat_Sheet.pdf)
# 1. [Cheat Sheet: Scikit Learn ](https://github.com/mjbahmani/10-steps-to-become-a-data-scientist/blob/master/cheatsheets/Scikit-Learn-Infographic.pdf)
# 1. [MLalgorithms CheatSheet](https://github.com/mjbahmani/10-steps-to-become-a-data-scientist/blob/master/cheatsheets/MLalgorithms-.pdf)
# 1. [Probability Basics  Cheat Sheet ](https://github.com/mjbahmani/10-steps-to-become-a-data-scientist/blob/master/cheatsheets/probability_cheatsheet.pdf)

# + [markdown] _uuid="a4a58d0f0aa8204797f2b0836d6204404280d6f3"
#
# <a id="61"></a> <br>
# ## 10 -Conclusion
# In this Kernel you have got an introduction to the main packages and ideas in the **data scientist's toolbox**.I hope you have had fun with it also, I want to hear your voice to update this kernel.

# + [markdown] _uuid="dc985ec01fe0e62afd495b8ec461359f085071b6"
#
#
# ---------------------------------------------------------------------
# Fork, Run and Follow this kernel on GitHub:
# > ###### [ GitHub](https://github.com/mjbahmani/10-steps-to-become-a-data-scientist)
#
#
# -------------------------------------------------------------------------------------------------------------
#  **I hope you find this kernel helpful and some <font color="red">UPVOTES</font> would be very much appreciated.**
#  
#  -----------
#

# + [markdown] _uuid="3eb9f355dc3885ea4b0f38968a4bf14d20de02fd"
# <a id="62"></a> <br>
# ## 11- References
# 1. [Coursera](https://www.coursera.org/specializations/data-science-python)
# 1. [GitHub](https://github.com/mjbahmani)
# 1. [plot.ly](https://plot.ly/python/offline/)
# 1. [tutorialspoint](https://www.tutorialspoint.com/python/python_classes_objects.htm)
# 1. [Top 28 Cheat Sheets for Machine Learning](https://www.analyticsvidhya.com/blog/2017/02/top-28-cheat-sheets-for-machine-learning-data-science-probability-sql-big-data/)
# ###### [Go to top](#top)

StarcoderdataJupyterScriptsDedupFiltered