# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/connorpheraty/DS-Unit-1-Sprint-3-Data-Storytelling/blob/master/Connor_Heraty_LS_DS3_224_Sequence_your_narrative.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] colab_type="text" id="JbDHnhet8CWy"
# _Lambda School Data Science_
#
# # Sequence your narrative
#
# Today we will create a sequence of visualizations inspired by [<NAME>'s 200 Countries, 200 Years, 4 Minutes](https://www.youtube.com/watch?v=jbkSRLYSojo).
#
# Using this [data from Gapminder](https://github.com/open-numbers/ddf--gapminder--systema_globalis/):
# - [Income Per Person (GDP Per Capital, Inflation Adjusted) by Geo & Time](https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--income_per_person_gdppercapita_ppp_inflation_adjusted--by--geo--time.csv)
# - [Life Expectancy (in Years) by Geo & Time](https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--life_expectancy_years--by--geo--time.csv)
# - [Population Totals, by Geo & Time](https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--population_total--by--geo--time.csv)
# - [Entities](https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--entities--geo--country.csv)
# - [Concepts](https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--concepts.csv)

# + [markdown] colab_type="text" id="zyPYtsY6HtIK"
# Objectives
# - sequence multiple visualizations
# - combine qualitative anecdotes with quantitative aggregates
#
# Links
# - [<NAME>’s TED talks](https://www.ted.com/speakers/hans_rosling)
# - [Spiralling global temperatures from 1850-2016](https://twitter.com/ed_hawkins/status/729753441459945474)
# - "[The Pudding](https://pudding.cool/) explains ideas debated in culture with visual essays."
# - [A Data Point Walks Into a Bar](https://lisacharlotterost.github.io/2016/12/27/datapoint-in-bar/): a thoughtful blog post about emotion and empathy in data storytelling

# + [markdown] colab_type="text" id="SxTJBgRAW3jD"
# ## Make a plan
#
# #### How to present the data?
#
# Variables --> Visual Encodings
# - Income --> x
# - Lifespan --> y
# - Region --> color
# - Population --> size
# - Year --> animation frame (alternative: small multiple)
# - Country --> annotation
#
# Qualitative --> Verbal
# - Editorial / contextual explanation --> audio narration (alternative: text)
#
#
# #### How to structure the data?
#
# | Year | Country | Region   | Income | Lifespan | Population |
# |------|---------|----------|--------|----------|------------|
# | 1818 | USA     | Americas | ###    | ##       | #          |
# | 1918 | USA     | Americas | ####   | ###      | ##         |
# | 2018 | USA     | Americas | #####  | ###      | ###        |
# | 1818 | China   | Asia     | #      | #        | #          |
# | 1918 | China   | Asia     | ##     | ##       | ###        |
# | 2018 | China   | Asia     | ###    | ###      | #####      |
#

# + [markdown] colab_type="text" id="3ebEjShbWsIy"
# ## Upgrade Seaborn
#
# Make sure you have at least version 0.9.0.
#
# In Colab, go to **Restart runtime** after you run the `pip` command.

# + colab_type="code" id="4RSxbu7rWr1p" outputId="6c602db8-2d88-4a01-a2a2-d632944cb314" colab={"base_uri": "https://localhost:8080/", "height": 446}
# !pip install --upgrade seaborn

# + colab_type="code" id="5sQ0-7JUWyN4" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="6bcb4ae0-6f1a-4321-9870-99d3e690b146"
import seaborn as sns
sns.__version__

# + [markdown] colab_type="text" id="S2dXWRTFTsgd"
# ## More imports

# + colab_type="code" id="y-TgL_mA8OkF" colab={}
# %matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

# + [markdown] colab_type="text" id="CZGG5prcTxrQ"
# ## Load & look at data

# + colab_type="code" id="-uE25LHD8CW0" colab={}
income = pd.read_csv('https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--income_per_person_gdppercapita_ppp_inflation_adjusted--by--geo--time.csv')

# + colab_type="code" id="gg_pJslMY2bq" colab={}
lifespan = pd.read_csv('https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--life_expectancy_years--by--geo--time.csv')

# + colab_type="code" id="F6knDUevY-xR" colab={}
population = pd.read_csv('https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--population_total--by--geo--time.csv')

# + colab_type="code" id="hX6abI-iZGLl" colab={}
entities = pd.read_csv('https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--entities--geo--country.csv')

# + colab_type="code" id="AI-zcaDkZHXm" colab={}
concepts = pd.read_csv('https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--concepts.csv')

# + colab_type="code" id="EgFw-g0nZLJy" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="75129cb4-6484-4c25-90d0-9e97d4b186cb"
income.shape, lifespan.shape, population.shape, entities.shape, concepts.shape

# + colab_type="code" id="I-T62v7FZQu5" colab={"base_uri": "https://localhost:8080/", "height": 204} outputId="c85436b5-b38b-4dd8-8e94-da78931ac193"
income.head()

# + colab_type="code" id="2zIdtDESZYG5" colab={"base_uri": "https://localhost:8080/", "height": 204} outputId="75d42b48-f6a9-4023-e33f-8767d0f39eed"
lifespan.head()

# + colab_type="code" id="58AXNVMKZj3T" colab={"base_uri": "https://localhost:8080/", "height": 204} outputId="a083e87c-d484-4300-e4ad-a3a27e108664"
population.head()

# + colab_type="code" id="0ywWDL2MZqlF" colab={"base_uri": "https://localhost:8080/", "height": 258} outputId="59602c4f-44ab-4d22-a7d6-d63828638310"
pd.options.display.max_columns = 500
entities.head()

# + colab_type="code" id="mk_R0eFZZ0G5" colab={"base_uri": "https://localhost:8080/", "height": 513} outputId="c5721f49-3209-40e4-b4cb-b1d48133484b"
concepts.head()

# + [markdown] colab_type="text" id="6HYUytvLT8Kf"
# ## Merge data

# + [markdown] colab_type="text" id="dhALZDsh9n9L"
# https://github.com/pandas-dev/pandas/blob/master/doc/cheatsheet/Pandas_Cheat_Sheet.pdf

# + colab_type="code" id="A-tnI-hK6yDG" colab={}
# We want to merge lifespan, income, and population

# + id="GJLICHAuMzpP" colab_type="code" colab={}
df = pd.merge(income, lifespan, on=['geo', 'time'], )

# + id="IhGrTxpwNIqS" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 204} outputId="8973898e-f087-4996-8c12-d6ef75e657ff"
df = pd.merge(df, population, on=['geo','time'])
df.head()

# + id="PV-X_aH-Qb3z" colab_type="code" colab={}
df = pd.merge(df, entities[['country', 'gapminder_list', 'world_6region']],
              how='inner', left_on='geo', right_on='country')

# + id="RHoNRXb2CrYm" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 204} outputId="698a433a-54c6-4ef3-b45c-9f6f1c1123cc"
df.head()

# + id="3tT7HGHACsls" colab_type="code" colab={}
df = df.drop(['geo','country'], axis=1)

# + id="Av-JSYmlC2g8" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 204} outputId="3c351170-edd0-486d-c68a-3ca0c3e887d2"
df = df.rename(columns={'time':'year',
                       'income_per_person_gdppercapita_ppp_inflation_adjusted' : 'income',
                       'life_expectancy_years': 'lifespan',
                       'population_total' : 'population' , 
                       'gapminder_list' : 'country' ,
                       'world_6region': 'region'
                       })
df.tail()

# + [markdown] colab_type="text" id="4OdEr5IFVdF5"
# ## Explore data

# + colab_type="code" id="4IzXea0T64x4" colab={"base_uri": "https://localhost:8080/", "height": 297} outputId="d3978127-ee6a-4f41-8077-cf920b24b9d9"
df.describe()

# + id="DamFrZSJEx0B" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 102} outputId="292705af-4792-4706-e91b-82b19665bf53"
df.skew()

# + id="lSOVzdeTFfFX" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 173} outputId="93234a69-d503-45c9-d3fe-d1ded2462f62"
df.describe(exclude=np.number)

# + id="o3kLHBK3FkxX" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 680} outputId="da68f6d5-f4ab-4630-c28e-2dce85677b3f"
df.country.unique()

# + id="fckVffzKFsTw" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 142} outputId="4f76a163-1bac-4dcb-a183-5463a4e1dd6f"
usa = df[df.country == 'United States']
usa[usa.year.isin([1818, 1918, 2018])]

# + id="JN5Q1lnxGCyY" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 142} outputId="8e575c6d-d9f4-4fd9-d0b5-afbdada6eea4"
china = df[df.country == 'China']
china[china.year.isin([1818, 1918, 2018])]

# + id="u0_3ysNcGonA" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="884eb63f-96a9-4cd9-db6d-d7ef70cae444"
now = df[df.year == 2018]
now.shape

# + id="UrP8pLxUHa_p" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 204} outputId="842a9324-2459-4306-f0e0-87001534f6af"
now.head()

# + [markdown] colab_type="text" id="hecscpimY6Oz"
# ## Plot visualization

# + colab_type="code" id="_o8RmX2M67ai" colab={"base_uri": "https://localhost:8080/", "height": 294} outputId="df4c072d-c215-4c0f-fa05-7a2a3e2f365e"
now.hist();

# + id="ybJ7Sp48HvMh" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 524} outputId="028c2f49-0705-4205-eca3-bd0caf4b044d"
plt.figure(figsize=(20,10))
pd.plotting.scatter_matrix(now)
plt.show();

# + id="XyW6B9WyIJwB" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 798} outputId="cc805ec7-b5ae-4fdf-a659-a0ba51e80101"
import seaborn as sns
sns.set(style="ticks")
sns.pairplot(now, hue='region');

# + id="kxazI30mSEuO" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 365} outputId="5ddf205b-d524-4104-a5b1-c05498fe89b3"
sns.relplot(x='income', y='lifespan', hue='region', size='population', data=now);

# + id="bnPau2M2SsIq" colab_type="code" colab={}
# Log base 10 of 100 = 2
# 10^2 = 100
# "A log is (gives you) an exponent"

# + [markdown] colab_type="text" id="8OFxenCdhocj"
# ## Analyze outliers

# + colab_type="code" id="D59bn-7k6-Io" colab={"base_uri": "https://localhost:8080/", "height": 142} outputId="ae7183b2-a33a-4170-de4d-7a4b222a2afe"
now[now.income > 80000].sort_values(by='income')

# + id="tRuD9RjfS4n3" colab_type="code" colab={}
qatar = now[now.country == 'Qatar']
qatar_income = qatar.income.values[0]
qatar_lifespan = qatar.lifespan.values[0]

# + id="C558EQJBS8nA" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 392} outputId="723e6f7f-9a2d-4adf-e0d8-1a5665a1306c"
plt.figure(figsize=(20,10))
sns.relplot(x='income', y='lifespan', hue='region', size='population', data=now)

plt.text(x=qatar_income - 5000, y=qatar_lifespan + 1, s="Qatar")
plt.title("Qatar has the highest income in 2018")
plt.show()


# + id="JAesmPKoS8pO" colab_type="code" colab={}


# + id="kcq_CuVAS8tl" colab_type="code" colab={}


# + id="_fNjYSkrS8rh" colab_type="code" colab={}


# + [markdown] colab_type="text" id="DNTMMBkVhrGk"
# ## Plot multiple years

# + colab_type="code" id="JkTUmYGF7BQt" colab={"base_uri": "https://localhost:8080/", "height": 372} outputId="3a80fd42-5ae4-4db3-b359-6ddb69859e80"
years = [1818, 1918, 2018]

centuries = df[df.year.isin(years)]

sns.relplot(x='income', y='lifespan', hue='region', size='population',
            col='year', data=centuries)

plt.xscale('log');

# + [markdown] colab_type="text" id="BB1Ki0v6hxCA"
# ## Point out a story

# + colab_type="code" id="eSgZhD3v7HIe" colab={"base_uri": "https://localhost:8080/", "height": 2207} outputId="28ea7c8a-6e9d-4933-caca-c0c6596802fa"
years = [1918, 1938, 1958, 1978, 1998, 2018]

for year in years:
  sns.relplot(x='income', y='lifespan', hue='region', size='population',
            data=df[df.year==year])
  
  plt.xscale('log')
  plt.xlim((150, 150000))
  plt.ylim((20,90))
  plt.title(year)
  plt.axhline(y=50, color='grey')

# + [markdown] id="gkOzO7RpD6We" colab_type="text"
# # ASSIGNMENT
# Replicate the lesson code
#
# # STRETCH OPTIONS
#
# ## 1. Animate!
# - [Making animations work in Google Colaboratory](https://medium.com/lambda-school-machine-learning/making-animations-work-in-google-colaboratory-new-home-for-ml-prototyping-c6147186ae75)
# - [How to Create Animated Graphs in Python](https://towardsdatascience.com/how-to-create-animated-graphs-in-python-bb619cc2dec1)
# - [The Ultimate Day of Chicago Bikeshare](https://chrisluedtke.github.io/divvy-data.html) (Lambda School Data Science student)
#
# ## 2. Work on anything related to your portfolio site / project

StarcoderdataJupyterScriptsDedupFiltered