# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/SMSinclair/DS-Unit-1-Sprint-2-Data-Wrangling-and-Storytelling/blob/master/Stephen_Sinclair_LS_DS4_123_Make_Explanatory_Visualizations.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] colab_type="text" id="-8-trVo__vRE"
# _Lambda School Data Science_
#
# # Make Explanatory Visualizations
#
# ### Objectives
#
# - identify misleading visualizations and how to fix them
# - use Seaborn to visualize distributions and relationships with continuous and discrete variables
# - add emphasis and annotations to transform visualizations from exploratory to explanatory
# - remove clutter from visualizations
#
# ### Links
#
# - [How to Spot Visualization Lies](https://flowingdata.com/2017/02/09/how-to-spot-visualization-lies/)
# - [Visual Vocabulary - Vega Edition](http://ft.com/vocabulary)
# - [Choosing a Python Visualization Tool flowchart](http://pbpython.com/python-vis-flowchart.html)
# - [Searborn example gallery](http://seaborn.pydata.org/examples/index.html) & [tutorial](http://seaborn.pydata.org/tutorial.html)
# - [Strong Titles Are The Biggest Bang for Your Buck](http://stephanieevergreen.com/strong-titles/)
# - [Remove to improve (the data-ink ratio)](https://www.darkhorseanalytics.com/blog/data-looks-better-naked)
# - [How to Generate FiveThirtyEight Graphs in Python](https://www.dataquest.io/blog/making-538-plots/)

# + [markdown] id="s-24T844-8qv" colab_type="text"
# # Avoid Misleading Visualizations
#
# Did you find/discuss any interesting misleading visualizations in your Walkie Talkie?

# + [markdown] id="Qzxt9ntsNjs0" colab_type="text"
# ## What makes a visualization misleading?
#
# [5 Ways Writers Use Misleading Graphs To Manipulate You](https://venngage.com/blog/misleading-graphs/)

# + [markdown] id="q7_DUiENNvxk" colab_type="text"
# ## Two y-axes
#
# <img src="https://kieranhealy.org/files/misc/two-y-by-four-sm.jpg" width="800">
#
# Other Examples: 
#  - [Spurious Correlations](https://tylervigen.com/spurious-correlations)
#  - <https://blog.datawrapper.de/dualaxis/>
#  - <https://kieranhealy.org/blog/archives/2016/01/16/two-y-axes/>
#  - <http://www.storytellingwithdata.com/blog/2016/2/1/be-gone-dual-y-axis>

# + [markdown] id="oIijNBDMNv2k" colab_type="text"
# ##  Y-axis doesn't start at zero.
#
# <img src="https://i.pinimg.com/originals/22/53/a9/2253a944f54bb61f1983bc076ff33cdd.jpg" width="600">

# + [markdown] id="ISB2p8vZNv6r" colab_type="text"
# ## Pie Charts are bad
#
# <img src="https://i1.wp.com/flowingdata.com/wp-content/uploads/2009/11/Fox-News-pie-chart.png?fit=620%2C465&ssl=1" width="600">

# + [markdown] id="67CsAzu1NwBJ" colab_type="text"
# ## Pie charts that omit data are extra bad
#  
# - A guy makes a misleading chart that goes viral
#
#  What does this chart imply at first glance? You don't want your user to have to do a lot of work in order to be able to interpret you graph correctly. You want that first-glance conclusions to be the correct ones.
#
#  <img src="https://pbs.twimg.com/media/DiaiTLHWsAYAEEX?format=jpg&name=medium" width="600">
#  
#  <https://twitter.com/michaelbatnick/status/1019680856837849090?lang=en>
#  
# - It gets picked up by overworked journalists (assuming incompetency before malice)
#  
#  <https://www.marketwatch.com/story/this-1-chart-puts-mega-techs-trillions-of-market-value-into-eye-popping-perspective-2018-07-18>
#  
# - Even after the chart's implications have been refuted, it's hard a bad (although compelling) visualization from being passed around.
#
#  <https://www.linkedin.com/pulse/good-bad-pie-charts-karthik-shashidhar/>
#
# **["yea I understand a pie chart was probably not the best choice to present this data."](https://twitter.com/michaelbatnick/status/1037036440494985216)**

# + [markdown] id="FYXmlToEOOTC" colab_type="text"
# ## Pie Charts that compare unrelated things are next-level extra bad
#
# <img src="http://www.painting-with-numbers.com/download/document/186/170403+Legalizing+Marijuana+Graph.jpg" width="600">
#

# + [markdown] id="439l2NTqFDSb" colab_type="text"
# ## And finally, a good pie chart
#
# ![A good pie chart](http://www.graphgraph.com/blog/wp-content/uploads/2011/11/PieChartSmall.png)

# + [markdown] id="IwtMQpY_QFUw" colab_type="text"
# ## Be careful about how you use volume to represent quantities:
#
# radius vs diameter vs volume
#
# <img src="https://static1.squarespace.com/static/5bfc8dbab40b9d7dd9054f41/t/5c32d86e0ebbe80a25873249/1546836082961/5474039-25383714-thumbnail.jpg?format=1500w" width="600">

# + [markdown] id="tTuAWjSBRsc7" colab_type="text"
# ## Don't cherrypick timelines or specific subsets of your data:
#
# <img src="https://wattsupwiththat.com/wp-content/uploads/2019/02/Figure-1-1.png" width="600">
#
# Look how specifically the writer has selected what years to show in the legend on the right side.
#
# <https://wattsupwiththat.com/2019/02/24/strong-arctic-sea-ice-growth-this-year/>
#
# Try the tool that was used to make the graphic for yourself
#
# <http://nsidc.org/arcticseaicenews/charctic-interactive-sea-ice-graph/>
#                                                                                            

# + [markdown] id="Xs13S7p4Srme" colab_type="text"
# ## Use Relative units rather than Absolute Units
#
# <img src="https://imgs.xkcd.com/comics/heatmap_2x.png" width="600">

# + [markdown] id="CIMt5OiuTlrr" colab_type="text"
# ## Avoid 3D graphs unless having the extra dimension is effective
#
# Usually you can Split 3D graphs into multiple 2D graphs
#
# 3D graphs that are interactive can be very cool. (See Plotly and Bokeh)
#
# <img src="https://thumbor.forbes.com/thumbor/1280x868/https%3A%2F%2Fblogs-images.forbes.com%2Fthumbnails%2Fblog_1855%2Fpt_1855_811_o.jpg%3Ft%3D1339592470" width="600">

# + [markdown] id="GATMu9IqUlIj" colab_type="text"
# ## Don't go against typical conventions
#
# <img src="http://www.callingbullshit.org/twittercards/tools_misleading_axes.png" width="600">

# + [markdown] id="g6bKgZ0m_ynS" colab_type="text"
# # Tips for choosing an appropriate visualization:

# + [markdown] id="WtBsVnO4VHiJ" colab_type="text"
# ## Use Appropriate "Visual Vocabulary"
#
# [Visual Vocabulary - Vega Edition](http://ft.com/vocabulary)

# + [markdown] id="H_QM9FHqVT7T" colab_type="text"
# ## What are the properties of your data?
# - Is your primary variable of interest continuous or discrete?
# - Is in wide or long (tidy) format?
# - Does your visualization involve multiple variables?
# - How many dimensions do you need to include on your plot?
#
# Can you express the main idea of your visualization in a single sentence?
#
# How hard does your visualization make the user work in order to draw the intended conclusion?

# + [markdown] id="5EqXxnJeB89_" colab_type="text"
# ## Which Visualization tool is most appropriate? 
#
# [Choosing a Python Visualization Tool flowchart](http://pbpython.com/python-vis-flowchart.html)

# + [markdown] id="5_na7Oy3NGKA" colab_type="text"
# # Making Explanatory Visualizations with Seaborn

# + [markdown] id="ORUwQD6F-VYg" colab_type="text"
# Today we will reproduce this [example by FiveThirtyEight:](https://fivethirtyeight.com/features/al-gores-new-movie-exposes-the-big-flaw-in-online-movie-ratings/)
#
#

# + colab_type="code" id="ya_w5WORGs-n" outputId="c8babe29-832e-477b-9d15-b685d866466e" colab={"base_uri": "https://localhost:8080/", "height": 355}
from IPython.display import display, Image

url = 'https://fivethirtyeight.com/wp-content/uploads/2017/09/mehtahickey-inconvenient-0830-1.png'
example = Image(url=url, width=400)

display(example)

# + [markdown] colab_type="text" id="HP4DALiRG3sC"
# Using this data: https://github.com/fivethirtyeight/data/tree/master/inconvenient-sequel

# + [markdown] colab_type="text" id="HioPkYtUG03B"
# Links
# - [Strong Titles Are The Biggest Bang for Your Buck](http://stephanieevergreen.com/strong-titles/)
# - [Remove to improve (the data-ink ratio)](https://www.darkhorseanalytics.com/blog/data-looks-better-naked)
# - [How to Generate FiveThirtyEight Graphs in Python](https://www.dataquest.io/blog/making-538-plots/)

# + [markdown] colab_type="text" id="0w_iMnQ6-VoQ"
# ## Make prototypes
#
# This  helps us understand the problem

# + colab_type="code" id="5uz0eEaEN-GO" outputId="0cc8cdd8-2290-4863-ca58-ba926cca2d09" colab={"base_uri": "https://localhost:8080/", "height": 285}
# %matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd


plt.style.use('fivethirtyeight')

fake = pd.Series([38, 3, 2, 1, 2, 4, 6, 5, 5, 33], 
                 index=range(1,11))

fake.plot.bar(color='C1', width=0.9);

# + colab_type="code" id="KZ0VLOV8OyRr" outputId="fac02955-6e11-4a5d-83b1-5d9e3720d73a" colab={"base_uri": "https://localhost:8080/", "height": 289}
fake2 = pd.Series(
    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
     2, 2, 2, 
     3, 3, 3,
     4, 4,
     5, 5, 5,
     6, 6, 6, 6,
     7, 7, 7, 7, 7,
     8, 8, 8, 8,
     9, 9, 9, 9, 
     10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10])

fake2.value_counts().sort_index().plot.bar(color='C1', width=0.9);

# + [markdown] colab_type="text" id="mZb3UZWO-q05"
# ## Annotate with text

# + colab_type="code" id="f6U1vswr_uWp" outputId="22326e29-eead-482a-81dc-a7f28048897b" colab={"base_uri": "https://localhost:8080/", "height": 340}
plt.style.use('fivethirtyeight')
fig = plt.figure()
fake = pd.Series([38, 3, 2, 1, 2, 4, 6, 5, 5, 33], 
                 index=range(1, 11))

ax = fake.plot.bar(color='#ef7030', width=0.9)

ax.set(yticks=range(0, 50, 10), facecolor='#f9f9f9')
fig.patch.set_facecolor('#f9f9f9')

plt.ylabel('Percent of total votes', fontsize=9, fontweight='bold')
plt.xlabel('Rating', fontsize=9, fontweight='bold')

ax.text(x=-2, y=46, s="'An Inconvenient Sequel: Truth To Power' is divisive",
        fontsize=12, fontweight='bold')
ax.text(x=-2, y=43, s='IMDb ratings for the film as of Aug. 29',
        fontsize=11)
ax.tick_params(labelrotation=0)

# + id="GY9yqTBqZz-n" colab_type="code" outputId="09510d69-0b23-467c-e0cd-f8072d5f9f6f" colab={"base_uri": "https://localhost:8080/", "height": 355}
display(example)

# + id="Vt2qQYMUZOEE" colab_type="code" outputId="c7ec5659-c959-471b-c989-6d6c35f5e0d3" colab={"base_uri": "https://localhost:8080/", "height": 170}
help(fig.patch.set_facecolor)

# + id="2EwOAbwhaZOi" colab_type="code" outputId="308c52ed-6136-4e58-eacb-68c67916f8f1" colab={"base_uri": "https://localhost:8080/", "height": 986}
help(ax.text)

# + [markdown] colab_type="text" id="x8jRZkpB_MJ6"
# ## Reproduce with real data

# + colab_type="code" id="3SOHJckDUPI8" colab={}
df = pd.read_csv('https://raw.githubusercontent.com/fivethirtyeight/data/master/inconvenient-sequel/ratings.csv')

# + colab_type="code" id="cDltXxhC_yG-" outputId="426b8421-8f9d-424a-8f3c-3364cffa7732" colab={"base_uri": "https://localhost:8080/", "height": 344}
pd.set_option('display.max_columns', 50)
df.head()

# + id="9pCJVeP0c-Sl" colab_type="code" outputId="8f75cc05-5e15-40f4-8b67-596daaa31909" colab={"base_uri": "https://localhost:8080/", "height": 34}
df.shape

# + id="fsOnVdpidBdD" colab_type="code" outputId="bb283adb-fd69-4a28-9a54-ceb36c94b0aa" colab={"base_uri": "https://localhost:8080/", "height": 886}
df.sample(1).T

# + id="_8w-MzXedkZn" colab_type="code" outputId="837f6da2-cb78-47bc-c980-3903797ff910" colab={"base_uri": "https://localhost:8080/", "height": 493}
df.dtypes

# + id="2jKWWOUbdxYa" colab_type="code" outputId="bf85ca59-4629-494b-b734-58c86fc18ba0" colab={"base_uri": "https://localhost:8080/", "height": 136}
df['timestamp'] = pd.to_datetime(df['timestamp'])
df['timestamp'].describe()

# + id="Nm7nUjvjeBI2" colab_type="code" outputId="cbdcabeb-958f-4891-a70e-63983b024cf9" colab={"base_uri": "https://localhost:8080/", "height": 3278}
df.set_index('timestamp', inplace=True)
df['2017-08-29']

# + id="tZ89KA1_eTp6" colab_type="code" outputId="c576a3c9-02f7-4320-fbe1-b53349fce44a" colab={"base_uri": "https://localhost:8080/", "height": 357}
df['category'].value_counts()

# + id="soc9ShqkeyHp" colab_type="code" outputId="76b7ef90-de22-486f-9963-76df20bf2119" colab={"base_uri": "https://localhost:8080/", "height": 1540}
lastday = df['2017-08-29']
lastday_filtered = lastday[lastday['category'] == 'IMDb users']
lastday_filtered.tail(30)

# + id="7nfIlG32fq36" colab_type="code" outputId="32f62a2b-fcb1-4adb-c272-8b308f403947" colab={"base_uri": "https://localhost:8080/", "height": 313}
lastday_filtered.respondents.plot()

# + id="as16xbw8f5UM" colab_type="code" outputId="7cf58968-5716-4aa9-acdb-2c8da78c189a" colab={"base_uri": "https://localhost:8080/", "height": 855}
final = lastday_filtered.tail(1)
final.T

# + id="PrKJBavdgHyG" colab_type="code" outputId="49990081-fb78-4476-c52e-db4103470dc6" colab={"base_uri": "https://localhost:8080/", "height": 111}
pct_columns = ['1_pct', '2_pct', '3_pct', '4_pct', '5_pct',
               '6_pct', '7_pct', '8_pct', '9_pct', '10_pct']
final[pct_columns]

# + id="Kvlf94G6gL9g" colab_type="code" outputId="31dcc417-a8c1-4a29-a6d1-bb70f1e45155" colab={"base_uri": "https://localhost:8080/", "height": 359}
plot_data = final[pct_columns].T
plot_data.index = range(1, 11)
plot_data

# + id="iNg661q2gbKU" colab_type="code" outputId="964c970d-ddc6-4e10-c9a9-48cdb950b054" colab={"base_uri": "https://localhost:8080/", "height": 361}
plt.style.use('fivethirtyeight')

agfont = {'fontname': 'Helvetica'}

fig = plt.figure()
#fake = pd.Series([38, 3, 2, 1, 2, 4, 6, 5, 5, 33], 
#                 index=range(1, 11))

ax = plot_data.plot.bar(color='#ef7030', width=0.9, legend=False)

ax.set(yticks=range(0, 50, 10), facecolor='#f9f9f9')
fig.patch.set_facecolor('#f9f9f9')

plt.ylabel('Percent of total votes', fontsize=9, fontweight='bold')
plt.xlabel('Rating', fontsize=9, fontweight='bold', labelpad=10)
#ax.get_legend().remove()

ax.text(x=-2, y=46, s="'An Inconvenient Sequel: Truth To Power' is divisive",
        fontsize=12, fontweight='bold')
ax.text(x=-2, y=43, s='IMDb ratings for the film as of Aug. 29',
        fontsize=11)
ax.tick_params(labelrotation=0)

# + id="KxVz7PAFg3z3" colab_type="code" outputId="bf937323-c3fa-4290-83a0-dc869aa70d47" colab={"base_uri": "https://localhost:8080/", "height": 355}
display(example)

# + [markdown] colab_type="text" id="NMEswXWh9mqw"
# # ASSIGNMENT
#
# Replicate the lesson code. I recommend that you [do not copy-paste](https://docs.google.com/document/d/1ubOw9B3Hfip27hF2ZFnW3a3z9xAgrUDRReOEo-FHCVs/edit).
#
# # STRETCH OPTIONS
#
# #### Reproduce another example from [FiveThityEight's shared data repository](https://data.fivethirtyeight.com/).
#
# For example:
# - [thanksgiving-2015](https://fivethirtyeight.com/features/heres-what-your-part-of-america-eats-on-thanksgiving/) (try the [`altair`](https://altair-viz.github.io/gallery/index.html#maps) library)
# - [candy-power-ranking](https://fivethirtyeight.com/features/the-ultimate-halloween-candy-power-ranking/) (try the [`statsmodels`](https://www.statsmodels.org/stable/index.html) library)
# - or another example of your choice!
#
# #### Make more charts!
#
# Choose a chart you want to make, from [Visual Vocabulary - Vega Edition](http://ft.com/vocabulary).
#
# Find the chart in an example gallery of a Python data visualization library:
# - [Seaborn](http://seaborn.pydata.org/examples/index.html)
# - [Altair](https://altair-viz.github.io/gallery/index.html)
# - [Matplotlib](https://matplotlib.org/gallery.html)
# - [Pandas](https://pandas.pydata.org/pandas-docs/stable/visualization.html)
#
# Reproduce the chart. [Optionally, try the "Ben Franklin Method."](https://docs.google.com/document/d/1ubOw9B3Hfip27hF2ZFnW3a3z9xAgrUDRReOEo-FHCVs/edit) If you want, experiment and make changes.
#
# Take notes. Consider sharing your work with your cohort!
#
#
#
#
#
#
#
#

# + id="qateyqpC8_OQ" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 533} outputId="c700f6b1-7bad-49b2-f4c5-67deab1bcee0"
url = 'https://fivethirtyeight.com/wp-content/uploads/2015/11/hickey-side-dish-1.png?w=575'
example2 = Image(url=url)
display(example2)

# + id="PunnGyeN9l-f" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 674} outputId="3825e235-4d77-4143-9297-b2d5f80989cd"
tg = pd.read_csv('https://raw.githubusercontent.com/fivethirtyeight/data/master/thanksgiving-2015/thanksgiving-2015-poll-data.csv')
pd.set_option('display.max_columns', 65)
tg.head()

# + id="FCuHr38zClVC" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="80eb8884-c95e-43d3-f0aa-1708c2961064"
tg.shape

# + id="OMFGDvTu94wp" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 846} outputId="cad72ea5-2204-41d0-8a97-308566c0b5d6"
tg.groupby(['US Region']).agg('count')

# + id="O53jkEPYDdG8" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 204} outputId="66f6da5a-4c25-48e4-8853-e2dc8c76f6e2"
states = pd.read_csv('https://raw.githubusercontent.com/cphalpert/census-regions/master/us%20census%20bureau%20regions%20and%20divisions.csv')
states.head()

# + id="XqdLvNAJFn2e" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="e8f4f523-bdd5-4e5d-e0e5-c435f990ed9e"
import plotly
plotly.__version__

# + id="JzHL6-5SVV4S" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 204} outputId="b5a0b4e5-356f-40da-93f0-cd233be989ed"
candy = pd.read_csv('https://raw.githubusercontent.com/SMSinclair/538/master/candy-data.csv')
candy.head()


# + id="MVpWWLaqVqxi" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="02048420-9391-4a01-a777-7867431bf221"
candy.shape

# + id="bcCrEcR-b248" colab_type="code" colab={}
candy.competitorname = candy.competitorname.str.replace("Õ", "'")


# + id="xo6ogquNV3Wl" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 204} outputId="2df5fa85-9de2-4337-901d-0f5fce2baf07"
s_candy = candy.sort_values(by='winpercent', ascending=False)
s_candy.head()

# + id="4_wECoxDQU66" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 1432} outputId="75bea1f6-39d4-45bf-81cd-8e9e9edf6d6b"
fig, ax = plt.subplots(figsize=(3,17))

y_pos = np.arange(len(s_candy['competitorname']))


ax.barh(y_pos, s_candy['winpercent'], align='center',
        color='green', ecolor='black')
ax.set_yticks(y_pos)
ax.set_yticklabels(s_candy['competitorname'])
ax.invert_yaxis()  # labels read top-to-bottom
ax.set_xlabel('win percentage')
ax.set_title('How often did a fun-sized candy of a given type win its\n matchups against the rest of the field?')

plt.savefig('candy.png')
plt.show()

# + id="umNXg-CqTa9l" colab_type="code" colab={}


StarcoderdataJupyterScriptsDedupFiltered