<jupyter_start><jupyter_text>Copyright 2020 <NAME>, <NAME> and made available under [CC BY-SA](https://creativecommons.org/licenses/by-sa/4.0) for text and [Apache-2.0](http://www.apache.org/licenses/LICENSE-2.0) for code.  Descriptive statistics: Problem solvingLet's practice some of the skills we learned in the last session on descriptive statistics.  Import librariesWe can calculate measures of central tendency and spread using `pandas dataframes`.First thing you need to do is import `pandas` below.[Use the Reference if you've forgotten any of these steps](Reference.html).<jupyter_code>import pandas as pd

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="zq]t,,UR`GWj6?Hub9%a">pd</variable></variables><block type="importAs" id="o[3wY[w:R*b$p^Ow/ZZw" x="125" y="352"><field name="libraryName">pandas</field><field name="libraryAlias" id="zq]t,,UR`GWj6?Hub9%a">pd</field></block></xml><jupyter_output><empty_output><jupyter_text>Load dataWe're going to work with data from before, on flowers.Load the CSV file `"datasets/flower-data-2020.csv"` into a dataframe.<jupyter_code>dataframe = pd.read_csv('datasets/flower-data-2020.csv')

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="[V~uW+0L/4GW;45ulv+l">dataframe</variable><variable id="zq]t,,UR`GWj6?Hub9%a">pd</variable></variables><block type="variables_set" id="i*U..F)p9r]#n*e(./*x" x="-9" y="195"><field name="VAR" id="[V~uW+0L/4GW;45ulv+l">dataframe</field><value name="VALUE"><block type="varDoMethod" id=":~$/ssC/m-m@jj8K*O4W"><field name="VAR" id="zq]t,,UR`GWj6?Hub9%a">pd</field><field name="MEMBER">read_csv</field><data>pd:read_csv</data><value name="INPUT"><block type="text" id="~Mspn1jJRE!J8ISd3!#V"><field name="TEXT">datasets/flower-data-2020.csv</field></block></value></block></value></block></xml><jupyter_output><empty_output><jupyter_text>Flower mean/median/modeIn the next three cells, use the dataframe to calculate the mean, median, and mode.<jupyter_code>dataframe.mean()

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="[V~uW+0L/4GW;45ulv+l">dataframe</variable></variables><block type="varDoMethod" id="]I:]}wMu=(Gnu*RD$$~9" x="8" y="188"><field name="VAR" id="[V~uW+0L/4GW;45ulv+l">dataframe</field><field name="MEMBER">mean</field><data>dataframe:mean</data></block></xml>
dataframe.median()

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="[V~uW+0L/4GW;45ulv+l">dataframe</variable></variables><block type="varDoMethod" id="]I:]}wMu=(Gnu*RD$$~9" x="8" y="188"><field name="VAR" id="[V~uW+0L/4GW;45ulv+l">dataframe</field><field name="MEMBER">median</field><data>dataframe:median</data></block></xml>
dataframe.mode()

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="[V~uW+0L/4GW;45ulv+l">dataframe</variable></variables><block type="varDoMethod" id="]I:]}wMu=(Gnu*RD$$~9" x="8" y="188"><field name="VAR" id="[V~uW+0L/4GW;45ulv+l">dataframe</field><field name="MEMBER">mode</field><data>dataframe:mode</data></block></xml><jupyter_output><empty_output><jupyter_text>**QUESTION:**How are the mean, median, and mode different here?Write your answer in the cell below. **ANSWER: (click here to edit)***In this case, the mean, median, and mode are almost the same* **QUESTION:**Why do the mean and the median have values for `Size`, but the mode has values for `PetalColor`, `PetalShape`, and `Size`?Write your answer in the cell below. **ANSWER: (click here to edit)***`Size` is numeric (actually ordinal), so it is the only variable Python will calculate mean and median for. It actually is incorrect to calculate mean because it is ordinal, but Python isn't smart enough to know that because we didn't tell it our data was ordinal. This is a good example of being careful with computers - they will do exactly what we tell them to do even if it is incorrect. It is up to us to make sure we understand the reasoning behind what we are doing so we can recognize when the computer gives us garbage answers.*  Outlier effectsLet's look at some different data: a students quiz grades across a course.Suppose a student received the following grades:- 0- 85- 82- 91- 82In the next cell, make a new dataframe (with a new variable) using this list of numbers (in a double list).<jupyter_code>grades = pd.DataFrame([0, 85, 82, 91, 82])

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="0Z~AP{;(K9p1).M_No)5">grades</variable><variable id="zq]t,,UR`GWj6?Hub9%a">pd</variable></variables><block type="variables_set" id="^,Y2E?u8,{P0eK?4*v(^" x="109" y="412"><field name="VAR" id="0Z~AP{;(K9p1).M_No)5">grades</field><value name="VALUE"><block type="varCreateObject" id=".`?GJi9%01TcebUauJK?"><field name="VAR" id="zq]t,,UR`GWj6?Hub9%a">pd</field><field name="MEMBER">DataFrame</field><data>pd:DataFrame</data><value name="INPUT"><block type="lists_create_with" id=":X:+~0D).,#BVd$qbNJy"><mutation items="1"></mutation><value name="ADD0"><block type="lists_create_with" id="Ec(e*=i[/o*JY1wPJ3b)"><mutation items="5"></mutation><value name="ADD0"><block type="math_number" id="rcV5tFvv,4uV3n%R]~7%"><field name="NUM">0</field></block></value><value name="ADD1"><block type="math_number" id="sb#uTkds?[6oppKnWe6a"><field name="NUM">85</field></block></value><value name="ADD2"><block type="math_number" id="]5VLtY%^_Vy#5w;#T_V0"><field name="NUM">82</field></block></value><value name="ADD3"><block type="math_number" id="wPfzb`0n;_XZ~_;v|w{H"><field name="NUM">91</field></block></value><value name="ADD4"><block type="math_number" id="WvJsD9l8`Y]rNR~fMT/N"><field name="NUM">82</field></block></value></block></value></block></value></block></value></block></xml><jupyter_output><empty_output><jupyter_text>Now `print` the mean, median, and mode of this grades dataframe in the next cell.<jupyter_code>print(grades.median())
print(grades.mean())
print(grades.mode())

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="0Z~AP{;(K9p1).M_No)5">grades</variable></variables><block type="text_print" id="`jn6J~kaejV!Cx!s{zP-" x="111" y="117"><value name="TEXT"><shadow type="text" id="X+4}*qR3]Q;+8cTr}T0?"><field name="TEXT">abc</field></shadow><block type="varDoMethod" id="Pl:^oR,*DCGup|yt@jhP"><field name="VAR" id="0Z~AP{;(K9p1).M_No)5">grades</field><field name="MEMBER">median</field><data>grades:median</data></block></value><next><block type="text_print" id=",V)HMF*rq4(p2J4=%*g^"><value name="TEXT"><shadow type="text"><field name="TEXT">abc</field></shadow><block type="varDoMethod" id="{^HKlDM0I%-#.W.[A$z."><field name="VAR" id="0Z~AP{;(K9p1).M_No)5">grades</field><field name="MEMBER">mean</field><data>grades:mean</data></block></value><next><block type="text_print" id="{OV%_W4ZT16c$9VH1k+Q"><value name="TEXT"><shadow type="text"><field name="TEXT">abc</field></shadow><block type="varDoMethod" id="(ObS8~x]Bf^k/pph|KA2"><field name="VAR" id="0Z~AP{;(K9p1).M_No)5">grades</field><field name="MEMBER">mode</field><data>grades:mode</data></block></value></block></next></block></next></block></xml><jupyter_output>0    82.0
dtype: float64
0    68.0
dtype: float64
    0
0  82<jupyter_text>In the cell below, copy the blocks where you created the grades dataframe but remove the `0` from the list.<jupyter_code>grades = pd.DataFrame([85, 82, 91, 82])

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="0Z~AP{;(K9p1).M_No)5">grades</variable><variable id="zq]t,,UR`GWj6?Hub9%a">pd</variable></variables><block type="variables_set" id="^,Y2E?u8,{P0eK?4*v(^" x="109" y="412"><field name="VAR" id="0Z~AP{;(K9p1).M_No)5">grades</field><value name="VALUE"><block type="varCreateObject" id=".`?GJi9%01TcebUauJK?"><field name="VAR" id="zq]t,,UR`GWj6?Hub9%a">pd</field><field name="MEMBER">DataFrame</field><data>pd:DataFrame</data><value name="INPUT"><block type="lists_create_with" id=":X:+~0D).,#BVd$qbNJy"><mutation items="1"></mutation><value name="ADD0"><block type="lists_create_with" id="Ec(e*=i[/o*JY1wPJ3b)"><mutation items="4"></mutation><value name="ADD0"><block type="math_number" id="sb#uTkds?[6oppKnWe6a"><field name="NUM">85</field></block></value><value name="ADD1"><block type="math_number" id="]5VLtY%^_Vy#5w;#T_V0"><field name="NUM">82</field></block></value><value name="ADD2"><block type="math_number" id="wPfzb`0n;_XZ~_;v|w{H"><field name="NUM">91</field></block></value><value name="ADD3"><block type="math_number" id="WvJsD9l8`Y]rNR~fMT/N"><field name="NUM">82</field></block></value></block></value></block></value></block></value></block></xml><jupyter_output><empty_output><jupyter_text>Now `print` the mean, median, and mode of this grades dataframe (without `0`) in the next cell.<jupyter_code>print(grades.median())
print(grades.mean())
print(grades.mode())

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="0Z~AP{;(K9p1).M_No)5">grades</variable></variables><block type="text_print" id="`jn6J~kaejV!Cx!s{zP-" x="111" y="117"><value name="TEXT"><shadow type="text" id="X+4}*qR3]Q;+8cTr}T0?"><field name="TEXT">abc</field></shadow><block type="varDoMethod" id="Pl:^oR,*DCGup|yt@jhP"><field name="VAR" id="0Z~AP{;(K9p1).M_No)5">grades</field><field name="MEMBER">median</field><data>grades:median</data></block></value><next><block type="text_print" id=",V)HMF*rq4(p2J4=%*g^"><value name="TEXT"><shadow type="text"><field name="TEXT">abc</field></shadow><block type="varDoMethod" id="{^HKlDM0I%-#.W.[A$z."><field name="VAR" id="0Z~AP{;(K9p1).M_No)5">grades</field><field name="MEMBER">mean</field><data>grades:mean</data></block></value><next><block type="text_print" id="{OV%_W4ZT16c$9VH1k+Q"><value name="TEXT"><shadow type="text"><field name="TEXT">abc</field></shadow><block type="varDoMethod" id="(ObS8~x]Bf^k/pph|KA2"><field name="VAR" id="0Z~AP{;(K9p1).M_No)5">grades</field><field name="MEMBER">mode</field><data>grades:mode</data></block></value></block></next></block></next></block></xml><jupyter_output>0    83.5
dtype: float64
0    85.0
dtype: float64
    0
0  82<jupyter_text>**QUESTION:**- How did the mean, median, and mode change after you dropped the zero?- What does dropping the zero mean to the student who took this class?- Which measures are most affected/least affected by outliers?Write your answer in the cell below. **ANSWER: (click here to edit)***Median changed very slightly and mode did not change at all.Mean changed dramatically from 68 to 85.This means the student would go from a D to a B in the class.This is an example of how an outlier can have a strong effect on the mean, and how the other measures of central tendency are resistant to outliers.*  Dataframe samplingLet's switch back to the flower dataframe.In the cell below, get the five number summary of the dataframe (i.e. `describe`).<jupyter_code>dataframe.describe()

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="[V~uW+0L/4GW;45ulv+l">dataframe</variable></variables><block type="varDoMethod" id="O3QcP4Y+iT~?:8x5/Mld" x="8" y="188"><field name="VAR" id="[V~uW+0L/4GW;45ulv+l">dataframe</field><field name="MEMBER">describe</field><data>dataframe:describe</data></block></xml><jupyter_output><empty_output><jupyter_text>SamplingNow let's take samples from the flower dataframe, get their five number summaries, and compare those to each other and the entire dataframe.In the cell below, take a sample of `10` from the flowers dataframe, store it in a variable, and display the sample.<jupyter_code>sample = dataframe.sample(10)

sample

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="X~V70-15t9L^b}[IjBoW">sample</variable><variable id="[V~uW+0L/4GW;45ulv+l">dataframe</variable></variables><block type="variables_set" id="X2Q0lhlIuB;M-}%}kt0@" x="114" y="312"><field name="VAR" id="X~V70-15t9L^b}[IjBoW">sample</field><value name="VALUE"><block type="varDoMethod" id="}THa7gMDOBGv~{O0J-}v"><field name="VAR" id="[V~uW+0L/4GW;45ulv+l">dataframe</field><field name="MEMBER">sample</field><data>dataframe:sample</data><value name="INPUT"><block type="math_number" id="x6aHWTYmPfX=;4W1C.io"><field name="NUM">10</field></block></value></block></value></block><block type="variables_get" id="*XWdVPs3p=}r`r6WYNWV" x="112" y="371"><field name="VAR" id="X~V70-15t9L^b}[IjBoW">sample</field></block></xml><jupyter_output><empty_output><jupyter_text>In the cell below, repeat what you did above (take a sample of `10` and store it in a variable) but add below that a block doing a five number summary of the sample.<jupyter_code>sample = dataframe.sample(10)

sample.describe()

#<xml xmlns="https://developers.google.com/blockly/xml"><variables><variable id="X~V70-15t9L^b}[IjBoW">sample</variable><variable id="[V~uW+0L/4GW;45ulv+l">dataframe</variable></variables><block type="variables_set" id="x.L#2+Kmq01S1!FI|%*2" x="16" y="135"><field name="VAR" id="X~V70-15t9L^b}[IjBoW">sample</field><value name="VALUE"><block type="varDoMethod" id="!L!ZRT9j}{ngeakBRIE!"><field name="VAR" id="[V~uW+0L/4GW;45ulv+l">dataframe</field><field name="MEMBER">sample</field><data>dataframe:sample</data><value name="INPUT"><block type="math_number" id="VWJ^n9kSRcY:)U5Kmi;8"><field name="NUM">10</field></block></value></block></value></block><block type="varDoMethod" id="p9K6asIXwdef{s%{s[|Q" x="8" y="188"><field name="VAR" id="X~V70-15t9L^b}[IjBoW">sample</field><field name="MEMBER">describe</field><data>sample:describe</data></block></xml><jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup