<jupyter_start><jupyter_text>Here, you'll learn about the principles of tidy data and more importantly, why you should care about them and how they make subsequent data analysis more efficient. You'll gain first hand experience with reshaping and tidying your data using techniques such as pivoting and melting.  Recognizing tidy data  Reshaping your data using meltMelting data is the process of turning columns of your data into rows of data. Consider the DataFrames from the previous exercise. In the tidy DataFrame, the variables Ozone, Solar.R, Wind, and Temp each had their own column. If, however, you wanted these variables to be in rows instead, you could melt the DataFrame. In doing so, however, you would make the data untidy! This is important to keep in mind: Depending on how your data is represented, you will have to reshape it differently (e.g., this could make it easier to plot values).In this exercise, you will practice melting a DataFrame using pd.melt(). There are two parameters you should be aware of: id_vars and value_vars. The id_vars represent the columns of the data you do not want to melt (i.e., keep it in its current shape), while the value_vars represent the columns you do wish to melt into rows. By default, if no value_vars are provided, all columns not set in the id_vars will be melted. This could save a bit of typing, depending on the number of columns that need to be melted.Load the (tidy) DataFrame airquality and then your job is to melt its Ozone, Solar.R, Wind, and Temp columns into rows. Later in this chapter, you'll learn how to bring this melted DataFrame back into a tidy form<jupyter_code>from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('airquality.csv')
df.head()
# Use pd.melt() to melt the Ozone, Solar.R, Wind, and Temp columns of airquality into rows. Do this by using id_vars to specify the columns you do not wish to melt: 'Month' and 'Day'

df_melt = pd.melt(frame = df, id_vars = ['Month', 'Day'])
df_melt.head()
df.head()
df_melt.head()<jupyter_output><empty_output><jupyter_text>Customizing melted dataWhen melting DataFrames, it would be better to have column names more meaningful than variable and value (the default names used by pd.melt()).The default names may work in certain situations, but it's best to always have data that is self explanatory.You can rename the variable column by specifying an argument to the var_name parameter, and the value column by specifying an argument to the value_name parameter. You will now practice doing exactly this. __Instruction__- Melt the columns of airquality with the defaultvariablecolumn renamed to'measurement'and the defaultvaluecolumn renamed to'reading'. You can do this by specifying, respectively, the `varname` and `valuename` parameters.<jupyter_code>df_melt = pd.melt(frame = df, id_vars = ['Month', 'Day'], var_name = 'measurement', value_name = 'reading')
df_melt.head()<jupyter_output><empty_output><jupyter_text>Pivot dataPivoting data is the opposite of melting it. Remember the tidy form that the airquality DataFrame was in before you melted it? You'll now begin pivoting it back into that form using the .pivot_table() method!While melting takes a set of columns and turns it into a single column, pivoting will create a new column for each unique value in a specified column..pivot_table() has an index parameter which you can use to specify the columns that you don't want pivoted: It is similar to the id_vars parameter of pd.melt(). Two other parameters that you have to specify are columns (the name of the column you want to pivot), and values (the values to be used when the column is pivoted). - Pivot df_melt by using .pivot_table() with the rows indexed by 'Month' and 'Day', the columns indexed by 'measurement', and the values populated with 'reading'<jupyter_code>df_melt.head()
df_pivot = df_melt.pivot_table(index = ['Month', 'Day'], columns = 'measurement', values = 'reading')
df_pivot.head()<jupyter_output><empty_output><jupyter_text>Resetting the index of a DataFrameAfter pivoting airquality_melt in the previous exercise, you didn't quite get back the original DataFrame.What you got back instead was a pandas DataFrame with a hierarchical index (also known as a MultiIndex).There's a very simple method you can use to get back the original DataFrame from the pivoted DataFrame: .reset_index()<jupyter_code># print index of df_pivot 
df_pivot.index
# reset the index of df_pivot using it's reset_index() method
df_pivot_reset = df_pivot.reset_index()
df_pivot_reset.index
df_pivot_reset.head()<jupyter_output><empty_output><jupyter_text>Pivoting duplicate valuesSo far, you've used the .pivot_table() method when there are multiple index values you want to hold constant during a pivot.Let's say your data collection method accidentally duplicated your dataset.You'll see that by using .pivot_table() and the aggfunc parameter, you can not only reshape your data, but also remove duplicates. Finally, you can then flatten the columns of the pivoted DataFrame using .reset_index()<jupyter_code>df_dup = pd.concat([df, df])
df_dup.columns
df_dup
df_dup = pd.concat([df, df], ignore_index=True)
df_dup
df_dup_melt = pd.melt(frame = df_dup, id_vars = ['Month', 'Day'], var_name = 'measurement', value_name = 'reading')
df_melt.head()
df_dup_pivot = df_dup_melt.pivot_table(index=['Month', 'Day'], columns='measurement', values='reading', aggfunc=np.mean)
df_dup_pivot
df_dup_pivot_reset = df_dup_pivot.reset_index()
df_dup_pivot_reset.head()<jupyter_output><empty_output><jupyter_text>Beyond melt and pivot  Splitting a column with .strload the dataset tb.csv,  consisting of case counts of tuberculosis by country, year, gender, and age group into a dataframe named tb<jupyter_code>tb = pd.read_csv('tb.csv')
tb.head()<jupyter_output><empty_output><jupyter_text>__In this exercise, you're going to tidy the `m014` column, which represents males aged 0-14 years of age. In order to parse this value, you need to extract the first letter into a new column for gender, and the rest into a column for age_group. Here, since you can parse values by position, you can take advantage of pandas' vectorized string slicing by using the str attribute of columns of type object.__ __Instructions:__- Melt tb keeping 'country' and 'year' fixed.- Create a 'gender' column by slicing the first letter of the variable column of tb_melt.- Create an 'age_group' column by slicing the rest of the variable column of tb_melt.<jupyter_code># Melt tb
tb_melt = pd.melt(frame = tb, id_vars = ['country', 'year'])
tb_melt.head()
# create the gender column
tb_melt['gender'] = tb_melt.variable.str[0]
# create an age-group column
tb_melt['age_group'] = tb_melt.variable.str[1:]
tb_melt.head()<jupyter_output><empty_output><jupyter_text>Splitting a column with .split() and .get()Another common way multiple variables are stored in columns is with a delimiter. You'll learn how to deal with such cases in this exercise, using a dataset consisting of Ebola cases and death counts by state and country.Print the columns of ebola in the IPython Shell using ebola.columns. Notice that the data has column names such as Cases_Guinea and Deaths_Guinea. Here, the underscore _ serves as a delimiter between the first part (cases or deaths), and the second part (country).This time, you cannot directly slice the variable by position as in the previous exercise. You now need to use Python's built-in string method called .split(). By default, this method will split a string into parts separated by a space. However, in this case you want it to split by an underscore. You can do this on Cases_Guinea, for example, using Cases_Guinea.split('_'), which returns the list ['Cases', 'Guinea'].The next challenge is to extract the first element of this list and assign it to a type variable, and the second element of the list to a country variable. You can accomplish this by accessing the str attribute of the column and using the .get() method to retrieve the 0 or 1 index, depending on the part you want. __Instructions__- Melt ebola using `Date` and `Day` as the id_vars, `type_country` as the var_name, and `counts` as the value_name.- Create a column called `str_split` by splitting the `type_country` column of ebola_melt on `'_'`. Note that you will first have   to access the str attribute of type_country before you can use .split().- Create a column called `type` by using the `.get()` method to retrieve index 0 of the `str_split` column of ebola_melt.- Create a column called `country` by using the `.get()` method to retrieve index 1 of the `str_split` column of ebola_melt.<jupyter_code>ebola = pd.read_csv('ebola.csv')
ebola.head()
# Melt ebola
ebola_melt = pd.melt(frame = ebola, id_vars = ['Date', 'Day'], var_name = 'type_country', value_name = 'counts')
ebola_melt.head()
# create the 'str_split' column
ebola_melt['str_split'] = ebola_melt.type_country.str.split('_')
ebola_melt.head(2)
# create the 'type' column
ebola_melt['type'] = ebola_melt.str_split.str.get(0)
ebola_melt.head(2)
# create the 'country' column
ebola_melt['country'] = ebola_melt.str_split.str.get(1)
ebola_melt.head(2)<jupyter_output><empty_output>