<jupyter_start><jupyter_text>Exploring the World Happiness Report Data Cleaning, Data Aggregation and Data VisualizationWorld Happiness Report is an annual report created by the UN Sustainable Development Solutions Network with the intent of guiding policy. The report assigns each country a happiness score based on the answers to a poll question that asks respondents to rank their life on a scale of 0 - 10.It also includes estimates of factors that may contribute to each country's happiness, including economic production, social support, life expectancy, freedom, absence of corruption, and generosity, to provide context for the score. Although these factors aren't actually used in the calculation of the happiness score, they can help illustrate why a country received a certain score. QuestionsThe following are the list of questions we need to find out in the dataset.* How can aggregating the data give us more insight into happiness scores?* How did world happiness change from 2015 to 2017?* Which factors contribute the most to the happiness score? Data Aggregation DatasetYou can download the dataset [here](https://www.kaggle.com/unsdsn/world-happiness)The data set is a CSV file called [`World_Happiness_2015.csv`](https://www.kaggle.com/unsdsn/world-happiness). Below are descriptions for columns:|**Column**|**Description**||---|---||**`Country`**|Name of the country.||**`Region`**|Name of the region the country belongs to.||**`Happiness Rank`**|The rank of the country, as determined by its happiness score.||**`Happiness Score`**|A score assigned to each country based on the answers to a poll question that asks respondents to rate their happiness on a scale of 0-10.||**`Family`**|The estimated extent to which family contributes to the happiness score.||**`Freedom`**|The estimated extent to which freedom contributes to the happiness score.||**`Generosity`**|The estimated extent to which generosity contributes to the happiness score.| Dataset Inspection<jupyter_code>%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

plt.style.use('ggplot')
happiness2015 = pd.read_csv('https://dsserver-prod-resources-1.s3.amazonaws.com/343/World_Happiness_2015.csv?versionId=jQYOlRzzMPFpIhNjLxwFNWDSOAsdpjnN')
happiness2015.head()
happiness2015.tail()
happiness2015.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 158 entries, 0 to 157
Data columns (total 12 columns):
 #   Column                         Non-Null Count  Dtype  
---  ------                         --------------  -----  
 0   Country                        158 non-null    object 
 1   Region                         158 non-null    object 
 2   Happiness Rank                 158 non-null    int64  
 3   Happiness Score                158 non-null    float64
 4   Standard Error                 158 non-null    float64
 5   Economy (GDP per Capita)       158 non-null    float64
 6   Family                         158 non-null    float64
 7   Health (Life Expectancy)       158 non-null    float64
 8   Freedom                        158 non-null    float64
 9   Trust (Government Corruption)  158 non-null    float64
 10  Generosity                     158 non-null    float64
 11  Dystopia Residual              158 non-null    float64
dtypes: float64(9), int64(1), object(2)
memory usage: 1[...]<jupyter_text>Mean Happiness Score by region<jupyter_code>happiness2015.groupby('Region')['Happiness Score'].mean().sort_values()
happiness2015.pivot_table(values='Happiness Score', index='Region', margins=True).plot(kind='barh', title='Mean Happiness Scores by Region', xlim=(0,10), legend=False)<jupyter_output><empty_output><jupyter_text>Did world happiness increase, decrease, or stay about the same from 2015 to 2017?<jupyter_code>happiness2016 = pd.read_csv('https://dsserver-prod-resources-1.s3.amazonaws.com/344/World_Happiness_2016.csv?versionId=g2kGflhRQVfDoiUtXaBqfKoLx_BrgtJs')
happiness2016.head()
happiness2017 = pd.read_csv('https://dsserver-prod-resources-1.s3.amazonaws.com/344/World_Happiness_2017.csv?versionId=VojSXR_mvHTzv41jGPwSvMFGZiHuKXCq')
happiness2017.head()
for i,df in enumerate([happiness2015,happiness2016,happiness2017]):
    df['Year']=2015+i
happiness2017.rename(columns={'Happiness.Score': 'Happiness Score'}, inplace=True)
combined = pd.concat([happiness2015, happiness2016, happiness2017])
combined.pivot_table(index='Year', values='Happiness Score').plot(kind='barh', title='Mean Happiness Scores by Year', xlim=(0,10))
plt.show()<jupyter_output><empty_output><jupyter_text>mean world happiness score stayed approximately the same from 2015 to 2017. Exploring the factors that contribute happiness.> the World Happiness Report assigns each country a happiness score based on a poll question that asks respondents to rank their life on a scale of 0 - 10.The factors incluce:* Economy (GDP per Capita)* Family* Health (Life Expectancy)* Freedom* Trust (Government Corruption)* Generosity Which of the factors above contribute the most to the happiness score?<jupyter_code>happiness2015=happiness2015.rename({'Economy (GDP per Capita)': 'Economy', 'Health (Life Expectancy)': 'Health', 'Trust (Government Corruption)': 'Trust' }, axis=1)
happiness2015.columns
factors_impact=happiness2015[['Economy', 'Family', 'Health', 'Freedom', 'Trust', 'Generosity']].applymap(lambda x: 'High' if x > 1 else 'low')
factors_impact<jupyter_output><empty_output><jupyter_text>> We can see from the results that, according to our definition, the `Economy` and `Family` columns had a high impact on the happiness scores of the first five countries.<jupyter_code>factors_impact.apply(pd.value_counts)<jupyter_output><empty_output><jupyter_text>> Now, we can easily see that the `Family` and `Economy` columns contain the most `'High'` values!<jupyter_code>main_cols = ['Country', 'Region', 'Happiness Rank', 'Happiness Score']
factors = ['Economy', 'Family', 'Health', 'Freedom', 'Trust', 'Generosity', 'Dystopia Residual']
melt=pd.melt(happiness2015, id_vars=main_cols,value_vars=factors)
melt['Percentage']=round((melt.value/melt['Happiness Score'])*100,2)
melt.head()
pv_melt=melt.pivot_table(index='variable', values='value')
pv_melt.plot(kind='pie', y='value', legend=False)
plt.show()<jupyter_output><empty_output><jupyter_text>World Development Indicatorsworking with the additional economic data from the World Bank. Column descriptionBelow are descriptions for the columns:|**Column Name**|**Column Description**||---|---||`ShortName`|Name of the country||`Region`|The region the country belongs to||`IncomeGroup`|The income group the country belongs to, based on Gross National Income (GNI) per capita||`CurrencyUnit`|Name of country's currency||`SourceOfMostRecentIncomeAndExpenditureData`|The name of the survey used to collect the income and expenditure data||`SpecialNotes`|Contains any miscellaneous notes about the data|<jupyter_code>world_dev = pd.read_csv("https://dsserver-prod-resources-1.s3.amazonaws.com/346/World_dev.csv?versionId=1G.IH6N10dXmFAG1mSTdtdLax4d9DPtC")
col_renaming = {'SourceOfMostRecentIncomeAndExpenditureData': 'IESurvey'}
world_dev.head()
merged=pd.merge(left=happiness2015, right=world_dev,how='left', left_on='Country', right_on='ShortName')
merged=merged.rename(col_renaming, axis=1)
merged.head()
merged['Currency'] = merged.CurrencyUnit.str.split().str.get(-1)
merged.Currency.head()<jupyter_output><empty_output><jupyter_text>National Accounts<jupyter_code>merged_national_accounts = merged[merged.SpecialNotes.str.contains(r"[Nn]ational accounts", na=False)]
merged_national_accounts.head()
merged["IncomeGroup"]=merged.IncomeGroup.str.upper().str.strip().str.replace(" INCOME","").str.strip().str.replace(":","")
pv_incomes=merged.pivot_table(index="IncomeGroup", values="Happiness Score")
pv_incomes.plot(kind='bar', rot=30,ylim=(0,10))
happiness2015.columns
happiness2016.columns
happiness2017.columns
happiness2017.columns = happiness2017.columns.str.replace('.', ' ').str.replace('\s+', ' ').str.strip().str.upper()
happiness2015.columns = happiness2015.columns.str.replace('(', '').str.replace(')', '').str.strip().str.upper()
happiness2016.columns = happiness2016.columns.str.replace('(', '').str.replace(')', '').str.strip().str.upper()

combined = pd.concat([happiness2015, happiness2016, happiness2017], ignore_index=True)
combined.isnull().sum()
happiness2015 = pd.read_csv("https://dsserver-prod-resources-1.s3.amazonaws.com/347/wh_2015.csv?versionId=jfEXotZkNMvPapsuIuIiUy9PD8d.yQpp")
happiness2016 = pd.read_csv("https://dsserver-prod-resources-1.s3.amazonaws.com/347/wh_2016.csv?versionId=YTFx.axVvCayn262yITV7dTcN5VWrd_8")
happiness2017 = pd.read_csv("https://dsserver-prod-resources-1.s3.amazonaws.com/347/wh_2017.csv?versionId=rpTqH_8SLrRgSWwCJvfx472qYfpLK5Qm")
happiness2017.columns = happiness2017.columns.str.replace('.', ' ').str.replace('\s+', ' ').str.strip().str.upper()
happiness2015.columns = happiness2015.columns.str.replace('(', '').str.replace(')', '').str.strip().str.upper()
happiness2016.columns = happiness2016.columns.str.replace('(', '').str.replace(')', '').str.strip().str.upper()

combined = pd.concat([happiness2015, happiness2016, happiness2017], ignore_index=True)
combined.isnull().sum()
combined_updated = combined.set_index('YEAR')
sns.heatmap(combined_updated.isnull(), cbar=False)<jupyter_output><empty_output><jupyter_text>**We can make the following observations:*** No values are missing in the COUNTRY column.* There are some rows in the 2015, 2016, and 2017 data with missing values in all columns EXCEPT the COUNTRY column.* Some columns only have data populated for one year.* It looks like the REGION data is missing for the year 2017.Since the regions are fixed values - the region a country was assigned to in 2015 or 2016 won't change - we should be able to assign the 2015 or 2016 region to the 2017 row.<jupyter_code>combined = pd.merge(left=combined, right=pd.read_csv("https://raw.githubusercontent.com/VictorOmondi1997/exploring-whr/master/data/regions.csv"), on="COUNTRY", how="left")
combined = combined.drop("REGION_x", axis=1)
combined.isna().sum()<jupyter_output><empty_output><jupyter_text>check for duplicate valuesSince we should only have one country for each year, we can be a little more thorough by defining rows with ONLY the same country and year as duplicates.<jupyter_code>combined[combined.duplicated(["COUNTRY", 'YEAR'])]<jupyter_output><empty_output><jupyter_text>Since the dataframe is empty, we can tell that there are no rows with exactly the same country AND year.<jupyter_code>combined["COUNTRY"]=combined["COUNTRY"].str.upper()
combined[combined.duplicated(['COUNTRY', 'YEAR'])]
combined = combined.drop_duplicates(["COUNTRY", "YEAR"])<jupyter_output><empty_output><jupyter_text>Missing valuesSince missing values make up more than half of the following columns and we don't need them to accomplish our end goal, we'll drop them:* STANDARD ERROR* LOWER CONFIDENCE INTERVAL* UPPER CONFIDENCE INTERVAL* WHISKER HIGH* WHISKER LOW<jupyter_code>combined=combined.drop(['LOWER CONFIDENCE INTERVAL', 'STANDARD ERROR', 'UPPER CONFIDENCE INTERVAL', 'WHISKER HIGH', 'WHISKER LOW'], axis=1)
combined.isna().sum()
sorted_happiness = combined.set_index('REGION_y').sort_values(['REGION_y', 'HAPPINESS SCORE'])
sns.heatmap(sorted_happiness.isnull(), cbar=False)<jupyter_output><empty_output><jupyter_text>From the visualization above, we can also identify that only three regions contain missing values:* Sub-Saharan Africa* Middle East and Northern Africa* Latin America and CarribbeanThe Sub-Saharan Africa region contains the most missing values, accounting for about 9 percent of that regions's values. >* Only about 4 percent of the values in each column are missing.* Dropping rows with missing values won't cause us to lose information in other columns.<jupyter_code>combined.pivot_table(index='REGION_y', values='HAPPINESS SCORE', margins=True)<jupyter_output><empty_output><jupyter_text>We can see that the world mean happiness score, 5.370728, is over 1 point higher than the mean happiness score for the Sub-Saharan Africa region, 4.150957.Also, if we think about the reasons why a country may not have participated in the happiness survey - war, natural disaster, etc - many of them would likely result in a lower happiness score than even the region's mean. We'll conclude that the mean for the whole world wouldn't be a good estimate for them.As a result, we'll decide that of these two options, it's better to drop the rows with missing values.<jupyter_code>combined = combined.dropna()
combined.isna().sum()<jupyter_output><empty_output>