# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: 'Python 3.8.10 64-bit (''myenv'': conda)'
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/rafaelhora/student-scores/blob/main/student_score.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] id="0bK0zUPX97Y7"
# ## Importing libraries and DataFrame

# + id="-pyXK1YB97Y-"
#importing libraries

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import os
from scipy.stats.stats import pearsonr

# + id="T-HSTcxM97ZA"
pwd = os.getcwd()

# + id="SqhmKi1O97ZB" outputId="9f97708b-6e0b-427c-8d39-b3a6d83d3c0c"
#importing data
#kaggle datasets download -d kwadwoofosu/predict-test-scores-of-students api command for dataset 
#source: https://www.kaggle.com/kwadwoofosu/predict-test-scores-of-students
dataframe = pd.read_csv(pwd + '\\test_scores.csv')
df = dataframe.copy()
dataframe.head()

# + id="yGe7Jldi97ZD" outputId="d9741ac8-76e4-4860-934b-97090233c473"
dataframe.info()

# + [markdown] id="rro6vame97ZD"
# No null values, so no necessity for value inputing or droping rows

# + [markdown] id="6ed--I3O97ZF"
# ## Exploratory Vizualization 

# + [markdown] id="oR6n6X0L97ZG"
# ### <b> Let's first understand the general characteristcs of our student population

# + id="TCQsqqMl97ZH" outputId="cca8d5ca-f3c7-4c27-a7d9-911a76d15e9c"
#plotting general data from population 
fig, ax = plt.subplots(2,2,figsize = [18,9])

cmap = {'coral', 'tab:blue', 'green'}

sns.set_style('whitegrid')

dataframe.groupby('gender').size().plot(kind = 'pie', autopct= '%1.2f%%', shadow = True, startangle = 45 , title = 'Gender Distribuition',ax = ax[0,0], colors = cmap)
dataframe.groupby('school_setting').size().plot(kind = 'pie', autopct= '%1.2f%%', shadow = True, startangle = 45 ,title = 'School Setting', ax = ax[0,1], colors = cmap)
dataframe.groupby('school_type').size().plot(kind = 'pie', autopct= '%1.2f%%', shadow = True, startangle = 45, title = 'School Type', ax = ax[1,0], colors = cmap)
dataframe.groupby('teaching_method').size().plot(kind = 'pie', autopct= '%1.2f%%', shadow = True, startangle = 45, title = 'Teaching Method' , ax = ax[1,1], colors = cmap)


plt.tight_layout()
plt.show()

# + [markdown] id="CryAL4BT97ZI"
# Some highlights gained by visualizing the general caractheristics of the general population:
#
# - Number of studends for each gender are split equally. 
#     - Is gender in any way a factor for influencing the test scores? 
# <br>
# <br>
# - The great marjority of students live in Urban/suburban settings. 
#     - Does any of these settings influence their performance in the test? 
# <br>
# <br>
# - As expected, most of the students attend public schools. Only 1/4 of the students study in non-public schools. 
#     - Does the difference between public and non-public schools scores show a significant gap in education quality? 
# <br>
# <br>
# - 1/3 of the students are in school that practice experimental teaching methods. 
#     - Do these novel methods improve students scores?

# + id="l0rnVAhn97ZJ"
#calculating the mean score for each demographic
gender_score = dataframe.groupby('gender').mean()['posttest'].reset_index()
type_score = dataframe.groupby('school_type').mean()['posttest'].reset_index()
lunch_score = dataframe.groupby('lunch').mean()['posttest'].reset_index()
method_score = dataframe.groupby('teaching_method').mean()['posttest'].reset_index()
setting_score = dataframe.groupby('school_setting').mean()['posttest'].reset_index()


# + id="O7JXM9km97ZJ" outputId="33e3b8b6-4a06-467d-9804-a6bf8ceaa5ff"
#plotting averages 
fig, ax = plt.subplots(2,2, figsize = [14, 9])

sns.set_palette('Set1')
sns.barplot(data=gender_score, y = 'posttest', x = 'gender', ax = ax[0,0])
sns.barplot(data=type_score, y = 'posttest', x = 'school_type', ax = ax[0,1])
sns.barplot(data=lunch_score, y = 'posttest', x = 'lunch', ax = ax[1,0])
sns.barplot(data=method_score, y = 'posttest', x = 'teaching_method', ax = ax[1,1])


sns.set_style('whitegrid')
plt.show()

# + [markdown] id="dnmUg1ZU97ZK"
# With the comparison of mean score for some properties of the population we can see that:
#
# - Gender appears not to be a relevant factor for test score.
#
# <n>
#
# - Students from public schools have a mean performance gap compared to private school students of more than 15%.
#
# <n>
#
# - Students in poor economic conditions (qualifies for lunch program) have a even more score gap compared with those who don't qualify for reduced/free lunch.
#
# <n>
#
# - Students from schools that use experimental teaching programs have a greater mean score compared to those from Standard schools. 

# + [markdown] id="QV9lDuhd97ZK"
# ### <b>Now, we direct our focus onto how school settings affect students performance <b>

# + id="JZ0lgFmg97ZL" outputId="05f2aa41-fbb7-4795-99cb-4ab90c59fe21"
#checking variance of scores for particular and public schools
fig, ax = plt.subplots(1,2,figsize = [20,10])
sns.swarmplot(data = dataframe ,x = 'school_type', y = 'posttest', ax = ax[0], hue = 'school_setting')
sns.boxplot(data = dataframe ,x = 'school_type', y = 'posttest', ax = ax[0], color = 'grey')

sns.swarmplot(data = dataframe ,x = 'teaching_method', y = 'posttest', ax = ax[1], hue = 'school_type')
sns.boxplot(data = dataframe ,x = 'teaching_method', y = 'posttest', ax = ax[1], color = 'grey')

fig.suptitle("Distribution of scores for School Types and teaching methods", fontsize = 18)
plt.tight_layout()
plt.show()

# + [markdown] id="6myM_FtD97ZL"
# In relation to school type, we can see that public schools have greater score variance compared to non-public ones. Also, depending on where is located a school can have significantly different results. Public suburban schools seem to excel in performance even compared to the non-public school general score. Since suburban schools have good performance, you could expect urban instituition to at least follow closely behind, however this is not the case. 
# <br>
# <br>
#     The factors on why Urban public schools have been notably underperforming. Questions to be raised: 
#     <br>
#     <br>
#     - What is the HDI level of these urban and suburban neighborhoods? <br>
#     - Urban schools are receiveing proper funding? <br>
#     - Is this difference caused by teaching methods? <br>
# <br>
# <br>
#
# In comparison between teaching methodology, it's possible to see the advantage in scores that was already observed. One thing to note is that in the group of schools with novel methods, the top 50% of performants are well distribuited between public and non-public schools. 
# <br>
# <br>
#
# It is imperative to answer the questions below:
#
# <br>
# <br>
#  - What are the experimental methods that these public schools are implementing? <br>
#  - Can they be reproduced in low performance standard schools in a attempt to improve their learning?<br>
#
# <b>Also note</b> 
# <br>
# <br>
#     - Some experimental schools are having scores below the 2nd quartile of Standard schools. Most of them are public. <br>
#     - Comparing to past performance, there was any improvement? <br>
#     - Consider reviewing those methods. <br>
#
# ---

# + [markdown] id="nxcukWpi97ZM"
# ### <b> Relation beteween pre-test and test scores

# + id="3O3PoMUp97ZM" outputId="7ef70bdd-6b46-49a5-b21d-2a94cba56d51"
#correlation between pre-test and post-test scores
x = np.linspace(0, 120, 100)
y = np.linspace(0, 120, 100)

fig, ax = plt.subplots(figsize = [8,6])

sns.scatterplot(x = dataframe['pretest'], y = dataframe['posttest'], hue =dataframe['school_type'])
plt.plot(x, y, color = 'green')

plt.annotate(xy = (60,60), text = 'pretest = posttest line', xytext = (50,0), xycoords = 'data', textcoords = 'offset points', arrowprops = dict(facecolor='green', shrink=0.05),
            horizontalalignment='left', verticalalignment='top', fontsize = 12)

plt.xlim(15, 105)
plt.ylim(15, 105)
plt.show()

# + [markdown] id="XgVVyeQl97ZN"
# <b>It's clear that, deriving from pre-test performance, we can expect at least similar or more likely higher scores at the real test. 

# + [markdown] id="v32uvGkN97ZN"
# ### <b> Visualizing performance per school

# + [markdown] id="tfV6YqUx97ZN"
# What are the characteristics of the best and worse schools?

# + id="ZkLda-Jc97ZO" outputId="744e36eb-2370-4984-8d21-4fdef9799821"
#Visualizing School performance
school_ranking = dataframe.groupby('school').agg(posttest = ('posttest', 'mean'), School_Type = ('school_type', pd.Series.mode), teaching = ('teaching_method', pd.Series.mode), school_setting = ('school_setting', pd.Series.mode), 
            n_student = ('n_student', pd.Series.mean) ).reset_index()
school_ranking.sort_values('posttest', ascending= False, ignore_index=True, inplace=True)
school_ranking.rename(columns={'school': 'School Name', 'posttest': 'Test Avg Score', 'School_Type':'School Type', 'teaching': 'Teaching Method', 'school_setting':'School Setting', 'n_student':'Avg students per class' }, inplace=True)
school_ranking

# + id="_Wn5ZVQ-97ZO" outputId="c2c180cc-6066-40ac-c45c-f425845ff549"
sns.scatterplot(data=school_ranking, x = 'Avg students per class', y = 'Test Avg Score', hue = 'School Type')

plt.show()

# + [markdown] id="5qK89jCg97ZP"
# We can't conclude yet the relation between number of students per class and score, but it already resembles a negative correlation. 

# + id="nO7YaZgK97ZP" outputId="6f1c8125-8c35-4910-b677-b18c209ca3a0"
corr = pearsonr(dataframe['posttest'], dataframe['n_student'])
f'Pearson correlation: {corr}' 

# + [markdown] id="HX9Mn4oD97ZQ"
# <b>Using Pearson Coefficient we can conclude that the number off students per class and test score has a very high inverse correlation, with a p-value close to zero indicating a significant statistical relevance for the result. 

# + [markdown] id="iBwL-TfJ97ZQ"
# ## Predicting Student Score

# + [markdown] id="gELJcX8T97ZR"
# ### <b> Dataset preparation

# + id="-NX1L5Kj97ZR"
df = dataframe.copy()

# + id="SGW2F0uu97ZR" outputId="4c44272e-375f-4c81-8cff-f364e847b594"
#dropping non-significant columns for this regression
df.drop(columns='student_id', inplace=True)

#encoding categorical variables
from sklearn.compose import ColumnTransformer
import category_encoders as ce
from sklearn.preprocessing import OrdinalEncoder

ct_be = ce.BinaryEncoder(cols =['school', 'school_setting', 'classroom'], return_df=True)
df_encoded = ct_be.fit_transform(df)#Using binary encoding for variables with high number of categories 
ct_oe = ce.OrdinalEncoder(cols = ['school_type', 'teaching_method', 'gender', 'lunch'], return_df=True, 

mapping = [{'col': 'school_type', 'mapping': {'Non-public': 0, 'Public':1}}, 
{'col':'teaching_method', 'mapping':{'Experimental':1, 'Standard':0}}, 
{'col':'gender', 'mapping': {'Female':1, 'Male':0}}, 
{'col':'lunch', 'mapping': {'Does not qualify':0, 'Qualifies for reduced/free lunch':1}}]) 

df_encoded = ct_oe.fit_transform(df_encoded)#ordinal encoding for 0 or 1 variables

df_encoded[['school_type', 'teaching_method', 'gender', 'lunch']] = df_encoded[['school_type', 'teaching_method', 'gender', 'lunch']].apply(pd.to_numeric)

# + [markdown] id="FLYEHB6d97ZS"
# ### <b> Visualizing dataset_encoded

# + id="jasbEWzF97ZS" outputId="d4f3876b-d6b2-4c22-9c58-4f7b1d4264d0"
df_encoded.info()

# + id="jWTKusx197ZS" outputId="c45dd876-a891-44e8-96cf-500976b95f91"
df_encoded

# + [markdown] id="n6kRyKmn97ZT"
# ### <b> Splitting Trainning and test datasets

# + id="BD_VzQRf97ZT"
from sklearn.model_selection import train_test_split
import xgboost as xg

#Split features and labels
X, y = df_encoded.iloc[:, :-1], df_encoded.iloc[:, -1]
  
# Splitting
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 42)
#transforming objects into DMatrix data
train_dmatrix = xg.DMatrix(data = X_train, label = y_train)
test_dmatrix = xg.DMatrix(data = X_test, label = y_test)


# + id="Nsvt2Srv97ZT"
train_dmatrix = xg.DMatrix(data = X_train, label = y_train)
test_dmatrix = xg.DMatrix(data = X_test, label = y_test)

# + [markdown] id="FPAvYHBI97ZT"
# ### <b> Trainning XGBRegressor model

# + id="wYCa5m5997ZT"
#trainning xboost regressor
params = {'booster':'gblinear', 'objective': 'reg:squarederror'}
reg = xg.train(dtrain = train_dmatrix, params = params, num_boost_round = 500)
y_pred = reg.predict(test_dmatrix)

# + [markdown] id="pDpPdTDZ97ZU"
# ## Results and performance Evaluation

# + id="wNBl8N3T97ZU" outputId="733ebfa3-29df-4a7f-f1ce-261c38ef9cb5"
from sklearn.metrics import mean_squared_error as MSE
rmse = np.sqrt(MSE(y_test, y_pred))
print("RMSE :% f " %(rmse))

from sklearn.metrics import r2_score
r2 = r2_score(y_test,y_pred)
x = 1 - r2
y = (X.shape[0] - 1) / (X.shape[0] - X.shape[1] - 1)
adjusted_r2 = 1 - (x * y)
print('R2 Score:% f' %(r2), '| Adjusted-R2:% f' %(adjusted_r2))

# + [markdown] id="4dFFySID97ZU"
# This model presents a good performance without utilizing more complex techiques such as feature reduction, and hyperparameter tunning. With R2 and Adjusted-R2 scores being very similar, it has a good bias-variance balance, and not prone to overfitting. 
# <br>
# <br>
# Using the model of the sections above we can predict the score of a certain student, or a whole school using just a few variables. 

StarcoderdataJupyterScriptsDedupFiltered