<jupyter_start><jupyter_text>MAC0459/MAC05865 - Data Science and Engineering Name: <NAME>USP: 9790930  Decision trees exercises--- **Q1. What is the approximate depth of a Decision Tree trained (without restrictions) on a training set with 10 million instances?** Since most decision tree algorithms consider binary decision trees, we could say the approximate depth is $\lceil \log_{2} 10000000\rceil = 24$ **Q2. Is a node’s Gini impurity generally lower or greater than its parent? Is it generally lower (greater), or always lower (greater)?** Let us show that the Gini impurity is generally lower than its parent. The Gini impurity is defined as:$$ i(N) = \sum\limits_{i \neq j} P(W_{i}|N)P(w_{j}|N) = \frac{1}{2} \left[1 - \sum\limits_{j}P^2(W_{j}|N)\right] $$Define:$P(W|N) = \frac{w}{n}$ where $w = \W$ and $n = \N$Define $N_{p}$ and $N_{c}$ as the parent node and the child node respectively, $d_{i} = w_{i} - u_{i}$, $d = \sum\limits_{i} d_{i}$ and $d > 0, n > 1$.$$i(N_{p}) = \frac{1}{2} \left[1 - \sum\limits_{j}\left(\frac{w_{j}}{n}\right)^2\right] = \frac{1}{2} \left[1 - \frac{\sum\limits_{j} w_{j}^2}{n^2}\right]$$$$i(N_{c}) = \frac{1}{2} \left[1 - \sum\limits_{j}\left(\frac{u_{j}}{n - d}\right)^2\right] = \frac{1}{2} \left[1 - \sum\limits_{j}\left(\frac{w_{j} - d_{j}}{n - d}\right)^2\right] =  \frac{1}{2} \left[1 - \frac{ \sum\limits_{j}w_{j}^2 - 2\sum\limits_{j}w_{j}d_{j} + \sum\limits_{j}d_{j}^2}{n^2 - 2nd + d^2}\right] = \frac{1}{2} \left[1 - \frac{ \sum\limits_{j}w_{j}^2 - 2nd + \sum\limits_{j}d_{j}^2}{n^2 - 2nd + d^2}\right]$$Now, we need to show that $i(N_{p}) - i(N_{c}) > 0$:$$i(N_{p}) - i(N_{c}) = \frac{1}{2} \left[1 - \frac{\sum\limits_{j} w_{j}^2}{n^2}\right] - \frac{1}{2} \left[1 - \frac{ \sum\limits_{j}w_{j}^2 - 2nd + \sum\limits_{j}d_{j}^2}{n^2 - 2nd + d^2}\right] = \frac{1}{2} \left[1 - \frac{\sum\limits_{j} w_{j}^2}{n^2} - \left(1 - \frac{ \sum\limits_{j}w_{j}^2 - 2nd + \sum\limits_{j}d_{j}^2}{n^2 - 2nd + d^2}\right) \right] = \frac{1}{2} \left[-\frac{\sum\limits_{j} w_{j}^2}{n^2} + \frac{ \sum\limits_{j}w_{j}^2 - 2nd + \sum\limits_{j}d_{j}^2}{n^2 - 2nd + d^2}\right]$$Thus, we must show that $\frac{ \sum\limits_{j}w_{j}^2 - 2nd + \sum\limits_{j}d_{j}^2}{n^2 - 2nd + d^2} - \frac{\sum\limits_{j} w_{j}^2}{n^2} > 0$$$\frac{ \sum\limits_{j}w_{j}^2 - 2nd + \sum\limits_{j}d_{j}^2}{n^2 - 2nd + d^2} - \frac{\sum\limits_{j} w_{j}^2}{n^2} = \left(\frac{1}{(n^2 - 2nd + d^2)n^2}\right)\left(n^2\sum\limits_{j}w_{j}^2 - 2n^3d + n^2\sum\limits_{j}d_{j}^2 - n^2\sum\limits_{j}w_{j}^2 + 2nd\sum\limits_{j}w_{j}^2 - d^2\sum\limits_{j}w_{j}^2\right) =\left(\frac{1}{(n^2 - 2nd + d^2)n^2}\right)\left(n^2\sum\limits_{j}d_{j}^2 + 2nd\sum\limits_{j}w_{j}^2- 2n^3d -  d^2\sum\limits_{j}w_{j}^2\right) = \left(\frac{1}{(n^2 - 2nd + d^2)n^2}\right)\left(n^2\sum\limits_{j}d_{j}^2 + 2nd(\sum\limits_{j}w_{j}^2 - n^2) - d^2\sum\limits_{j}w_{j}^2\right)$$ **Q3. If a Decision Tree is overfitting the training set, is it a good idea to try decreasing maxdepth? Why?** Yes, when we decrease the maxdepth we are also reducing the information gain that the last depth would have gained. Hence this new decision tree is less specialized in the training set. **Q4. If a Decision Tree is underfitting the training set, is it a good idea to try scaling the input features? Why?** No, scaling the input features is good when our classifier relies on Eucliadean distance. The decision trees relies on information gain to make the splits and the information gain measurents make use of probabilities, not Eucliadean distance. **Q5. If it takes one hour to train a Decision Tree on a training set containing 10 million instances, roughly how much time will it take to train another Decision Tree on a similar training set containing 100 million instances?** The [Scikit-learn documentation of the decision trees](https://scikit-learn.org/stable/modules/tree.htmlcomplexity) states that the complexity of the training algorithm is $O(n_{features}n_{samples}^{2}\log(n_{samples}))$.The $n_{features}$ is the same for both trainings. By a simple cross- multiplication we can make the following estimation:$$\frac{n_{features}(10^7)^2\log(n_{10^7})}{n_{features}(10^8)^2\log(n_{10^7})} = \frac{60}{x} \implies \frac{10^{14} * 24}{10^{16} * 27} \approx \frac{60}{x} \implies x \approx \frac{10^{16} * 27 * 60}{10^{14} * 24} \implies x \approx \frac{100 * 27 * 60}{24}$$$$\implies x \approx \frac{100 * 27 * 60}{24} \implies x \approx \frac{162000}{24} \implies x \approx 6750$$Hence, it will take approximately 6750 minutes or 112 hours and 30 minutes. **Q6.** **Train and  fine-tune a Decision Tree for the Cardiovascular Disease dataset. The dataset has very few information on acquisition protocol except that all of the dataset values were collected at the moment of medical examination. There are three types of input features:****–Objective:  factual information;****–Examination:  results of medical examination;****–Subjective:  information given by the patient.****Besides that, features are classified according to the following values:****–1:  normal,****–2:  above normal,****–3:  well above normal****Questions or tasks to be addressed:** **1) Present an EDA analysis of the dataset.** Let us import necessary data science packages, load, check and prepare the dataset if necessary.<jupyter_code>import pandas as pd
from statistics import mean, median
import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt

sns.set()

# Loading CSV's
print("Loading Dataset")
cardio = pd.read_csv('http://linux.ime.usp.br/~guilhevieira/cardio_train.csv', error_bad_lines = False)
print("Dataset loaded successfully")
cardio.info()
cardio.head()
cardio.tail()<jupyter_output><empty_output><jupyter_text>Okay, everything is alright. How many records do we have and how many features for each record?<jupyter_code>cardio.shape<jupyter_output><empty_output><jupyter_text>Let us generate some statistics that summarize each numerical feature, like mean, variance, min, median and and max values.* Age (days)<jupyter_code>pd.DataFrame(cardio['age'].describe())<jupyter_output><empty_output><jupyter_text>The age values are in days, which is hard for human readability because we are accustomed with years, let us see the statistics in years.<jupyter_code>age_in_years = pd.DataFrame(cardio['age'].copy())
age_in_years['age'] = age_in_years['age'].apply(lambda x: x/365)
age_in_years.describe()<jupyter_output><empty_output><jupyter_text>* Height (cm)<jupyter_code>pd.DataFrame(cardio['height'].describe())<jupyter_output><empty_output><jupyter_text>* Weigth (kg)<jupyter_code>pd.DataFrame(cardio['weight'].describe())<jupyter_output><empty_output><jupyter_text>* Systolic blood pressure<jupyter_code>pd.DataFrame(cardio['ap_hi'].describe())<jupyter_output><empty_output><jupyter_text>* Diastolic blood pressure<jupyter_code>pd.DataFrame(cardio['ap_lo'].describe())<jupyter_output><empty_output><jupyter_text>Alright, there are some interesting things over there. **CONCLUSIONS**Let us move on and visualize how weel distributed the datasets are in relation to each attribute throw histograms for numerical features and bar plots for categorical values.<jupyter_code>sns.distplot(cardio['age'])
sns.distplot(cardio['height'])
sns.distplot(cardio['weight'])
cardio['gender'].value_counts().plot(kind="bar")
plt.ylabel('Quantity')
plt.xlabel('Gender')
plt.show()
sns.distplot(cardio['ap_hi'])
sns.distplot(cardio['ap_lo'])
cardio['cholesterol'].value_counts().plot(kind="bar")
plt.ylabel('Quantity')
plt.xlabel('Cholesterol')
plt.show()
cardio['gluc'].value_counts().plot(kind="bar")
plt.ylabel('Quantity')
plt.xlabel('Glucose')
plt.show()
cardio['smoke'].value_counts().plot(kind="bar")
plt.ylabel('Quantity')
plt.xlabel('Smoking')
plt.show()
cardio['alco'].value_counts().plot(kind="bar")
plt.ylabel('Quantity')
plt.xlabel('Alcohol intake')
plt.show()
cardio['active'].value_counts().plot(kind="bar")
plt.ylabel('Quantity')
plt.xlabel('Physical activity')
plt.show()
cardio['cardio'].value_counts().plot(kind="bar")
plt.ylabel('Quantity')
plt.xlabel('Presence or absence of cardiovascular disease')
plt.show()<jupyter_output><empty_output><jupyter_text>Write some **CONCLUSIONS** **2)  Split it into a training set and a test set using ```train_test_split``` method from sklearn.**<jupyter_code>from sklearn.model_selection import train_test_split

df_train, df_test = train_test_split(cardio, test_size=0.2, random_state=145)

df_train.shape
df_train.tail()
df_test.shape
df_train.tail()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup