# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Getting started with the practicals
#
# ***These notebooks are best viewed in Jupyter. GitHub might not display all content of the notebook properly.***
#
# ## Goal of the practical exercises
#
# The exercises have two goals:
#
# 1. Give you the opportunity to obtain 'hands-on' experience in implementing, training and evaluation machine learning models in Python. This experience will also help you better understand the theory covered during the lectures. 
#
# 2. Occasionally demonstrate some 'exam-style' questions that you can use as a reference when studying for the exam. Note however that the example questions are (as the name suggests) only examples and do not constitute a complete and sufficient list of 'things that you have to learn for the exam'. You can recognize example questions as (parts of) exercises by <font color="#770a0a">this font color</font>.
#
# For each set of exercises (one Python notebook such as this one $==$ one set of exercises) you have to submit deliverables that will then be graded and constitute 25% of the final grade. Thus, the work that you do during the practicals has double contribution towards the final grade: as 30% direct contribution and as a preparation for the exam that will define the other 65% of the grade.
#
# ## Deliverables
#
# For each set of exercises, you have to submit:
# 1. Python functions and/or classes (`.py` files) that implement basic functionalities (e.g. a $k$-NN classifier) and 
# 2. A *single* Python notebook that contains the experiments, visualization and answer to the questions and math problems. *Do not submit your answers as Word or PDF documents (they will not be graded)*. The submitted code and notebook should run without errors and be able to fully reproduce the reported results.
#
# We recommend that you clone the provided notebooks (such as this one) and write your code in them. The following rubric will be used when grading the practical work:
#
# Component  | Insufficient | Satisfactory | Excellent
# --- | --- | --- | ---
# **Code** | Missing or incomplete code structure, runs with errors, lacks documentation | Self-contained, does not result in errors, contains some documentation, can be easily used to reproduce the reported results | User-friendly, well-structured (good separation of general functionality and experiments, i.e. between `.py` files and the Pyhthon notebook), detailed documentation, optimized for speed, use of a version control system (such as GitHub)
# **Answers to questions** | Incorrect, does not convey understanding of the material, appears to be copied from another source | Correct, conveys good understanding of the material, description in own words | Correct, conveys excellent level of understanding, makes connections between topics
#
# ## A word on notation
#
# When we refer to Python variables, we will use a monospace font. For example, `X` is a Python variable that contains the data matrix. When we refer to mathematical variables, we will use the de-facto standard notation: $a$ or $\lambda$ is a scalar variable, $\boldsymbol{\mathrm{w}}$ is a vector and $\boldsymbol{\mathrm{X}}$ is a matrix (e.g. a data matrix from the example above). You should use the same notation when writing your answers and solutions.
#
# # Two simple machine learning models
#
# ## Preliminaries
#
# Throughout the practical curriculum of this course, we will use the Python programming language and its ecosystem of libraries for scientific computing (such as `numpy`, `scipy`, `matplotlib`, `scikit-learn` etc). The practicals for the deep learning part of the course will use the `keras` deep learning framework. If you are not sufficiently familiar with this programming language and/or the listed libraries and packages, you are strongly advised to go over the corresponding tutorials from the ['Essential skills'](https://github.com/tueimage/essential-skills) module (the `scikit-learn` library is not covered by the tutorial, however, an extensive documentation is available [here](https://scikit-learn.org/stable/documentation.html).
#
# In this first set of exercises, we will use two toy datasets that ship together with `scikit-learn`. 
#
# The first dataset is named `diabetes` and contains 442 patients described with 10 features: age, sex, body mass index, average blood pressure, and six blood serum measurements. The target variable is a continuous quantitative measure of the disease (diabetes) progression one year after the baseline measurements were recorded. More information is available [here](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/datasets/descr/diabetes.rst) and [here](https://www4.stat.ncsu.edu/~boos/var.select/diabetes.html).
#
# The second dataset is named `breast_cancer` and is a copy of the UCI ML Breast Cancer Wisconsin (Diagnostic) datasets (more infortmation is available [here](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/datasets/descr/breast_cancer.rst) and [here](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)). The datasets contains of 569 instances represented with 30 features that are computed from a images of a fine needle aspirate of a breast mass. The features describe characteristics of the cell nuclei present in the image. Each instance is associated with a binary target variable ('malignant' or 'benign'). 
#
# You can load the two datasets in the following way:

# +
import numpy as np
from sklearn.datasets import load_diabetes, load_breast_cancer

diabetes = load_diabetes()

breast_cancer = load_breast_cancer()
# -

# In the majority of the exercises in this course, we will use higher-level libraries and packages such as `scikit-learn` and `keras` to implement, train and evaluate machine learning models. However, the goal of this first set of exercises is to illustrate basic mathematical tools and machine learning concepts. Because of this, we will impose a restriction of only using basic `numpy` functionality. Furthermore, you should as much as possible restrict the use of for-loops (e.g. use a vector-to-matrix product instead of a for loop when appropriate).
#
# If `X` is a 2D data matrix, we will use the convention that the rows of the matrix contain the samples (or instances) and the columns contain the features (inputs to the model). That means that a data matrix with a shape `(122, 13)` represents a dataset with 122 samples, each represented with 13 features. Similarly, if `Y` is a 2D matrix containing the targets, the rows correspond to the samples and the columns to the different targets (outputs of the model). Thus, if the shape of `Y` is `(122, 3)` that means that there are 122 samples and each sample is has 3 targets (note that in the majority of the examples we will only have a single target and thus the number of columns of `Y` will be 1).
#
# You can obtain the data and target matrices from the two datasets in the following way:

# +
X = diabetes.data
Y = diabetes.target[:, np.newaxis]

print(X.shape)
print(Y.shape)
# -

# If you want to only use a subset of the available features, you can obtain a reduced data matrix in the following way:

# +
# use only the fourth feature
X = diabetes.data[:, np.newaxis, 3]
print(X.shape)

# use the third, and tenth features
X = diabetes.data[:, (3,9)]
print(X.shape)
# -

# ***Question***: Why we need to use the `np.newaxis` expression in the examples above? 
#
# _To make the data a 2-D matrix, which is necessary for performing matrix operations in Numpy._
#
# Note that in all your experiments in the exercises, you should use and independent training and testing sets. You can split the dataset into a training and testing subsets in the following way:

# use the fourth feature
# use the first 300 training samples for training, and the rest for testing
X_train = diabetes.data[:300, np.newaxis, 3]
y_train = diabetes.target[:300, np.newaxis]
X_test = diabetes.data[300:, np.newaxis, 3]
y_test = diabetes.target[300:, np.newaxis]
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)

# ## Exercises
#
# ### Linear regression
#
# Implement training and evaluation of a linear regression model on the diabetes dataset using only matrix multiplication, inversion and transpose operations. Report the mean squared error of the model.
#
# To get you started we have implemented the first part of this exercise (fitting of the model) as an example.

# +
# import data
import numpy as np
from sklearn.datasets import load_diabetes, load_breast_cancer
diabetes = load_diabetes()

# add subfolder that contains all the function implementations
# to the system path so we can import them
import importlib as il
import sys
import matplotlib.pyplot as plt
sys.path.append('code/')

# the actual implementation is in linear_regression.py,
# here we will just use it to fit a model
import linear_regression
from linear_regression import *
il.reload(linear_regression)

# load the dataset
# same as before, but now we use all features
X_train = diabetes.data[:300, :]
y_train = diabetes.target[:300, np.newaxis]
X_test = diabetes.data[300:, :]
y_test = diabetes.target[300:, np.newaxis]

beta = lsq(X_train, y_train)

# print the parameters
print("Beta = {}".format(beta))

# evaluate the linear regression model & calculate the (Root) Mean Squared Error
MSE, y_out = eval_lin_model(beta, X_test, y_test)
print("Mean Squared Error = {}".format(MSE))
print("Root Mean Squared Error = {}".format(np.sqrt(MSE)))

# -

# ### Weighted linear regression
#
# Assume that in the dataset that you use to train a linear regression model, there are identical versions of some samples. This problem can be reformulated to a weighted linear regression problem where the matrices $\boldsymbol{\mathrm{X}}$ and $\boldsymbol{\mathrm{Y}}$ (or the vector $\boldsymbol{\mathrm{y}}$ if there is only a single target/output variable) contain only the unique data samples, and a vector $\boldsymbol{\mathrm{d}}$ is introduced that gives more weight to samples that appear multiple times in the original dataset (for example, the sample that appears 3 times has a corresponding weight of 3). 
#
# <p><font color='#770a0a'>Derive the expression for the least-squares solution of a weighted linear regression model (note that in addition to the matrices $\boldsymbol{\mathrm{X}}$ and $\boldsymbol{\mathrm{Y}}$, the solution should include a vector of weights $\boldsymbol{\mathrm{d}}$).</font></p>

# $WSS(\boldsymbol{\mathrm{w}})=(\boldsymbol{\mathrm{Y}} - \boldsymbol{\mathrm{X}}\boldsymbol{\mathrm{w}})^T\boldsymbol{\mathrm{D}}(\boldsymbol{\mathrm{Y}} - \boldsymbol{\mathrm{X}}\boldsymbol{\mathrm{w}})$ should be minimised. <br>
#
# $ (\boldsymbol{\mathrm{Y}} - \boldsymbol{\mathrm{X}}\boldsymbol{\mathrm{w}})^T\boldsymbol{\mathrm{D}}(\boldsymbol{\mathrm{Y}} - \boldsymbol{\mathrm{X}}\boldsymbol{\mathrm{w}}) = (\boldsymbol{\mathrm{Y}}^T - \boldsymbol{\mathrm{w}}^T\boldsymbol{\mathrm{X}}^T)\boldsymbol{\mathrm{D}}(\boldsymbol{\mathrm{Y}} - \boldsymbol{\mathrm{X}}\boldsymbol{\mathrm{w}}) = \boldsymbol{\mathrm{Y}}^T\boldsymbol{\mathrm{D}}\boldsymbol{\mathrm{Y}} - \boldsymbol{\mathrm{Y}}^T\boldsymbol{\mathrm{D}}\boldsymbol{\mathrm{X}}\boldsymbol{\mathrm{w}} - \boldsymbol{\mathrm{w}}^T\boldsymbol{\mathrm{X}}^T\boldsymbol{\mathrm{D}}\boldsymbol{\mathrm{Y}} + \boldsymbol{\mathrm{w}}^T\boldsymbol{\mathrm{X}}^T\boldsymbol{\mathrm{D}}\boldsymbol{\mathrm{X}}\boldsymbol{\mathrm{w}} = \boldsymbol{\mathrm{Y}}^T\boldsymbol{\mathrm{D}}\boldsymbol{\mathrm{Y}} - 2\boldsymbol{\mathrm{w}}^T\boldsymbol{\mathrm{X}}^T\boldsymbol{\mathrm{D}}\boldsymbol{\mathrm{Y}} + \boldsymbol{\mathrm{w}}^T\boldsymbol{\mathrm{X}}^T\boldsymbol{\mathrm{D}}\boldsymbol{\mathrm{X}}\boldsymbol{\mathrm{w}}$
#
# Computing the derivative w.r.t. $\boldsymbol{\mathrm{w}}$ gives $ -2\boldsymbol{\mathrm{X}}^T\boldsymbol{\mathrm{D}}\boldsymbol{\mathrm{Y}} + 2\boldsymbol{\mathrm{X}}^T\boldsymbol{\mathrm{D}}\boldsymbol{\mathrm{X}}\boldsymbol{\mathrm{w}}$.<br> 
#
# Equating this expression to 0 gives us the least-squares weighted regression coefficient vector $\boldsymbol{\mathrm{\hat{w}}}$:
# $\boldsymbol{\mathrm{\hat{w}}} = (\boldsymbol{\mathrm{X}}^T\boldsymbol{\mathrm{D}}\boldsymbol{\mathrm{X}})^{-1}\boldsymbol{\mathrm{X}}^T\boldsymbol{\mathrm{D}}\boldsymbol{\mathrm{Y}}$<br>
# where $\boldsymbol{\mathrm{D}} = diag(\boldsymbol{\mathrm{d}})$ 

# ### $k$-NN classification
#
# Implement a $k$-Nearest neighbors classifier from scratch in Python using only basic matrix operations with `numpy` and `scipy`. Train and evaluate the classifier on the breast cancer dataset, using all features. Show the performance of the classifier for different values of $k$ (plot the results in a graph). Note that for optimal results, you should normalize the features (e.g. to the $[0, 1]$ range or to have a zero mean and unit standard deviation).

#

# +
import numpy as np 
from sklearn.datasets import load_breast_cancer
from NNfunction import *
import matplotlib.pyplot as plt 

"""
Load the breast cancer dataset find the true targets from the test data set and choose how many neighbour points should be looked at.
 
"""
breast_cancer = load_breast_cancer()
real_values=breast_cancer.target[300:,np.newaxis]
nn= 60
k=range(1,nn) 

"""
The for-loop classfies the test dataset using different values for k. After that, the true values and the calculated values are compared 
and the amount of correctly and falsely classefied samples are stored in the lists: Good and Wrong.
These lists are then plotted as a function of k.
"""
    
Good=[]
Wrong=[]
for i in range(1,nn):
    calssification=NNfunction(i,breast_cancer,30)
    Incorrect_predictions=0
    Correct_predictions=0
    for q in range(len(real_values)):
        if calssification[q,:]==real_values[q,:]:
            Correct_predictions=Correct_predictions+1
        else:
            Incorrect_predictions=Incorrect_predictions+1
    Good.append(Correct_predictions)
    Wrong.append(Incorrect_predictions)

f, axis = plt.subplots(2,1,)

axis[0].plot(k,Good,color='green')
axis[0].set_xlabel('Number of Neighbours')
axis[0].set_ylabel('Correct classifications') 
 
axis[1].plot(k,Wrong,color='red')
axis[1].set_xlabel('Number of Neighbours')
axis[1].set_ylabel('False classifications')
     

plt.show()




# -

# The script that is shown above makes use of a function called NNfunction which classifies a test dataset using the nn nearest neighbours of the samples with the training dataset. The breast cancer dataset was split up into a training and a test dataset and test dataset was classified using the training data. This script's classification results were compared to the correct classification of the samples. The graphs shows the amount of correctly and wrongfully classified samples. 
#
# One can see that the amount of correct classifications goes up in the beginning, but eventually the amount of correct classifications goes down , and the amount of false classifications goes up. 

# ### $k$-NN regression
#
# Modify the $k$-NN implementation to do regression instead of classification. Compare the performance of the linear regression model and the $k$-NN regression model on the diabetes dataset for different values of $k$..

# +
import numpy as np 
from sklearn.datasets import load_diabetes
from NNfunctionRegression import *
import matplotlib.pyplot as plt 

'''
Loading the diabetes and using the NNfunctionRegression to get an average value for the targets of the test data. After that the
targets of the test data was compared to the calculated average values and the mean squared error and the root mean squared error were
calculated. The calculations for these errors were executed for multiple values of k and the results were visualised in subplots.
'''
Diabetes=load_diabetes()
nn=50
Z=Diabetes.data
k=range(1,nn) 
MSEList=[]
RMSEList=[]
a=np.zeros((221,2))
for j in range(1,nn):
    results=NNfunctionRegression(j,Diabetes,10)
    Test=Diabetes.target[221:,np.newaxis]
    MSE= sum((Test-results)**2)/221
    RMSEList.append(np.sqrt(MSE))
    MSEList.append(MSE)

  
f, axis = plt.subplots(1,2,sharex=False, sharey=False)

axis[0].plot(k, MSEList) 
axis[0].set_xlabel('No Nearest Neighbours(-)')
axis[0].set_ylabel('Mean Squared Error')

axis[1].plot(k,RMSEList)
axis[1].set_xlabel('No Nearest Neighbours(-)')
axis[1].set_ylabel('Root Mean Squared Error')

plt.show()
# -

# In the subplot above, the Mean Squared Error and the Root Mean Squared Error were plotted as a function of the amount of Neighbours that were used in the Nearest Neighbour Regression. When we compare these results with the results of the linear regression, we can see that the MSE of the KNN regression classifier is bigger than the MSE of the linear regression classifier. 

# ### Class-conditional probability
#
# Compute and visualize the class-conditional probability (conditional probability where the class label is the conditional variable, i.e. $P(X = x \mid Y = y)$ for all features in the breast cancer dataset. Assume a Gaussian distribution.
#
# <p><font color='#770a0a'>Based on visual analysis of the plots, which individual feature can best discriminate between the two classes? Motivate your answer.</font></p>
#
#

# +
import class_cond_prop
import matplotlib.pyplot as plt
from class_cond_prop import *
il.reload(class_cond_prop)

x_cancer = breast_cancer.data
y_cancer = breast_cancer.target[:, np.newaxis]

# compute matrix containing all conditional probabilities per attribute
CP_pos, CP_neg = cond_prop(x_cancer, y_cancer)

# indices for positive and negative class
i = y_cancer[:, 0] > 0   # positive class
j = y_cancer[:, 0] == 0  # negative class

# plotting the (normalised) conditional probabilities (CP)
fig, axes = plt.subplots(nrows=5, ncols=6, figsize=(30, 25), sharey=True)
idx = 0
for row in axes:
    for cell in row:
        negx, posx = x_cancer[j, idx], x_cancer[i, idx]
        negy, posy = CP_neg[:, idx], CP_pos[:, idx]

        orderneg = np.argsort(negx)
        orderpos = np.argsort(posx)
        
        distneg = cell.plot(negx[orderneg], negy[orderneg], 'b-', linewidth=3)
        distpos = cell.plot(posx[orderpos], posy[orderpos], 'r-', linewidth=3)
        
        cell.set_xlabel("Feature value")
        cell.set_ylabel("Conditional Probability")
        cell.set_title("Feature "+ str(idx+1))
        idx += 1
plt.show()
# -

# These graphs show the probability of measuring a particular attribute value X given the instance is in class Y (0, blue or 1, red). The conditional probabilities were normalised, i.e. summing the probabilities over all instances for one feature would yield 1.
#
# The best feature to discriminate between the two classes is feature 28 since the probability distributions of class 0 and 1 do least overlap and the probability values are (relatively) high. For attribute values from approximately 0 to 0.15 there is a high probability these values occur given that the instance is in class 0 and there is low/no probability these values occur given that the instance is in class 1. For attribute values from 0.15 to 0.30 vice versa.
#
# Based on the conditional probability feature 19 would e.g. not be a good feature to use since the graphs overlap a lot. The probability to measure a particular attribute value X is high for both classes. 
#

StarcoderdataJupyterScriptsDedupFiltered