<jupyter_start><jupyter_text><jupyter_code># preamble to be able to run notebooks in Jupyter and Colab
try:
    from google.colab import drive
    import sys
    
    drive.mount('/content/drive')
    notes_home = "/content/drive/My Drive/CSC310/notes/"
    user_home = "/content/drive/My Drive/"
    
    sys.path.insert(1,notes_home) # let the notebook access the notes folder

except ModuleNotFoundError:
    notes_home = "" # running native Jupyter environment -- notes home is the same as the notebook
    user_home = ""  # under Jupyter we assume the user directory is the same as the notebook<jupyter_output><empty_output><jupyter_text>Classification Confidence Intervals **Observation:** It does not matter how careful we are with our model evaluation techniques, there remains a fundamental uncertainty about the ability of our training data to effectively represent our (possibly infinite) data universe.This uncertainty reflects into our model evaluation. If our training data is a poor representation of the data universe then the models we construct using it will generalize poorly to the rest of the data universe. If our training data is a good representation of the data universe then we can expect that our model will generalize well.Here we will deal with this uncertainty using *confidence intervals*. First, let us define confidence intervals formally. Given a model accuracy, *acc*, then the confidence interval is defined as the probability *p that our model accuracy acc* lies between some lower bound *lb* and some upper bound *ub*, \begin{equation}Pr(lb ≤ acc ≤ ub) = p\end{equation} Paraphrasing this equation with *p = 95%*:>We are 95% percent sure that our model accuracy is not worse than *lb* and not better than *ub*.Ultimitely we are interested in the lower and upper bounds of the 95% confidence interval. We can use the following formula to compute the bounds: \begin{equation}ub = acc + \sqrt\frac{acc(1-acc)}{n}\end{equation} \begin{equation}lb = acc - \sqrt\frac{acc(1-acc)}{n}\end{equation} Here, *n* is the number of observations in the testing dataset used to estimate *acc*. The constant 1.96 is called the *z-score* and expresses the fact that we are computing the 95% confidence interval.  Classification Confidence Intervals in Python<jupyter_code># cross-validation Iris
import pandas as pd
import numpy as np
np.set_printoptions(formatter={'float_kind':"{:3.2f}".format})
from sklearn import tree
# grab cross validation code
from sklearn.model_selection import cross_val_score

# get data
df = pd.read_csv(notes_home+"assets/iris.csv")
X  = df.drop(['id','Species'],axis=1)
y = df['Species']

# set up the model
model = tree.DecisionTreeClassifier(criterion='entropy', max_depth=2)

# do the 5-fold cross validation
scores = cross_val_score(model, X, y, cv=5)
print("Fold Accuracies: {}".format(scores))<jupyter_output>Fold Accuracies: [0.93 0.97 0.90 0.87 1.00]<jupyter_text>Let's do a simple example using the function classification_confint.<jupyter_code># compute 95% confidence intervals for classification and regression
# problems

def classification_confint(acc, n):
    '''
    Compute the 95% confidence interval for a classification problem.
      acc -- classification accuracy
      n   -- number of observations used to compute the accuracy
    Returns a tuple (lb,ub)
    '''
    import math
    interval = 1.96*math.sqrt(acc*(1-acc)/n)
    lb = max(0, acc - interval)
    ub = min(1.0, acc + interval)
    return (lb,ub)

def regression_confint(rs_score, n, k):
    '''
    Compute the 95% confidence interval for a regression problem.
      rs_score -- R^2 score
      n        -- number of observations used to compute the R^2 score
      k        -- number of independent variables in dataset
    Returns a tuple (lb,ub)
    Reference:
    https://books.google.com/books?id=gkalyqTMXNEC&pg=PA88#v=onepage&q&f=false
    '''
    import math
    interval = 2*math.sqrt((4*rs_score*(1-rs_score)**2*(n-k-1)**2)/((n**2 - 1)*(n+3)))
    lb = max(0, rs_score - interval)
    ub = min(1.0, rs_score + interval)
    return (lb,ub)
observations = 100
acc = .88
lb,ub = classification_confint(acc,observations)
print('Accuracy: {} ({:3.2f},{:3.2f})'.format(acc,lb, ub))<jupyter_output>Accuracy: 0.88 (0.82,0.94)<jupyter_text>Now, let's do an actual example using the Wisconsin breast cancer dataset. We want to print out the testing accuracy together with it's 95% confidence interval.<jupyter_code>import pandas as pd
from sklearn import tree
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

# read the data
df = pd.read_csv(notes_home+"assets/wdbc.csv")

# set up the feature matrix and target vector
X  = df.drop(['ID','Diagnosis'],axis=1)
y = df['Diagnosis']

# split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8, test_size=0.2, random_state=2)

# set up the tree model object - limit the complexity to put us somewhere in the middle of the graph.
model = tree.DecisionTreeClassifier(criterion='entropy', max_depth=4, random_state=1)

# fit the model on the training set of data
model.fit(X_train, y_train)

# Test results: evaluate the model on the testing set of data
y_test_model = model.predict(X_test)
acc = accuracy_score(y_test, y_test_model)
observations = X_test.shape[0]
lb,up = classification_confint(acc, observations)
print("Accuracy: {:3.2f} ({:3.2f},{:3.2f})".format(acc,lb,ub))<jupyter_output>Accuracy: 0.91 (0.86,0.94)<jupyter_text>Regression Confidence Intervals When performing regression we use the $R^2$ score to examine the quality of our models. Given that we only use a small training dataset for fitting the model compared to the rest of the data universe it is only natural to ask what the 95% confidence interval for this score might be. We have a formula for that -- it is not as straight forward as the confidence interval for classification, \begin{equation}lb = R^2 - 2\sqrt\frac{4R^2(1-R^2)(n-k-1)}{(n^2-1)(n+3)}\end{equation} \begin{equation}ub = R^2 + 2\sqrt\frac{4R^2(1-R^2)(n-k-1)}{(n^2-1)(n+3)}\end{equation} Here, n is the number of observations in the validation/testing dataset and k is the number of independent variables.<jupyter_code># from assets.confint import regression_confint

rs_score = .75
observations = 100
variables = 4 # independent variables

lb,ub = regression_confint(rs_score, observations, variables)
print("R^2 Score: {:3.2f} ({:3.2f}, {:3.2f})".format(rs_score,lb,ub))<jupyter_output>R^2 Score: 0.75 (0.67, 0.83)<jupyter_text>Let's look at an actual regression problem and compute the R2 score and it's 95% confidence interval. We will use the cars problem from before.<jupyter_code>import numpy as np
import pandas
from sklearn.tree import DecisionTreeRegressor
#from assets.confint import regression_confint

# get our dataset
cars_df = pandas.read_csv(notes_home+"assets/cars.csv")

# build model object
model = DecisionTreeRegressor(max_depth=None)

# fit model
# We have to reshape the values array to make 'fit' happy because
# the array only has a single feature
model.fit(cars_df['speed'].values.reshape(-1,1),cars_df['dist'])

# R^2 score
rs_score = model.score(cars_df['speed'].values.reshape(-1,1),cars_df['dist'])
observations = cars_df.shape[0]
variables = 1
lb,ub = regression_confint(rs_score, observations, variables)

# print out R^2 score with its 95% confidence interval
print("R^2 Score: {:3.2f} ({:3.2f}, {:3.2f})".format(rs_score,lb,ub))<jupyter_output>R^2 Score: 0.79 (0.69, 0.89)<jupyter_text>Statistical Significance Besides giving us an idea of the uncertainty of our model the 95% confidence intervals also have something to say about the significance of scores of different models. That is, if the confidence intervals overlap then the difference in model performance of two different models on the same dataset is not statistically significant.Consider the following,<jupyter_code>#from assets.confint import classification_confint

observations = 100

# first classifier
acc1 = .88
lb1,ub1 = classification_confint(acc1,observations)
print('Accuracy: {} ({:3.2f},{:3.2f})'.format(acc1,lb1, ub1))

# second classifier
acc2 = .92
lb2,ub2 = classification_confint(acc2,observations)
print('Accuracy: {} ({:3.2f},{:3.2f})'.format(acc2,lb2, ub2))<jupyter_output>Accuracy: 0.88 (0.82,0.94)
Accuracy: 0.92 (0.87,0.97)
StarcoderdataJupyterStructuredCleanDedup