<jupyter_start><jupyter_text>Solving an overdetermined system using  pseudo inverseConsider the overdetermined system corresponding to cat-brain from Chapter 2.There are 15 training examples, each with input and desired outputs specified.Our goal is to determine 3 unkwnowns (w0, w1, b).This can be cast as an over-determined system of equations$$A\vec{w} = \vec{y}$$where$$ A =\begin{bmatrix}        0.11 & 0.09 & 1.00 \\        0.01 & 0.02 & 1.00 \\        0.98 & 0.91 & 1.00 \\        0.12 & 0.21 & 1.00 \\        0.98 & 0.99 & 1.00 \\        0.85 & 0.87 & 1.00 \\        0.03 & 0.14 & 1.00 \\        0.55 & 0.45 & 1.00 \\        0.49 & 0.51 & 1.00 \\        0.99 & 0.01 & 1.00 \\        0.02 & 0.89 & 1.00 \\        0.31 & 0.47 & 1.00 \\        0.55 & 0.29 & 1.00 \\        0.87 & 0.76 & 1.00 \\        0.63 & 0.24 & 1.00\end{bmatrix}\;\;\;\;\;\;\;\vec{y} = \begin{bmatrix}        -0.8 \\        -0.97 \\        0.89 \\         -0.67 \\         0.97 \\         0.72 \\         -0.83 \\         0.00 \\        0.00 \\        0.00 \\        -0.09 \\        -0.22 \\         -0.16 \\        0.63 \\        0.37\end{bmatrix}\;\;\;\;\;\;\;\vec{w} = \begin{bmatrix} w_{0}\\w_{1}\\b\end{bmatrix}$$We solve for $\vec{w}$ using the pseudo inverse formula $\space\space\large{\vec{w} = (A^TA)^{-1}A^Ty}$<jupyter_code>import numpy as np
# Let us revisit our cat brain data set
# Notice that there are 15 training examples, with 3
# unkwnowns (w0, w1, b).
# This is an over determined system.
# It can be easily seen that the solution is roughly
# $w_{0} = 1, w_{1} = 1, b = -1$.
# It has been deliberately chosen as such.
# But the equations are not fully consistent (i.e., there is
# no solution that satisfies all the equations).
# We want to find the best values such that it minimizes Aw - b.
# This is what the pseudo-inverse does.

def pseudo_inverse(A):
    return np.matmul(np.linalg.inv(np.matmul(A.T, A)), A.T)

# The usual cat-brain input dataset
X = np.array([[0.11, 0.09], [0.01, 0.02], [0.98, 0.91], [0.12, 0.21],
              [0.98, 0.99], [0.85, 0.87], [0.03, 0.14], [0.55, 0.45],
              [0.49, 0.51], [0.99, 0.01], [0.02, 0.89], [0.31, 0.47],
              [0.55, 0.29], [0.87, 0.76], [0.63, 0.24]])

# Output threat score modeled as a vector
y = np.array([-0.8, -0.97, 0.89, -0.67, 0.97, 0.72, -0.83, 0.00, 0.00,
              0.00, -0.09, -0.22, -0.16, 0.63, 0.37])
A = np.column_stack((X, np.ones(15)))

# Column stack will add an additional column of 1s to the training
# dataset to represent the coefficient of the bias
w = np.matmul(pseudo_inverse(A), y)

print("The solution is {}\n"
      "Note that this is almost equal to [1.0, 1.0, -1.0])".format(w))<jupyter_output>The solution is [ 1.07661761  0.89761672 -0.95816936]
Note that this is almost equal to [1.0, 1.0, -1.0])
StarcoderdataJupyterStructuredCleanDedup