<jupyter_start><jupyter_text>Introduction to Data Science  Lecture 22: Neural Networks I*COMP 5360 / MATH 4100, University of Utah, http://datasciencecourse.net/*In this lecture, we'll discuss Neural Networks, which can be used for both Classification and Regression. In particular, we'll discuss * perceptrons and multi-layer perceptrons (MLP)* neural networks with scikit-learn* how to train a neural network* intro to TensorFlowRecommended Reading:* <NAME>, [Hands-On Machine Learning with Scikit-Learn & TensorFlow](http://proquest.safaribooksonline.com/book/programming/9781491962282) (2017), Ch. 9,10. See also the [associated github page](https://github.com/ageron/handson-ml). * Welch Labs, [Neural Networks demystified](https://github.com/stephencwelch/Neural-Networks-Demystified)* <NAME> and <NAME> and <NAME>, [Deep Learning](http://www.deeplearningbook.org/)<jupyter_code>import numpy as np
from sklearn import datasets
from sklearn.datasets import load_iris, make_moons, load_breast_cancer, fetch_mldata, fetch_california_housing
from sklearn.linear_model import Perceptron
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, MinMaxScaler
from sklearn.neural_network import MLPClassifier, MLPRegressor
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.linear_model import LinearRegression

import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
from mpl_toolkits.mplot3d import Axes3D
%matplotlib inline
plt.rcParams['figure.figsize'] = (10, 6)
plt.style.use('ggplot')<jupyter_output><empty_output><jupyter_text>Neural Networks and deep learning[Artificial Neural Networks](https://en.wikipedia.org/wiki/Artificial_neural_network) were originally motivated by the brain, which is composed of a network of neurons. Each neuron recieves a (chemical) signal from other neurons, does a small computation and then decides if and how to release more chemicals. This composition of small calculations can perform complicated tasks! Similiarly, an artificial neural network is a network composed of neurons, which we simply think of as a computational units. Large scale neural networks are at the core of [deep learning](https://en.wikipedia.org/wiki/Deep_learning), which has gained much publicity for performing very impressive machine learning tasks in the past few years, such as, * classifying billions of images (*e.g.*, Google Images)+ speech recognition (*e.g.*, Amazon's Alexa or Apple’s Siri)+ video recommendation (*e.g.*, YouTube), + beating the world champion at the game of Go (DeepMind’s AlphaGo).Neural Networks can generally be used for supervised learning tasks, such as classification and regression.   PerceptronsThe simplest neural network is called the [perceptron](https://en.wikipedia.org/wiki/Perceptron). The *perceptron* is an binary classifier. It maps a real input $x \in \mathbb R^m$ to a binary output$$f(x) = \begin{cases}1 & \textrm{if } \ w\cdot x + b > 0 \\0 & \textrm{otherwise}\end{cases}.$$Here, $w \in \mathbb R^m$ is a vector of weights and $b$ is a scalar called the *bias*. (This is very similiar to the binary classifier we saw when looking at support vector machines.) It is customary to represent this function by the following diagram. Just like for previous classification methods, we first *train* the network on data, which is to say that we find good choices of $w$ and $b$ for our application. Then we use the neural network to classify new data points. Of course, a single perceptron is only a linear discriminator (similiar to logistic regression and linear SVM). But things become much more interesting when you start composing many neurons, that is, considering networks with more *layers*. The way in which we put together the neurons is referred to as the **network architecture**. There are many ways to do this. Here is a peak at the [neural-network-zoo](http://www.asimovinstitute.org/neural-network-zoo/): <img src="neuralnetworks.png" title="http://www.asimovinstitute.org/neural-network-zoo/" width="700">There are many different network architectures. The most important thing about all of these neural networks is that there is an *input layer*, typically drawn on the left hand side and an *output layer*, typically drawn on the right hand side. The middle layers are sometimes called *hidden layers*. <img src="Colored_neural_network.svg" title="https://en.wikipedia.org/wiki/Artificial_neural_network/media/File:Colored_neural_network.svg" width="500">In all of these neural network designs, each layer has its own weight vectors and biases that need to be trained. Consequently, *training* a neural network is a much harder job than we have seen for previous methods. It also requires more data.   Neural Networks in PracticeIn the past, we have use scikit-learn for classification. Scikit-learn also has Neural Network library [here](http://scikit-learn.org/stable/modules/neural_networks_supervised.html). However, this implementation does not scale to large-scale applications (no GPU support or deep learning architectures). There are many other packages that have more complete implementations of neural networks. Here is a partial list with short statements taken from the packages.* [TensorFlow](https://github.com/tensorflow/tensorflow): TensorFlow™ is an open source C++ software library for numerical computation using data flow graphs. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) communicated between them. The flexible architecture allows you to deploy computation to one or more CPUs or GPUs in a desktop, server, or mobile device with a single API. + [Torch](http://torch.ch/): Torch is a scientific computing framework with wide support for machine learning algorithms that puts GPUs first.+ [CNTK](https://github.com/Microsoft/cntk)  Cognitive Toolkit (CNTK) is an open source deep-learning toolkit developed by Microsoft. + [Theano](https://github.com/Theano/Theano): Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It can use GPUs and perform efficient symbolic differentiation.+ [keras](https://github.com/keras-team/keras): Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano.+ [MXNet](https://github.com/dmlc/mxnet): Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more+ [Caffe](http://caffe.berkeleyvision.org/): Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research (BAIR) and by community contributors.+ [Lasagne](https://github.com/Lasagne/Lasagne): Lightweight library to build and train neural networks in Theano. + [prettytensor](https://github.com/google/prettytensor/): Pretty Tensor provides a high level builder API for TensorFlow. It provides thin wrappers on Tensors so that you can easily build multi-layer neural networks.+ [Deeplearning4j](https://deeplearning4j.org/): Open-Source, Distributed, Deep Learning Library for the JVM+ [H2O](https://github.com/h2oai): Fast Scalable Machine Learning For Smarter ApplicationsWe'll start with the scikit-learn implementation, since this environment is familiar and then use TensorFlow later.   Neural networks with scikit-learnscikit-learn has a few different neural network functions:1. [perceptron](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Perceptron.html)+ [multi-layer perceptron (MLP) classifier](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html)+ [multi-layer perceptron (MLP) regressor](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html)The scikit-learn user guide for supervised learning using neural networks is [here](http://scikit-learn.org/stable/modules/neural_networks_supervised.html). Let's first test the `MLPClassifier` on the [two moons dataset](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html).<jupyter_code># X contains two features
# y contains labels
X,y = make_moons(n_samples=1000,random_state=1,noise=0.2)
X = StandardScaler().fit_transform(X)

# Plot the data, color by class
plt.scatter(X[y == 1, 0], X[y == 1, 1], color="DarkBlue", marker="s",label="class 1")
plt.scatter(X[y == 0, 0], X[y == 0, 1], color="DarkRed", marker="o",label="class 2")
plt.legend(scatterpoints=1)
plt.title('Two Moons Dataset')
plt.xlabel('feature 1')
plt.ylabel('feature 2')
plt.show()
model = MLPClassifier(hidden_layer_sizes=(3,3,3), max_iter=1000, alpha=1e-4,
                    solver='adam', verbose=10, random_state=1, 
                    learning_rate_init=.1)
model.fit(X, y)
# Plot the data, color by class
plt.scatter(X[y == 1, 0], X[y == 1, 1], color="DarkBlue", marker="s",label="class 1")
plt.scatter(X[y == 0, 0], X[y == 0, 1], color="DarkRed", marker="o",label="class 2")
plt.legend(scatterpoints=1)

# Plot the predictions made by NN
x_min, x_max = X[:,0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),np.linspace(y_min, y_max, 200))
zz = model.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
plt.contourf(xx, yy, zz, cmap=ListedColormap(['DarkRed', 'DarkBlue']), alpha=.2)
plt.contour(xx, yy, zz, colors="black", alpha=1, linewidths=0.2) 

plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)
plt.title('Classification of Two Moons using MLPClassifier')
plt.xlabel('feature 1')
plt.ylabel('feature 2')
plt.show()
print('After ', model.n_iter_, ' iterations, the loss is ', model.loss_)
print('model coef shapes')
[print(coef.shape) for coef in model.coefs_]
print('model coefs')
[print(coef) for coef in model.coefs_]
print('model intercepts')
[print(coef) for coef in model.intercepts_]

print(model.get_params())<jupyter_output>After  26  iterations, the loss is  0.10547183623600889
model coef shapes
(2, 3)
(3, 3)
(3, 3)
(3, 1)
model coefs
[[-1.45538319  1.95266486 -2.19153839]
 [ 0.48119992 -0.52172742 -1.14917163]]
[[ 0.84881629 -1.21287704  1.98892248]
 [-0.31298184  1.607918   -0.55635986]
 [ 1.04945602  1.3496739  -0.47041188]]
[[-0.80397104 -1.7798498   2.82996346]
 [ 0.28897265  1.58600398 -0.9980886 ]
 [-2.12063679  0.16940877  1.1990305 ]]
[[ 0.29733504]
 [ 1.20505347]
 [-1.5105632 ]]
model intercepts
[-0.64310462 -1.12094634  0.94005427]
[-2.86559139 -0.17624245  1.47203736]
[-1.33906156  0.06421013  1.42286222]
[-0.06217724]
{'alpha': 0.0001, 'shuffle': True, 'warm_start': False, 'tol': 0.0001, 'max_iter': 1000, 'momentum': 0.9, 'nesterovs_momentum': True, 'beta_1': 0.9, 'learning_rate_init': 0.1, 'verbose': 10, 'early_stopping': False, 'power_t': 0.5, 'epsilon': 1e-08, 'beta_2': 0.999, 'validation_fraction': 0.1, 'hidden_layer_sizes': (3, 3, 3), 'activation': 'relu', 'solver': 'adam', 'learning_ra[...]<jupyter_text>There are a lot more function parameters for [`MLPClassifier`](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html) than for other scikit-learn classification methods. You'll find that tweaking them also makes a very big difference in the output. Here are some of the important parameters: Network architecture parameters+ **hidden_layer_sizes**: tuple, length = n_layers - 2, default (100,). The ith element represents the number of neurons in the ith hidden layer.+ **activation**: {‘identity’, ‘logistic’, ‘tanh’, ‘relu’}, default ‘relu’. Activation function for the hidden layer. - ‘identity’, no-op activation, useful to implement linear bottleneck, returns f(x) = x - ‘logistic’, the logistic sigmoid function, returns f(x) = 1 / (1 + exp(-x)). - ‘tanh’, the hyperbolic tan function, returns f(x) = tanh(x). - ‘relu’, the rectified linear unit function, returns f(x) = max(0, x) Optimization related parameters+ **solver**: {‘lbfgs’, ‘sgd’, ‘adam’}, default ‘adam’. The solver for weight optimization. - ‘lbfgs’ is an optimizer in the family of quasi-Newton methods. - ‘sgd’ refers to stochastic gradient descent. - ‘adam’ refers to a stochastic gradient-based optimizer proposed by Kingma, Diederik, and <NAME>+ **alpha**: float, optional, default 0.0001.L2 penalty (regularization term) parameter.+ **max_iter**: int, optional, default 200. Maximum number of iterations. The solver iterates until convergence (determined by ‘tol’) or this number of iterations. For stochastic solvers (‘sgd’, ‘adam’), note that this determines the number of epochs (how many times each data point will be used), not the number of gradient steps.+ **random_state**: int, RandomState instance or None, optional, default None. If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random.+ **tol**: float, optional, default 1e-4. Tolerance for the optimization. When the loss or score is not improving by at least tol for two consecutive iterations, unless learning_rate is set to ‘adaptive’, convergence is considered to be reached and training stops.+ **verbose**: bool, optional, default False. Whether to print progress messages to stdout. **Exercise**: By chainging the hidden_layer_sizes, activation function, and the random_state, see if you can find a better classification of the above two moons dataset.   Multi-layer PerceptronsHere, we take a closer look at Multi-layer Perceptrons; this material is taken from the [scikit-learn user guide](http://scikit-learn.org/stable/modules/neural_networks_supervised.html)Given a set of features $X = \{x_1, x_2, ..., x_n\}$ and a target $y$, a **Multi-layer Perceptron** can learn a non-linear function for either classification or regression. Here is an example with one hidden layer.  <img src="multilayerperceptron_network.png" title="http://scikit-learn.org/stable/_images/multilayerperceptron_network.png" width="300">Each neuron applies an affine transformation $$x \mapsto w\cdot x + b$$and then a non-linear *activation function* $g\colon \mathbb R \to \mathbb R$. The composition looks like:$$x \mapsto g(w\cdot x + b).$$The output layer receives the values from the last hidden layer and transforms them into output values.In the previous example, we used saw the trained cofficients by printing `model.coefs_` and `model.intercepts_`. There are several choices of **activation function**: hyperbolic tangent, logistic, and rectified linear unit (ReLU). We used the default activiation function, ReLU.<jupyter_code># see <NAME>. 10

def logit(z):
    return 1 / (1 + np.exp(-z))

def relu(z):
    return np.maximum(0, z)

def derivative(f, z, eps=0.000001):
    return (f(z + eps) - f(z - eps))/(2 * eps)

z = np.linspace(-5, 5, 200)

plt.figure(figsize=(11,4))

plt.subplot(121)
plt.plot(z, np.sign(z), "r-", linewidth=2, label="Step")
plt.plot(z, logit(z), "g--", linewidth=2, label="Logit")
plt.plot(z, np.tanh(z), "b-", linewidth=2, label="Tanh")
plt.plot(z, relu(z), "m-.", linewidth=2, label="ReLU")
plt.grid(True)
plt.legend(loc="center right", fontsize=14)
plt.title("Activation functions", fontsize=14)
plt.axis([-5, 5, -1.2, 1.2])

plt.subplot(122)
plt.plot(z, derivative(np.sign, z), "r-", linewidth=2, label="Step")
plt.plot(0, 0, "ro", markersize=5)
plt.plot(0, 0, "rx", markersize=10)
plt.plot(z, derivative(logit, z), "g--", linewidth=2, label="Logit")
plt.plot(z, derivative(np.tanh, z), "b-", linewidth=2, label="Tanh")
plt.plot(z, derivative(relu, z), "m-.", linewidth=2, label="ReLU")
plt.grid(True)
#plt.legend(loc="center right", fontsize=14)
plt.title("Derivatives", fontsize=14)
plt.axis([-5, 5, -0.2, 1.2])

plt.show()<jupyter_output><empty_output><jupyter_text>Training a neural networkWe'll follow the scikit-learn  [user guide](http://scikit-learn.org/stable/modules/neural_networks_supervised.html) to see how the multi-layer perceptron (MLP) neural network is trained. The MLP uses a loss function of the form $$Loss(\hat{y},y,W) =  \frac{1}{2} \sum_{i=1}^n f(\hat{y}_i(W),y_i) + \frac{\alpha}{2} \|W\|_2^2$$Here, + $y_i$ are the labels for the $i$-th example, + $\hat{y}_i(W)$ are the predicted label for the $i$-th example, + $f$ is a function that measures the error, typically $L^2$ difference for regression or cross-entropy for classification, and + $\alpha$ is a regularization parameter. Starting from initial random weights, the loss function is minimized by repeatedly updating these weights. The details of this depend on the chosen method, either a quasi-Newton method `lbfgs`, stochastic gradident descent `sgd`, or `adam`. In the **gradient descent method**, the gradient $\nabla_{W} Loss$ of the loss with respect to the weights is computed. The weights are then changed in the negative gradient direction using a step-length or learning-rate $\varepsilon>0$: $$W \leftarrow W - \varepsilon \nabla_W {Loss}.$$The algorithm stops when it reaches a preset maximum number of iterations, `max_iter`, or when the improvement in loss is below a preset small number, `tol`.The gradient of $W$ is simply computed using the chain rule from calculus. In principle the idea is simple, but in practice it is a complicated job. Data analysts have figured out a clever way how to organize this calculation. This is sometimes called *back propagation*. A complete description of `lbfgs`, `sgd`, and `adam` is beyond the scope of the course. I'll just say that they are clever modifications to the gradient descent method. Let's see a comparison of optimization methods, taken from [this page](http://scikit-learn.org/stable/auto_examples/neural_networks/plot_mlp_training_curves.html).<jupyter_code># different learning rate schedules and momentum parameters
params = [{'solver': 'sgd', 'learning_rate': 'constant', 'momentum': 0,
           'learning_rate_init': 0.2},
          {'solver': 'sgd', 'learning_rate': 'constant', 'momentum': .9,
           'nesterovs_momentum': False, 'learning_rate_init': 0.2},
          {'solver': 'sgd', 'learning_rate': 'constant', 'momentum': .9,
           'nesterovs_momentum': True, 'learning_rate_init': 0.2},
          {'solver': 'sgd', 'learning_rate': 'invscaling', 'momentum': 0,
           'learning_rate_init': 0.2},
          {'solver': 'sgd', 'learning_rate': 'invscaling', 'momentum': .9,
           'nesterovs_momentum': True, 'learning_rate_init': 0.2},
          {'solver': 'sgd', 'learning_rate': 'invscaling', 'momentum': .9,
           'nesterovs_momentum': False, 'learning_rate_init': 0.2},
          {'solver': 'adam', 'learning_rate_init': 0.01}]

labels = ["constant learning-rate", "constant with momentum",
          "constant with Nesterov's momentum",
          "inv-scaling learning-rate", "inv-scaling with momentum",
          "inv-scaling with Nesterov's momentum", "adam"]

plot_args = [{'c': 'red', 'linestyle': '-'},
             {'c': 'green', 'linestyle': '-'},
             {'c': 'blue', 'linestyle': '-'},
             {'c': 'red', 'linestyle': '--'},
             {'c': 'green', 'linestyle': '--'},
             {'c': 'blue', 'linestyle': '--'},
             {'c': 'black', 'linestyle': '-'}]


def plot_on_dataset(X, y, ax, name):
    # for each dataset, plot learning for each learning strategy
    print("\nlearning on dataset %s" % name)
    ax.set_title(name)
    X = MinMaxScaler().fit_transform(X)
    mlps = []
    if name == "digits":
        # digits is larger but converges fairly quickly
        max_iter = 15
    else:
        max_iter = 400

    for label, param in zip(labels, params):
        print("training: %s" % label)
        mlp = MLPClassifier(verbose=0, random_state=0,
                            max_iter=max_iter, **param)
        mlp.fit(X, y)
        mlps.append(mlp)
        print("Training set score: %f" % mlp.score(X, y))
        print("Training set loss: %f" % mlp.loss_)
    for mlp, label, args in zip(mlps, labels, plot_args):
            ax.plot(mlp.loss_curve_, label=label, **args)


fig, axes = plt.subplots(2, 2, figsize=(15, 10))
# load / generate some toy datasets
iris = datasets.load_iris()
digits = datasets.load_digits()
data_sets = [(iris.data, iris.target),
             (digits.data, digits.target),
             datasets.make_circles(noise=0.2, factor=0.5, random_state=1),
             datasets.make_moons(noise=0.3, random_state=0)]

for ax, data, name in zip(axes.ravel(), data_sets, ['iris', 'digits',
                                                    'circles', 'moons']):
    plot_on_dataset(*data, ax=ax, name=name)

fig.legend(ax.get_lines(), labels, ncol=3, loc="upper center")
plt.show()<jupyter_output>learning on dataset iris
training: constant learning-rate<jupyter_text>Some advise on optimization methods according to [this page](http://scikit-learn.org/stable/modules/neural_networks_supervised.html): * Empirically, we observed that L-BFGS converges faster and with better solutions on small datasets. For relatively large datasets, however, Adam is very robust. It usually converges quickly and gives pretty good performance. SGD with momentum or nesterov’s momentum, on the other hand, can perform better than those two algorithms if learning rate is correctly tuned.  Example: breast cancer data<jupyter_code>cancer = load_breast_cancer()
print(cancer.keys())

# 569 data points with 30 features
cancer['data'].shape

# full description:
print(cancer['DESCR'])

X = cancer['data']
y = cancer['target']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.2)

scaler = StandardScaler()
scaler.fit(X_train) # Fit only to the training data

# Apply scaling to data
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

mlp = MLPClassifier(hidden_layer_sizes=(30,30,30),random_state=1)
mlp.fit(X_train,y_train)

y_pred = mlp.predict(X_test)
print(confusion_matrix(y_test,y_pred))<jupyter_output>[[36  4]
 [ 1 73]]<jupyter_text>Example: MNIST datasetLet's train a multi layer perceptron on the MNIST dataset.<jupyter_code>mnist = fetch_mldata("MNIST original")

# rescale the data, use the traditional train/test split
X, y = mnist.data / 255., mnist.target
X_train, X_test = X[:60000], X[60000:]
y_train, y_test = y[:60000], y[60000:]
mlp = MLPClassifier(hidden_layer_sizes=(100, 100), max_iter=400, alpha=1e-4,
                     solver='adam', verbose=10, tol=1e-4, random_state=1)
mlp.fit(X_train, y_train)

print("Training set score: %f" % mlp.score(X_train, y_train))
print("Test set score: %f" % mlp.score(X_test, y_test))

y_pred = mlp.predict(X_test)
print(confusion_matrix(y_test,y_pred))<jupyter_output>Iteration 1, loss = 0.40451061
Iteration 2, loss = 0.15583057
Iteration 3, loss = 0.11098385
Iteration 4, loss = 0.08411502
Iteration 5, loss = 0.06846576
Iteration 6, loss = 0.05652346
Iteration 7, loss = 0.04633846
Iteration 8, loss = 0.03762209
Iteration 9, loss = 0.03365538
Iteration 10, loss = 0.02884553
Iteration 11, loss = 0.02224694
Iteration 12, loss = 0.01873422
Iteration 13, loss = 0.01647119
Iteration 14, loss = 0.01460615
Iteration 15, loss = 0.01141324
Iteration 16, loss = 0.01314417
Iteration 17, loss = 0.01043379
Iteration 18, loss = 0.00753701
Iteration 19, loss = 0.00898900
Iteration 20, loss = 0.00699743
Iteration 21, loss = 0.00740891
Iteration 22, loss = 0.00942422
Iteration 23, loss = 0.00850921
Training loss did not improve more than tol=0.000100 for two consecutive epochs. Stopping.
Training set score: 0.997483
Test set score: 0.976600
[[ 971    0    0    1    1    2    2    1    2    0]
 [   0 1124    1    4    0    1    2    1    2    0]
 [   3    1 1004   11 [...]<jupyter_text>**Exercise**: By adjusting the parameters in the MLPClassifier, imporve the test set score.     **Note**: [This webpage](http://scikit-learn.org/stable/auto_examples/neural_networks/plot_mnist_filters.html)tries to interpret the MLP classification weights learned for the MNIST dataset.   Example: Regression with Neural Networks in scikit-learnLet's use a multi-layer perceptron for regression. This can be done with the scikit-learn [`MLPRegressor`](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html) function.<jupyter_code>housing = fetch_california_housing()

print(housing.keys())

# 20640 data points with 8 features
housing['data'].shape

# full description:
print(housing['DESCR'])

X = housing['data']
y = housing['target']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.2)

# scale data
scaler = StandardScaler()
scaler.fit(X_train) # Fit only to the training data

# Apply scaling to data
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
# Linear Regression with Scikit-Learn
lin_reg = LinearRegression()
print(lin_reg.get_params())

lin_reg.fit(X_train, y_train)
print(lin_reg.intercept_)
print(lin_reg.coef_)

print(lin_reg.score(X_test,y_test)) # score = 1 is good
# MLP regression with Scikit-Learn
mlp_reg = MLPRegressor(hidden_layer_sizes=(8,8),verbose=0,random_state=2,solver='adam')
print(mlp_reg.get_params())

mlp_reg.fit(X_train, y_train)

print(mlp_reg.score(X_test,y_test)) # score = 1 is good<jupyter_output>{'alpha': 0.0001, 'shuffle': True, 'warm_start': False, 'tol': 0.0001, 'max_iter': 200, 'momentum': 0.9, 'nesterovs_momentum': True, 'beta_1': 0.9, 'learning_rate_init': 0.001, 'verbose': 0, 'early_stopping': False, 'power_t': 0.5, 'epsilon': 1e-08, 'beta_2': 0.999, 'validation_fraction': 0.1, 'hidden_layer_sizes': (8, 8), 'activation': 'relu', 'solver': 'adam', 'learning_rate': 'constant', 'batch_size': 'auto', 'random_state': 2}
0.7400491068801395<jupyter_text>Neural Networks demystifiedThere is a good sequence of 7 videos called **Neural Networks demystified** from Welch Labs that builds and trains a neural network *from scratch* in python. * Part 1: [Data + Architecture](https://www.youtube.com/watch?v=bxe2T-V8XRs)+ Part 2: [Forward Propagation](https://www.youtube.com/watch?v=UJwK6jAStmg)+ Part 3: [Gradient Descent](https://www.youtube.com/watch?v=5u0jaA3qAGk)+ Part 4: [Backpropagation](https://www.youtube.com/watch?v=GlcnxUlrtek)+ Part 5: [Numerical Gradient Checking](https://www.youtube.com/watch?v=pHMzNW8Agq4&t=22s)+ Part 6: [Training](https://www.youtube.com/watch?v=9KM9Td6RVgQ)+ Part 7: [Overfitting, Testing, and Regularization](https://www.youtube.com/watch?v=S4ZUwgesjS8)If you're interested in learning more about how Neural Networks are trained, I would recommend watching these videos. The accompanying code is [on github](https://github.com/stephencwelch/Neural-Networks-Demystified) and can be obtained via ```git clone https://github.com/stephencwelch/Neural-Networks-Demystified.git```  Neural Networks with TensorFlow"[TensorFlow™](https://www.tensorflow.org/) is an open source software library for numerical computation using data flow graphs. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) communicated between them. The flexible architecture allows you to deploy computation to one or more CPUs or GPUs in a desktop, server, or mobile device with a single API. TensorFlow was originally developed by researchers and engineers working on the Google Brain Team within Google's Machine Intelligence research organization for the purposes of conducting machine learning and deep neural networks research, but the system is general enough to be applicable in a wide variety of other domains as well."  Installing TensorFlowInstructions for installing TensorFlow are available at [the tensorflow install page](https://www.tensorflow.org/versions/r1.0/install/).It is recommended that you use the command: ```sudo pip3 install --upgrade tensorflow```I used the command: ```sudo conda install -c conda-forge tensorflow```<jupyter_code>import tensorflow as tf
print(tf.__version__)<jupyter_output>/opt/anaconda3/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: compiletime version 3.6 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.5
  return f(*args, **kwds)
/opt/anaconda3/lib/python3.5/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters<jupyter_text>Creating and running a graph with TensorFlow<jupyter_code># build a computational graph
x = tf.Variable(3, name="x")
y = tf.Variable(4, name="y")
f = x*x*y + y + 2

# evaluate the graph
with tf.Session() as sess:
    x.initializer.run()
    y.initializer.run()
    print(f.eval())<jupyter_output>42<jupyter_text>Linear regression using TensorFlow<jupyter_code>import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import LinearRegression


housing = fetch_california_housing()
m, n = housing.data.shape
housing_data_plus_bias = np.c_[np.ones((m, 1)), housing.data]

X = tf.constant(housing_data_plus_bias, dtype=tf.float32, name="X")
y = tf.constant(housing.target.reshape(-1, 1), dtype=tf.float32, name="y")
XT = tf.transpose(X)
theta = tf.matmul(tf.matmul(tf.matrix_inverse(tf.matmul(XT, X)), XT), y)

with tf.Session() as sess:
    theta_tf = theta.eval()
    
print(theta_tf, '\n')


# compare with pure NumPy
X = housing_data_plus_bias
y = housing.target.reshape(-1, 1)
theta_np = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)
print(theta_np, '\n')


# Compare with Scikit-Learn
lin_reg = LinearRegression()
lin_reg.fit(housing.data, housing.target.reshape(-1, 1))
print(np.r_[lin_reg.intercept_.reshape(-1, 1), lin_reg.coef_.T])<jupyter_output>[[-3.7465141e+01]
 [ 4.3573415e-01]
 [ 9.3382923e-03]
 [-1.0662201e-01]
 [ 6.4410698e-01]
 [-4.2513184e-06]
 [-3.7732250e-03]
 [-4.2664889e-01]
 [-4.4051403e-01]] 

[[-3.69419202e+01]
 [ 4.36693293e-01]
 [ 9.43577803e-03]
 [-1.07322041e-01]
 [ 6.45065694e-01]
 [-3.97638942e-06]
 [-3.78654265e-03]
 [-4.21314378e-01]
 [-4.34513755e-01]] 

[[-3.69419202e+01]
 [ 4.36693293e-01]
 [ 9.43577803e-03]
 [-1.07322041e-01]
 [ 6.45065694e-01]
 [-3.97638942e-06]
 [-3.78654265e-03]
 [-4.21314378e-01]
 [-4.34513755e-01]]
StarcoderdataJupyterStructuredCleanDedup