<jupyter_start><jupyter_text>Analyzing with iNNvestigate **iNNvestigate** got created to make analyzing neural network's predictions easy! The library should help the user to focus on research and development by providing implemented analysis methods and facilitating rapid development of new methods. In this notebook we will show you how to use **iNNvestigate** and for a better understanding we recommend to read [iNNvestigate neural networks!](https://arxiv.org/abs/1808.04260) first! How to use **iNNvestigate** you can read in this notebook: [Developing with iNNvestigate](introduction_development.ipynb)-----**The intention behind iNNvestigate is to make it easy to use analysis methods, but it is not to explain the underlying concepts and assumptions. Please, read the according publication(s) when using a certain method and when publishing please cite the according paper(s) (as well as the [iNNvestigate paper](https://arxiv.org/abs/1808.04260)). Thank you!** You can find most related publication in [iNNvestigate neural networks!](https://arxiv.org/abs/1808.04260) and in the README file. Analysis methodsThe field of analyizing neural network's predictions is about gaining insights how and why a potentially complex network gave as output a certain value or choose a certain class over others. This is often called interpretability or explanation of neural networks. We just call it analyzing a neural network's prediction to be as neutral as possible and to leave any conclusions to the user.Most methods have in common that they analyze the input features w.r.t. a specific neuron's output. Which insights a method reveals about this output can be grouped into (see [Learning how to explain: PatternNet and PatternAttribution](https://arxiv.org/abs/1705.05598)):* **function:** analyzing the operations the network function uses to extract or compute the output. E.g., how would changing an input feature change the output.* **signal:** analyzing the components of the input that cause the output. E.g., which parts of an input image or which directions of an input are used to determine the output.* **attribution:** attributing the "importance" of input features for the output. E.g., how much would changing an input feature change the output.----In this notebook we will introduce methods for each of these categories and along show how to use different features of **iNNvestigate**, namely how to:* analyze a prediction.* train an analyzer.* analyze a prediction w.r.t to a specific output neuron.Let's dive right into it!  Training a networkTo analyze a network, we need a network! As a base for **iNNvestigate** we chose the Keras deep learning library, because it is easy to use and allows to inspect build models.In this first piece of code we import all the necessary modules:<jupyter_code>import warnings
warnings.simplefilter('ignore')
%matplotlib inline  

import imp
import matplotlib.pyplot as plot
import numpy as np
import os

import keras
import keras.backend
import keras.layers
import keras.models
import keras.utils

import innvestigate
import innvestigate.utils as iutils

# Use utility libraries to focus on relevant iNNvestigate routines.
mnistutils = imp.load_source("utils_mnist", "../utils_mnist.py")<jupyter_output>Using TensorFlow backend.<jupyter_text>to load the data:<jupyter_code># Load data
# returns x_train, y_train, x_test, y_test as numpy.ndarray
data_not_preprocessed = mnistutils.fetch_data()

# Create preprocessing functions
input_range = [-1, 1]
preprocess, revert_preprocessing = mnistutils.create_preprocessing_f(data_not_preprocessed[0], input_range)

# Preprocess data
data = (
    preprocess(data_not_preprocessed[0]), keras.utils.to_categorical(data_not_preprocessed[1], 10),
    preprocess(data_not_preprocessed[2]), keras.utils.to_categorical(data_not_preprocessed[3], 10),
)

if keras.backend.image_data_format == "channels_first":
    input_shape = (1, 28, 28)
else:
    input_shape = (28, 28, 1)<jupyter_output><empty_output><jupyter_text>and to now create and train a CNN model:<jupyter_code>model = keras.models.Sequential([
    keras.layers.Conv2D(32, (3, 3), activation="relu", input_shape=input_shape),
    keras.layers.Conv2D(64, (3, 3), activation="relu"),
    keras.layers.MaxPooling2D((2, 2)),
    keras.layers.Flatten(),
    keras.layers.Dense(512, activation="relu"),
    keras.layers.Dense(10, activation="softmax"),
])

model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
model.fit(data[0], data[1], epochs=20, batch_size=128)

scores = model.evaluate(data[2], data[3], batch_size=128)
print("Scores on test set: loss=%s accuracy=%s" % tuple(scores))<jupyter_output>Epoch 1/20
60000/60000 [==============================] - 19s 313us/step - loss: 0.1519 - acc: 0.9543
Epoch 2/20
60000/60000 [==============================] - 10s 161us/step - loss: 0.0405 - acc: 0.9876
Epoch 3/20
60000/60000 [==============================] - 10s 162us/step - loss: 0.0230 - acc: 0.9924
Epoch 4/20
60000/60000 [==============================] - 10s 161us/step - loss: 0.0148 - acc: 0.9948
Epoch 5/20
60000/60000 [==============================] - 10s 161us/step - loss: 0.0114 - acc: 0.9960
Epoch 6/20
60000/60000 [==============================] - 10s 161us/step - loss: 0.0075 - acc: 0.9975
Epoch 7/20
60000/60000 [==============================] - 10s 163us/step - loss: 0.0058 - acc: 0.9981
Epoch 8/20
60000/60000 [==============================] - 9s 150us/step - loss: 0.0065 - acc: 0.9979
Epoch 9/20
60000/60000 [==============================] - 8s 129us/step - loss: 0.0055 - acc: 0.9983
Epoch 10/20
60000/60000 [==============================] - 9s 144us/step - loss: 0.0[...]<jupyter_text>Analyzing a predicitionLet's first choose an image to analyze:<jupyter_code># Choosing a test image for the tutorial:
image = data[2][7:8]

plot.imshow(image.squeeze(), cmap='gray', interpolation='nearest')
plot.show()<jupyter_output><empty_output><jupyter_text>In this first part we show how to create and use an analyzer. To do so we use an analyzer from *function* category, namely the gradient. The gradient shows how the linearized network function reacts on changes of a single feature.This is simply done by passing the model without a softmax to the analyzer class:<jupyter_code># Stripping the softmax activation from the model
model_wo_sm = iutils.keras.graph.model_wo_softmax(model)

# Creating an analyzer
gradient_analyzer = innvestigate.analyzer.Gradient(model_wo_sm)

# Applying the analyzer
analysis = gradient_analyzer.analyze(image)

# Displaying the gradient
plot.imshow(analysis.squeeze(), cmap='seismic', interpolation='nearest')
plot.show()<jupyter_output><empty_output><jupyter_text>For convience there is a function that creates an analyzer for you. It passes all the parameter on to the class instantiation:<jupyter_code># Creating an analyzer
gradient_analyzer = innvestigate.create_analyzer("gradient", model_wo_sm)

# Applying the analyzer
analysis = gradient_analyzer.analyze(image)

# Displaying the gradient
plot.imshow(analysis.squeeze(), cmap='seismic', interpolation='nearest')
plot.show()<jupyter_output><empty_output><jupyter_text>To emphasize different compontents of the analysis many people use instead of the "plain" gradient the absolute value or the square of it. With the gradient analyzer this can be done specifying additional parameters when creating the analyzer:<jupyter_code># Creating a parameterized analyzer
abs_gradient_analyzer = innvestigate.create_analyzer("gradient", model_wo_sm, postprocess="abs")
square_gradient_analyzer = innvestigate.create_analyzer("gradient", model_wo_sm, postprocess="square")<jupyter_output><empty_output><jupyter_text>Similar other analyzers can be parameterized.Now we visualize the result by projecting the gradient into a gray-color-image:<jupyter_code># Applying the analyzers
abs_analysis = abs_gradient_analyzer.analyze(image)
square_analysis = square_gradient_analyzer.analyze(image)

# Displaying the analyses, use gray map as there no negative values anymore
plot.imshow(abs_analysis.squeeze(), cmap='gray', interpolation='nearest')
plot.show()
plot.imshow(square_analysis.squeeze(), cmap='gray', interpolation='nearest')
plot.show()<jupyter_output><empty_output><jupyter_text>Training an analyzerSome analyzers are data-dependent and need to be trained. In **iNNvestigate** this realized with a SKLearn-like interface. In the next piece of code we train the method PatternNet that analyzes the *signal*:<jupyter_code># Creating an analyzer
patternnet_analyzer = innvestigate.create_analyzer("pattern.net", model_wo_sm, pattern_type="relu")

# Train (or adapt) the analyzer to the training data
patternnet_analyzer.fit(data[0], verbose=True)

# Applying the analyzer
analysis = patternnet_analyzer.analyze(image)<jupyter_output>Epoch 1/1
1875/1875 [==============================] - 23s 12ms/step - loss: 4.0000 - broadcast_1_loss: 1.0000 - broadcast_2_loss: 1.0000 - broadcast_3_loss: 1.0000 - broadcast_4_loss: 1.0000<jupyter_text>And visualize it:<jupyter_code># Displaying the signal (projected back into input space)
plot.imshow(analysis.squeeze()/np.abs(analysis).max(), cmap="gray", interpolation="nearest")
plot.show()<jupyter_output><empty_output><jupyter_text>Choosing the output neuronIn the previous examples we always analyzed the output of the neuron with the highest activation. In the next one we show how one can choose the neuron to analyze:<jupyter_code># Creating an analyzer and set neuron_selection_mode to "index"
inputXgradient_analyzer = innvestigate.create_analyzer("input_t_gradient", model_wo_sm,
                                                       neuron_selection_mode="index")<jupyter_output><empty_output><jupyter_text>The gradient\*input analyzer is an example from the *attribution* category and we visualize it by means of a colored heatmap to highlight positive and negative attributions:<jupyter_code>for neuron_index in range(10):
    print("Analysis w.r.t. to neuron", neuron_index)
    # Applying the analyzer and pass that we want 
    analysis = inputXgradient_analyzer.analyze(image, neuron_index)
    
    # Displaying the gradient
    plot.imshow(analysis.squeeze(), cmap='seismic', interpolation='nearest')
    plot.show()<jupyter_output>Analysis w.r.t. to neuron 0
StarcoderdataJupyterStructuredCleanDedup