MNIST, the only difference is that the intensity of the pixels superior to 255 2 where set to 1 and the others to 0. USPS dataset consists of a training set with 7291 images and a test set with 2007 images, the validation set was formed using the last 2000 images of the training set. The model F(x), with parameters θ = {W(1), . . . , W(n), b(1)*, . . . , b*(n)},
we considered to solve the classification task was a standard neural network with one or more hidden layers, and a hyperbolic tangent non-linearity in between the layers. For example, with one hidden layer we have:
F(x) = σ(W(2) tanh(W(1).x + b
(1)) + b
(2))
where σ(.) is the logistic sigmoid function. In this setting we can write the Frobenius norm of the Jacobian of F with respect to x as being:

$$||J_{F}(x)||^{2}=\sum_{i,j}J_{i j}^{2}$$

and with some calculus we get:

$$J_{i j}=F_{i}(x)(1-F_{i}(x))\sum_{l}W_{i l}^{(2)}W_{l j}^{(1)}\left(1-\operatorname{tanh}^{2}\left(\sum_{m}W_{l m}^{(1)}x_{m}+b_{l}^{(1)}\right)\right).$$

where Fi(x) is i-th output of the network. For the results in table (1), we used a number of hidden units ranging from 400 to 1000 per layer, the best results were obtained with two hidden layers on MNIST, and one hidden layer on MNIST-BINARY and USPS. The parameters of the model were learned through stochastic gradient descent with a learning rate ranging from 0.1 to 0.001. We also investigated the use of *Rectifying* units (i.e. max(0, x)) (Nair and Hinton, 2010; Glorot *et al.*, 2010) as non-linearity in the hidden layer, surprisingly they seemed to benefit less from the added noise to the input than from the regularization term alone, they achieved a test classification performance of 4.8% on the USPS dataset equaling the best performance of the hyperbolic tangent activation with both regularization and added noise to the input. The best results where obtained with a gaussian isotropic noise with a standard deviation of 0.1 around training samples. Figure 1 shows the histograms of activation values on the MNIST test set of our best MLPs with and without Jacobian regularization. The activations of the regularized model are more densely distributed at the saturation and linear regime.

## 9 Discussion 9.1 Constraining The Solution Space

When optimizing a non-convex function with an infinite amount of local minimum, it is not clear which of them yields a reasonable generalization performance, the concept of overfitting clearly illustrate this point. The proposed regularizer tries to avoid this scheme by flattening the mapping function over the training examples inducing a local