to a non-linear feature extractor of the exact same form: a linear mapping followed by a sigmoid non-linearity2. From this perspective, these algorithms are but different unsupervised techniques to learn the parameters of a mapping of that form. It is not yet fully understood what properties of such a mappings contribute to superior classification performance (for classifiers initialized with the produced features). It has been argued that mappings that produce a sparse representation are to be encouraged, which inspired several variants of sparse auto-encoders.

The research we present here is motivated by a different property: our working hypothesis is that a good representation of a likely input (under the unknown data distribution) should be expected to remain rather stable (i.e. be robust, invariant, insensitive) under tiny perturbations of that input. This prompts us to propose an alternative regularization term for auto-encoders.

To encourage robustness of the representation f(x) obtained for a training input x we propose to penalize its *sensitivity* to that input, measured as the Frobenius norm of the Jacobian Jf (x) of the non-linear mapping. Formally, if input x ∈ IRdxis mapped by encoding function f to hidden representation h ∈ IRdh, this sensitivity penalization term is the sum of squares of all partial derivatives of the extracted features with respect to input dimensions:

$$\|J_{f}(x)\|_{F}^{2}=\sum_{i j}\left({\frac{\partial h_{j}(x)}{\partial x_{i}}}\right)^{2}\,.$$
2. (1)
Penalizing kJf k 2 F
encourages the mapping to the feature space to be *contractive* in the neighborhood of the training data. This geometric perspective, which gives its name to our algorithm, will be further elaborated on, in section 5.3, based on experimental evidence. The *flatness* induced by having low valued first derivatives will imply an invariance or *robustness* of the representation for small variations of the input. Thus in this study, terms like invariance, (in-)sensitivity, robustness, flatness and contraction all point to the same notion.

While such a Jacobian term alone would encourage mapping to a useless constant representation, it is counterbalanced in auto-encoder training3 by the need for the learnt representation to allow a good reconstruction of the input4.

## 3 Auto-Encoders Variants

In its simplest form, an auto-encoder (AE) is composed of two parts, an encoder and a decoder. It was introduced in the late 80's Rumelhart *et al.* (1986); Baldi and Hornik (1989) as a technique for dimensionality reduction, where the output of the encoder represents the reduced representation and where the decoder is tuned to reconstruct the initial input from the encoder's representation through the minimization of a cost function. More specifically when the encoding activation functions are linear and the number of hidden units is inferior to the input 3