this viewpoint, several techniques have been developed to encourage sparsity of the representation (Kavukcuoglu *et al.*, 2008; Lee *et al.*, 2008).

Denoising Auto-encoders (DAE). A successful alternative form of regularization is obtained through the technique of denoising auto-encoders (DAE)
put forward by Vincent *et al.* (2008, 2010), where one simply corrupts input x before sending it through the auto-encoder, that is trained to reconstruct the clean version (i.e. to denoise). This yields the following objective function:

$${\mathcal{I}}_{\mathrm{DAE}}(\theta)=\sum_{x\in D_{n}}\mathbb{E}_{{\bar{x}}\sim q({\bar{x}}|x)}[L(x,g(f({\bar{x}})))],$$
$$(6)$$

where the expectation is over corrupted versions ˜x of examples x obtained from a corruption process q(˜x|x). This objective is optimized by stochastic gradient descent (sampling corrupted examples).

Typically, we consider corruptions such as additive isotropic Gaussian noise:
x˜ = x + ,  ∼ N (0, σ2I) and a binary masking noise, where a fraction ν of input components (randomly chosen) have their value set to 0. The degree of the corruption (σ or ν) controls the degree of regularization.

## 4 Contracting Auto-Encoders (Cae)

From the motivation of robustness to small perturbations around the training points, as discussed in section 2, we propose an alternative regularization that favors mappings that are more strongly contracting at the training samples (see section 5.3 for a longer discussion). The Contracting auto-encoder (CAE) is obtained with the regularization term of eq. 1 yielding objective function

$${\mathcal{I}}_{\mathrm{CAE}}(\theta)=\sum_{x\in D_{n}}\left(L(x,g(f(x)))+\lambda\|J_{f}(x)\|_{F}^{2}\right)$$
$$(7)$$
(7)
Relationship with weight decay. It is easy to see that the squared Frobenius norm of the Jacobian corresponds to a L2 weight decay in the case of a *linear encoder* (i.e. when sf is the identity function). In this special case JCAE and JAE+wd are identical. Note that in the linear case, keeping weights small is the only way to have a contraction. But with a sigmoid non-linearity, contraction and robustness can also be achieved by driving the hidden units to their saturated regime.

Relationship with sparse auto-encoders. Auto-encoder variants that encourage sparse representations aim at having, for each example, a majority of the components of the representation close to zero. For these features to be close to zero, they must have been computed in the left saturated part of the sigmoid nonlinearity, which is almost flat, with a tiny first derivative. This yields a corresponding small entry in the Jacobian Jf (x). Thus, sparse autoencoders that output many close-to-zero features, are likely to correspond to a highly contractive mapping, even though contraction or robustness are not explicitly encouraged through their learning criterion.

Relationship with denoising auto-encoders. Robustness to input perturbations was also one of the motivation of the denoising auto-encoder, as stated in Vincent *et al.* (2010). The CAE and the DAE differ however in the following ways: