- *bg-img-rot*: digits with rotation and image background. Two artificial shape classification problems from the benchmark of Larochelle et al. (2007):
- *rect*: Discriminate between tall and wide rectangles (white on black).

- *rect-img*: Discriminate between tall and wide rectangular image on a different background image.

## 5.1 Classification Performance 5.1.1 Mnist And Cifar-Bw

Our first series of experiments focuses on the MNIST and CIFAR-bw datasets.

We compare the classification performance obtained by a neural network with one hidden layer of 1000 units, initialized with each of the unsupervised algorithms under consideration. For each case, we selected the value of hyperparameters (such as the strength of regularization) that yielded, after supervised fine-tuning, the best classification performance on the validation set. Final classification error rate was then computed on the test set. With the parameters obtained after unsupervised pre-training (before fine-tuning), we also computed in each case the average value of the encoder's contraction kJf (x)kF on the validation set, as well as a measure of the average fraction of saturated units per example7. These results are reported in Table 1. We see that the local contraction measure (the average kJf kF ) on the pre-trained model strongly correlates with the final classification error. The CAE, which explicitly tries to minimize this measure while maintaining a good reconstruction, is the bestperforming model. datasets.

Results given in Table 2 compare the performance of stacked CAEs on the benchmark problems of Larochelle *et al.* (2007) to the three-layer models reported in Vincent *et al.* (2010). Stacking a second layer CAE on top of a first layer appears to significantly improves performance, thus demonstrating their usefulness for building deep networks. Moreover on the majority of datasets, 2-layer CAE beat the state-of-the-art 3-layer model.

## 5.1.2 Cifar-10

The pipeline of preprocessing steps we used here is similar to ?. We randomly extracted 160000 patches 8 × 8 from the 10000 first images of CIFAR-10. For each patch, we substract the mean and divide by the standard deviation ( local contrast normalization). Then, a PCA is fitted on this set of patches. The 2 first components (corresponding to black patches) are dropped but we kept the next 80 first components (over 192). For building the final training set of patches, we project these patches on the PCA components, perform whitening i.e divide by the eigen values, and pass it through a logistic funtion in order to map it to [0, 1].

A Contracting-Auto-Encoder with a number of hidden units nhid ∈ {50, 100, 200, 400}
is trained on this set by minimizing the cross-entropy reconstruction error and the regularizer with stochastic gradient descent. We present some filters learned during this process in Figure XX.

7We consider a unit saturated if its activation is below 0.05 or above 0.95. Note that in general the set of saturated units is expected to vary with each example.