text: Autoencoders are widely used for unsupervised learning and as a regularization scheme in semi-supervised learning. However, theoretical understanding of their generalization properties and of the manner in which they can assist supervised learning has been lacking. We utilize recent advances in the theory of deep learning generalization, together with a novel reconstruction loss, to provide generalization bounds for autoencoders. To the best of our knowledge, this is the first such bound. We further show that, under appropriate assumptions, an autoencoder with good generalization properties can improve any semi-supervised learning scheme. We support our theoretical results with empirical demonstrations.
  @cite_24: This paper presents a margin-based multiclass generalization bound for neural networks that scales with their margin-normalized "spectral complexity": their Lipschitz constant, meaning the product of the spectral norms of the weight matrices, times a certain correction factor. This bound is empirically investigated for a standard AlexNet network trained with SGD on the mnist and cifar10 datasets, with both original and random labels; the bound, the Lipschitz constants, and the excess risks are all in direct correlation, suggesting both that SGD selects predictors whose complexity scales with the difficulty of the learning task, and secondly that the presented bound is sensitive to this complexity.
  @cite_18: 
  @cite_1: We present a generalization bound for feedforward neural networks in terms of the product of the spectral norm of the layers and the Frobenius norm of the weights. The generalization bound is derived using a PAC-Bayes analysis.
  @cite_2: We study the sample complexity of learning neural networks, by providing new bounds on their Rademacher complexity assuming norm constraints on the parameter matrix of each layer. Compared to previous work, these bounds have improved dependence on the network depth, and under some additional assumptions, are fully independent of the network size (both depth and width). These results are derived using some novel techniques, which may be of independent interest.
summary: Within the framework of statistical learning theory, several recent papers have significantly improved previous generalization bounds for deep networks by incorporating more refined attributes of the network structure, aiming to explain the paradoxical effect of improved performance while over-training the network. Using covering number techniques, @cite_24 provide margin based bounds that relate generalization error to the network's Lipschitz constant and matrix norms of the weights. @cite_1 establish similar matrix-norm-based margin bounds using a PAC-Bayes approach. @cite_18 present compression-based results by compressing the weights of a well performing network and bounding the error of the compressed network. Finally, @cite_2 are able, under certain (restrictive) assumptions on matrix norms, to achieve generalization bounds that are completely independent of the network size.