Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: On the Regularization of Autoencoders   While much work has been devoted to understanding the implicit (and explicit)
regularization of deep nonlinear networks in the supervised setting, this paper
focuses on unsupervised learning, i.e., autoencoders are trained with the
objective of reproducing the output from the input. We extend recent results
[Jin et al. 2021] on unconstrained linear models and apply them to (1)
nonlinear autoencoders and (2) constrained linear autoencoders, obtaining the
following two results: first, we show that the unsupervised setting by itself
induces strong additional regularization, i.e., a severe reduction in the
model-capacity of the learned autoencoder: we derive that a deep nonlinear
autoencoder cannot fit the training data more accurately than a linear
autoencoder does if both models have the same dimensionality in their last
hidden layer (and under a few additional assumptions). Our second contribution
is concerned with the low-rank EDLAE model [Steck 2020], which is a linear
autoencoder with a constraint on the diagonal of the learned low-rank
parameter-matrix for improved generalization: we derive a closed-form
approximation to the optimum of its non-convex training-objective, and
empirically demonstrate that it is an accurate approximation across all
model-ranks in our experiments on three well-known data sets.

27
The categories of this paper are: comp-gas
22
The categories of this paper are: chao-dyn
36
The categories of this paper are: cmp-lg
18
The categories of this paper are: econ
21
The categories of this paper are: adap-org
20
The categories of this paper are: acc-phys
26
The categories of this paper are: bayes-an
25
The categories of this paper are: patt-sol
16
The categories of this paper are: stat
5
The categories of this paper are: gr-qc
3
The categories of this paper are: physics
15
The categories of this paper are: nucl-ex
29
The categories of this paper are: funct-an
33
The categories of this paper are: plasm-ph
14
The categories of this paper are: math-ph
13
The categories of this paper are: quant-ph