$$D=\int{\mathcal{L}}(x_{i}+\varepsilon){\mathcal{N}}_{0,\sigma^{2}}(\varepsilon)d\varepsilon-{\mathcal{L}}(x_{i})$$
$$(13)$$
7
We can approximate the first term by a Taylor series:

$${\mathcal{L}}(x+\varepsilon)={\mathcal{L}}(x)+\langle J_{\mathcal{L}}(x),\varepsilon\rangle+{\frac{1}{2}}\varepsilon^{T}.H_{\mathcal{L}}(x).\varepsilon+o(\varepsilon)$$

By using this approximation and simplifying our integral, we obtain the following expression for our function:

$$(14)$$
 $$\phi(\theta)\approx\frac{\sigma^2}{2n}\sum_i^n\text{Tr}(H_{\mathcal{L}}(x_i))$$ $$\mathcal{C}^{\text{noise}}(\theta)\approx\mathcal{C}^{\text{clean}}(\theta)+\frac{\sigma^2}{2n}\sum_i^n\text{Tr}(H_{\mathcal{C}}(\theta))$$ autencoder cost function can thus be approximated. 
The de-noising autoencoder cost function can thus be approximated by using a classical auto-encoder with a penality on the trace of the Hessian of the cost fucntion versus the inputs. When the cost function is the MSE of the reconstruction, we can write the Hessian as:

$$H_{L}(x)=\frac{\partial}{\partial x}\left(\frac{\partial}{\partial x}\left(\parallel g(f(x))-x\parallel^{2}\right)\right)$$ $$=2\frac{\partial}{\partial x}\left(\frac{\partial g\left(f(x)\right)}{\partial x}\left(g(f(x))-x\right)\right)$$ $$=2H_{g\circ f}(x)\left(g(f(x))-x\right)+2\left\langle J_{g\circ f}(x)^{T},J_{g\circ f}(x)\right\rangle$$
$$\left(15\right)$$
$$(16)$$
$$\quad(17)$$
$\left(18\right)$ . 
By taking the trace of the above results, we get:

$$\mathrm{Tr}\left(H_{L}(x)\right)=2\left(g(f(x))-x\right)\mathrm{Tr}\left(H_{g\circ f}(x)\right)+2\left\vert\left\vert J_{g\circ f}(x)\right\vert\right\vert_{F}^{2}$$

The first term of the equation scales with the reconstruction cost and will diminish accordingly. The second term of the equation is the Froebenius norm of the Jacobian. Note however that the Jacobian in this case is on the *reconstruction* and not on the *representation* as with our proposed penality.

Why Jf (x) is a better choice than Jg◦f (x) . Adding Gaussian noise during the training of an auto-encoder is thus asymptotically equivalent to adding a regularization term to the objective function. In the DAE setting with MSE cost, the penalty term is the norm of the Jacobian of the reconstruction units with respect to the input units. This encourages the output to be invariant to small changes in the inputs. Note that *the invariance requirement is shared between the two parts of the auto-encoder* and not explicitly on the *representation*.

If the goal of the auto-encoder is to initialise the weights of a Multi-Layer Perceptron (MLP), we do not care about the invariance of the *reconstruction* since we only use the *representation*. We would like the invariances captured by the autoecnoder to be predominantly on the *representation*. By using as a penality the norm of the Jacobian of the *representation*, we are doing this explicitly.

The other drawback of having a regularization over Jg◦f (x) comes from an optimization point of view. The gradient of the total cost function with respect