a second order Taylor approximation of the noisy objective under two necessary assumptions on the noise distribution. In statistics regularizing the norm of the derivatives of the model to be tuned is often referred as roughness penalty (Green, 1993) and is used in the context of cubic splines (De Boor, 1998).

## 6 Higher Order Terms Of The Taylor Expansion

In this section we are interested in the higher order terms of the cost approximation, we find it convenient to use the following formalism:
if T
n L
(*z, ε*) denotes the n-th order derivative of L with respect to z, then:

$$T_{\mathcal{L}}^{n}(z,\varepsilon)=\frac{1}{n!}\sum_{i_{1},\ldots,i_{n}}\varepsilon_{i_{1}},\ldots,\varepsilon_{i_{n}}T_{i_{1},\ldots,i_{n}}^{n}(z)$$
7
where T
nis a tensor of order n and

$$T_{i_{1},...,i_{n}}^{n}(z)=\frac{\partial^{n}{\mathcal{L}}(z)}{\partial z_{i_{1}},...,\partial z_{i_{n}}}$$

using this formalism we can write the fourth order derivative as being:

$$T_{\mathcal{L}}^{4}(z,\varepsilon)=\frac{1}{24}\sum_{i,j,k,l}\varepsilon_{i}\varepsilon_{j}\varepsilon_{k}\varepsilon_{l}T_{i j k l}^{4}(z)$$

Using the two assumptions made on the noise distribution, we know that the third order derivative of the approximation is zero. As for the fourth order derivative, using the second assumption of the noise distribution we know that only the terms that are on the diagonal of the T
4 will be non-zero, we can then write:

$$\left[\frac{}{}\int T_{\mathcal{L}}^{4}(z,\varepsilon){\mathcal{N}}_{0,\sigma^{2}}(\varepsilon)d\varepsilon=\frac{\sigma^{4}}{4!}\sum_{i}T_{i i i i}^{4}(z)\right]$$

Using the above result we can approximate our cost function in the noisy input setting more finely, for this purpose we will use the results obtained above for the Hessian and differentiate them again twice with respect to x.

$$T_{\mathcal{L}}^{4}(x)=\frac{\partial^{2}\left(H_{\mathcal{L}}\left(x\right)\right)}{(\partial x)^{2}}$$
$$=\frac{\partial^{2}}{(\partial x)^{2}}\left[2H_{F}(x)\left(F(x,\theta)-y\right)+2\left\langle J_{F}(x)^{T},J_{F}(x)\right\rangle\right]$$ $$=2\frac{\partial}{\partial x}\left[T_{F}^{3}(x)\left(F(x,\theta)-y\right)+3H_{F}(x)J_{F}(x)\right]$$ $$=2\left[T_{F}^{4}(x)\left(F(x,\theta)-y\right)+4T_{F}^{3}(x)J_{F}(x)+3\left\langle H_{F}(x)^{T},H_{F}(x)\right\rangle\right]$$
