# Adding Noise To The Input Of A Model Trained With A Regularized Objective

Salah Rifai, Xavier Glorot, Yoshua Bengio and Pascal Vincent October 22, 2018 Dept. IRO, Universit´e de Montr´eal. Montr´eal (QC), H3C 3J7, Canada Technical report 1359

## 1 Abstract

Regularization is a well studied problem in the context of neural networks. It is usually used to improve the generalization performance when the number of input samples is relatively small or heavily contaminated with noise. The regularization of a parametric model can be achieved in different manners some of which are early stopping (Morgan and Bourlard, 1990), weight decay, output smoothing that are used to avoid overfitting during the training of the considered model. From a Bayesian point of view, many regularization techniques correspond to imposing certain prior distributions on model parameters (Krogh and Hertz, 1991). Using Bishop's approximation (Bishop, 1995) of the objective function when a restricted type of noise is added to the input of a parametric function, we derive the higher order terms of the Taylor expansion and analyze the coefficients of the regularization terms induced by the noisy input. In particular we study the effect of penalizing the Hessian of the mapping function with respect to the input in terms of generalization performance. We also show how we can control independently this coefficient by explicitly penalizing the Jacobian of the mapping function on corrupted inputs.

## 1 Introduction

Regularization is a well studied problem in the context of neural networks. It is usually used to improve the generalization performance when the number of input samples is relatively small or heavily contaminated with noise. The regularization of a parametric model can be achieved in different manners some of which are early stopping (Morgan and Bourlard, 1990), weight decay or output smoothing, and are used to avoid overfitting during the training of the considered model. From a Bayesian point of view, many regularization techniques correspond to imposing certain prior distributions on model parameters (Krogh and Hertz, 1991). In this paper we propose a novel approach to achieve regularization that combines noise in the input and explicit output smoothing by regularizing the L2-norm of the Jacobian's mapping function with respect to the input. Bishop (1995) has proved that the arXiv:1104.3250v1 [cs.AI] 16 Apr 2011