  We explore the impact of entropy change in deep learning systems via noise
injection at different levels, i.e., the latent space and input image. The
series of models that employ our methodology are collectively known as Noisy
Neural Networks (NoisyNN), with examples such as NoisyViT and NoisyCNN. Noise
is conventionally viewed as a harmful perturbation in various deep learning
architectures, such as convolutional neural networks (CNNs) and vision
transformers (ViTs), as well as different learning tasks like image
classification and transfer learning. However, this work shows noise can be an
effective way to change the entropy of the learning system. We demonstrate that
specific noise can boost the performance of various deep architectures under
certain conditions. We theoretically prove the enhancement gained from positive
noise by reducing the task complexity defined by information entropy and
experimentally show the significant performance gain in large image datasets,
such as the ImageNet. Herein, we use the information entropy to define the
complexity of the task. We categorize the noise into two types, positive noise
(PN) and harmful noise (HN), based on whether the noise can help reduce the
complexity of the task. Extensive experiments of CNNs and ViTs have shown
performance improvements by proactively injecting positive noise, where we
achieved an unprecedented top 1 accuracy of over 95$\%$ on ImageNet. Both
theoretical analysis and empirical evidence have confirmed that the presence of
positive noise, can benefit the learning process, while the traditionally
perceived harmful noise indeed impairs deep learning models. The different
roles of noise offer new explanations for deep models on specific tasks and
provide a new paradigm for improving model performance. Moreover, it reminds us
that we can influence the performance of learning systems via information
entropy change.
