  Machine learning methods like neural networks are extremely successful and
popular in a variety of applications, however, they come at substantial
computational costs, accompanied by high energy demands. In contrast, hardware
capabilities are limited and there is evidence that technology scaling is
stuttering, therefore, new approaches to meet the performance demands of
increasingly complex model architectures are required. As an unsafe
optimization, noisy computations are more energy efficient, and given a fixed
power budget also more time efficient. However, any kind of unsafe optimization
requires counter measures to ensure functionally correct results.
  This work considers noisy computations in an abstract form, and gears to
understand the implications of such noise on the accuracy of
neural-network-based classifiers as an exemplary workload. We propose a
methodology called "Walking Noise" that allows to assess the robustness of
different layers of deep architectures by means of a so-called "midpoint noise
level" metric. We then investigate the implications of additive and
multiplicative noise for different classification tasks and model
architectures, with and without batch normalization. While noisy training
significantly increases robustness for both noise types, we observe a clear
trend to increase weights and thus increase the signal-to-noise ratio for
additive noise injection. For the multiplicative case, we find that some
networks, with suitably simple tasks, automatically learn an internal binary
representation, hence becoming extremely robust. Overall this work proposes a
method to measure the layer-specific robustness and shares first insights on
how networks learn to compensate injected noise, and thus, contributes to
understand robustness against noisy computations.
