  The performance of highly parallel applications on distributed-memory systems
is influenced by many factors. Analytic performance modeling techniques aim to
provide insight into performance limitations and are often the starting point
of optimization efforts. However, coupling analytic models across the system
hierarchy (socket, node, network) fails to encompass the intricate interplay
between the program code and the hardware, especially when execution and
communication bottlenecks are involved. In this paper we investigate the effect
of "bottleneck evasion" and how it can lead to automatic overlap of
communication overhead with computation. Bottleneck evasion leads to a gradual
loss of the initial bulk-synchronous behavior of a parallel code so that its
processes become desynchronized. This occurs most prominently in memory-bound
programs, which is why we choose memory-bound benchmark and application codes,
specifically an MPI-augmented STREAM Triad, sparse matrix-vector
multiplication, and a collective-avoiding Chebyshev filter diagonalization code
to demonstrate the consequences of desynchronization on two different
supercomputing platforms. We investigate the role of idle waves as possible
triggers for desynchronization and show the impact of automatic asynchronous
communication for a spectrum of code properties and parameters, such as
saturation point, matrix structures, domain decomposition, and communication
concurrency. Our findings reveal how eliminating synchronization points (such
as collective communication or barriers) precipitates performance improvements
that go beyond what can be expected by simply subtracting the overhead of the
collective from the overall runtime.
