  Atomic operations (atomics) such as Compare-and-Swap (CAS) or Fetch-and-Add
(FAA) are ubiquitous in parallel programming. Yet, performance tradeoffs
between these operations and various characteristics of such systems, such as
the structure of caches, are unclear and have not been thoroughly analyzed. In
this paper we establish an evaluation methodology, develop a performance model,
and present a set of detailed benchmarks for latency and bandwidth of different
atomics. We consider various state-of-the-art x86 architectures: Intel Haswell,
Xeon Phi, Ivy Bridge, and AMD Bulldozer. The results unveil surprising
performance relationships between the considered atomics and architectural
properties such as the coherence state of the accessed cache lines. One key
finding is that all the tested atomics have comparable latency and bandwidth
even if they are characterized by different consensus numbers. Another insight
is that the hardware implementation of atomics prevents any instruction-level
parallelism even if there are no dependencies between the issued operations.
Finally, we discuss solutions to the discovered performance issues in the
analyzed architectures. Our analysis enables simpler and more effective
parallel programming and accelerates data processing on various architectures
deployed in both off-the-shelf machines and large compute systems.
