  Attention mechanisms, primarily designed to capture pairwise correlations
between words, have become the backbone of machine learning, expanding beyond
natural language processing into other domains. This growth in adaptation comes
at the cost of prohibitively large memory requirements and computational
complexity, especially at higher number of input elements. This limitation is
due to inherently limited data reuse opportunities and quadratic growth in
memory footprints, leading to severe memory-boundedness and limited scalability
of input elements. This work addresses these challenges by devising a tailored
dataflow optimization, called FLAT, for attention mechanisms without altering
their functionality. This dataflow processes costly attention operations
through a unique fusion mechanism, transforming the memory footprint quadratic
growth to merely a linear one. To realize the full potential of this bespoke
mechanism, we propose a tiling approach to enhance the data reuse across
attention operations. Our method both mitigates the off-chip bandwidth
bottleneck as well as reduces the on-chip memory requirement. FLAT delivers
1.94x (1.76x) speedup and 49% and (42%) of energy savings compared to the
state-of-the-art Edge (Cloud) accelerators with no customized dataflow
optimization. When on-chip resources are scarce (20 KB-200 KB), FLAT yields, on
average, 1.5x end-to-end latency reduction across a diverse range of
conventional attention-based models with input sequence lengths ranging from
512-token to 64K-token. Our evaluations demonstrate that state-of-the-art DNN
dataflow applied to attention operations reach the efficiency limit for inputs
above 512 elements. In contrast, FLAT unblocks transformer models for inputs
with up to 64K elements
