  Tensor decomposition (TD) is an important method for extracting latent
information from high-dimensional (multi-modal) sparse data. This study
presents a novel framework for accelerating fundamental TD operations on
massively parallel GPU architectures. In contrast to prior work, the proposed
Blocked Linearized Coordinate (BLCO) format enables efficient out-of-memory
computation of tensor algorithms using a unified implementation that works on a
single tensor copy. Our adaptive blocking and linearization strategies not only
meet the resource constraints of GPU devices, but also accelerate data
indexing, eliminate control-flow and memory-access irregularities, and reduce
kernel launching overhead. To address the substantial synchronization cost on
GPUs, we introduce an opportunistic conflict resolution algorithm, in which
threads collaborate instead of contending on memory access to discover and
resolve their conflicting updates on-the-fly, without keeping any auxiliary
information or storing non-zero elements in specific mode orientations. As a
result, our framework delivers superior in-memory performance compared to prior
state-of-the-art, and is the only framework capable of processing out-of-memory
tensors. On the latest Intel and NVIDIA GPUs, BLCO achieves 2.12-2.6X
geometric-mean speedup (with up to 33.35X speedup) over the state-of-the-art
mixed-mode compressed sparse fiber (MM-CSF) on a range of real-world sparse
tensors.
