  As part of the Exascale Computing Project (ECP), a recent focus of
development efforts for the SUite of Nonlinear and DIfferential/ALgebraic
equation Solvers (SUNDIALS) has been to enable GPU-accelerated time integration
in scientific applications at extreme scales. This effort has resulted in
several new GPU-enabled implementations of core SUNDIALS data structures,
support for programming paradigms which are aware of the heterogeneous
architectures, and the introduction of utilities to provide new points of
flexibility. In this paper, we discuss our considerations, both internal and
external, when designing these new features and present the features
themselves. We also present performance results for several of the features on
the Summit supercomputer and early access hardware for the Frontier
supercomputer, which demonstrate negligible performance overhead resulting from
the additional infrastructure and significant speedups when using both NVIDIA
and AMD GPUs.
