  GPUs are widely used to accelerate many important classes of workloads today.
However, we observe that several important emerging classes of workloads,
including simulation engines for deep reinforcement learning and dynamic neural
networks, are unable to fully utilize the massive parallelism that GPUs offer.
These applications tend to have kernels that are small in size, i.e., have few
thread blocks that do not saturate compute resources. Executing independent
kernels concurrently is a promising approach to improve parallelism and
utilization. However, this inter-kernel concurrency is difficult to leverage in
such workloads with existing approaches: First, the inter-kernel dependencies
and computational graph are input-dependent and vary each time the application
is executed. Second, the computational graphs tend to be irregular, requiring
fine-grain scheduling and synchronization; thus incurring significant
synchronization overheads if kernel execution is parallelized. In this work, we
propose ACS, a framework that enables lightweight detection of inter-kernel
dependencies and low overhead kernel scheduling at runtime. The key idea behind
ACS is to perform inter-kernel dependency checks for a small window of kernels
at runtime, similar to out-of order instruction scheduling. This enables
concurrent execution of kernels in applications whose computational graphs are
input dependent and require fine-grained scheduling. We propose ACS-SW, a
software-only open-source implementation of ACS and ACS-HW, a hardware-software
cooperative implementation. ACS-HW further reduces synchronization overheads by
reducing communication between the CPU and GPU. We evaluate ACS for deep RL
simulation and dynamic DNNs on both real hardware and a GPU simulator. We
demonstrate speedups of up to 2.19x (1.56x on average) by improving GPU
utilization with concurrent kernel execution.
