  In the rapidly expanding field of parallel processing, job schedulers are the
"operating systems" of modern big data architectures and supercomputing
systems. Job schedulers allocate computing resources and control the execution
of processes on those resources. Historically, job schedulers were the domain
of supercomputers, and job schedulers were designed to run massive,
long-running computations over days and weeks. More recently, big data
workloads have created a need for a new class of computations consisting of
many short computations taking seconds or minutes that process enormous
quantities of data. For both supercomputers and big data systems, the
efficiency of the job scheduler represents a fundamental limit on the
efficiency of the system. Detailed measurement and modeling of the performance
of schedulers are critical for maximizing the performance of a large-scale
computing system. This paper presents a detailed feature analysis of 15
supercomputing and big data schedulers. For big data workloads, the scheduler
latency is the most important performance characteristic of the scheduler. A
theoretical model of the latency of these schedulers is developed and used to
design experiments targeted at measuring scheduler latency. Detailed
benchmarking of four of the most popular schedulers (Slurm, Son of Grid Engine,
Mesos, and Hadoop YARN) are conducted. The theoretical model is compared with
data and demonstrates that scheduler performance can be characterized by two
key parameters: the marginal latency of the scheduler $t_s$ and a nonlinear
exponent $\alpha_s$. For all four schedulers, the utilization of the computing
system decreases to < 10\% for computations lasting only a few seconds.
Multilevel schedulers that transparently aggregate short computations can
improve utilization for these short computations to > 90\% for all four of the
schedulers that were tested.
