  As numerous machine learning and other algorithms increase in complexity and
data requirements, distributed computing becomes necessary to satisfy the
growing computational and storage demands, because it enables parallel
execution of smaller tasks that make up a large computing job. However, random
fluctuations in task service times lead to straggling tasks with long execution
times. Redundancy, in the form of task replication and erasure coding, provides
diversity that allows a job to be completed when only a subset of redundant
tasks is executed, thus removing the dependency on the straggling tasks. In
situations of constrained resources (here a fixed number of parallel servers),
increasing redundancy reduces the available resources for parallelism. In this
paper, we characterize the diversity vs. parallelism trade-off and identify the
optimal strategy, among replication, coding and splitting, which minimizes the
expected job completion time. We consider three common service time
distributions and establish three models that describe scaling of these
distributions with the task size. We find that different distributions with
different scaling models operate optimally at different levels of redundancy,
and thus may require very different code rates.
