  Training deep learning models in the cloud or on dedicated hardware is
expensive. A more cost-efficient option are hyperscale clouds offering spot
instances, a cheap but ephemeral alternative to on-demand resources. As spot
instance availability can change depending on the time of day, continent, and
cloud provider, it could be more cost-efficient to distribute resources over
the world. Still, it has not been investigated whether geo-distributed,
data-parallel spot deep learning training could be a more cost-efficient
alternative to centralized training.
  This paper aims to answer the question: Can deep learning models be
cost-efficiently trained on a global market of spot VMs spanning different data
centers and cloud providers? To provide guidance, we extensively evaluate the
cost and throughput implications of training in different zones, continents,
and clouds for representative CV, NLP and ASR models. To expand the current
training options further, we compare the scalability potential for hybrid-cloud
scenarios by adding cloud resources to on-premise hardware to improve training
throughput. Finally, we show how leveraging spot instance pricing enables a new
cost-efficient way to train models with multiple cheap VMs, trumping both more
centralized and powerful hardware and even on-demand cloud offerings at
competitive prices.
