  The field of deep learning has witnessed significant progress, particularly
in computer vision (CV), natural language processing (NLP), and speech. The use
of large-scale models trained on vast amounts of data holds immense promise for
practical applications, enhancing industrial productivity and facilitating
social development. With the increasing demands on computational capacity,
though numerous studies have explored the efficient training, a comprehensive
summarization on acceleration techniques of training deep learning models is
still much anticipated. In this survey, we present a detailed review for
training acceleration. We consider the fundamental update formulation and split
its basic components into five main perspectives: (1) data-centric: including
dataset regularization, data sampling, and data-centric curriculum learning
techniques, which can significantly reduce the computational complexity of the
data samples; (2) model-centric, including acceleration of basic modules,
compression training, model initialization and model-centric curriculum
learning techniques, which focus on accelerating the training via reducing the
calculations on parameters; (3) optimization-centric, including the selection
of learning rate, the employment of large batchsize, the designs of efficient
objectives, and model average techniques, which pay attention to the training
policy and improving the generality for the large-scale models; (4) budgeted
training, including some distinctive acceleration methods on source-constrained
situations; (5) system-centric, including some efficient open-source
distributed libraries/systems which provide adequate hardware support for the
implementation of acceleration algorithms. By presenting this comprehensive
taxonomy, our survey presents a comprehensive review to understand the general
mechanisms within each component and their joint interaction.
