  With the development of deep learning (DL) techniques, rotating machinery
intelligent diagnosis has gone through tremendous progress with verified
success and the classification accuracies of many DL-based intelligent
diagnosis algorithms are tending to 100\%. However, different datasets,
configurations, and hyper-parameters are often recommended to be used in
performance verification for different types of models, and few open source
codes are made public for evaluation and comparisons. Therefore, unfair
comparisons and ineffective improvement may exist in rotating machinery
intelligent diagnosis, which limits the advancement of this field. To address
these issues, we perform an extensive evaluation of four kinds of models,
including multi-layer perception (MLP), auto-encoder (AE), convolutional neural
network (CNN), and recurrent neural network (RNN), with various datasets to
provide a benchmark study within the same framework. We first gather most of
the publicly available datasets and give the complete benchmark study of
DL-based intelligent algorithms under two data split strategies, five input
formats, three normalization methods, and four augmentation methods. Second, we
integrate the whole evaluation codes into a code library and release this code
library to the public for better development of this field. Third, we use
specific-designed cases to point out the existing issues, including class
imbalance, generalization ability, interpretability, few-shot learning, and
model selection. By these works, we release a unified code framework for
comparing and testing models fairly and quickly, emphasize the importance of
open source codes, provide the baseline accuracy (a lower bound) to avoid
useless improvement, and discuss potential future directions in this field. The
code library is available at
https://github.com/ZhaoZhibin/DL-based-Intelligent-Diagnosis-Benchmark.
