  Accuracies of deep learning (DL) classifiers are often unstable in that they
may change significantly when retested on adversarial images, imperfect images,
or perturbed images. This paper adds to the fundamental body of work on
benchmarking the robustness of DL classifiers on defective images. To measure
robust DL classifiers, previous research reported on single-factor corruption.
We created comprehensive 69 benchmarking image sets, including a clean set,
sets with single factor perturbations, and sets with two-factor perturbation
conditions. The state-of-the-art two-factor perturbation includes (a) two
digital perturbations (salt & pepper noise and Gaussian noise) applied in both
sequences, and (b) one digital perturbation (salt & pepper noise) and a
geometric perturbation (rotation) applied in both sequences. Previous research
evaluating DL classifiers has often used top-1/top-5 accuracy. We innovate a
new two-dimensional, statistical matrix to evaluating robustness of DL
classifiers. Also, we introduce a new visualization tool, including minimum
accuracy, maximum accuracy, mean accuracies, and coefficient of variation (CV),
for benchmarking robustness of DL classifiers. Comparing with single factor
corruption, we first report that using two-factor perturbed images improves
both robustness and accuracy of DL classifiers. All source codes and related
image sets are shared on the Website at http://cslinux.semo.edu/david/data to
support future academic research and industry projects.
