  Deep neural networks (DNNs) and, in particular, convolutional neural networks
(CNNs) have brought significant advances in a wide range of modern computer
application problems. However, the increasing availability of large amounts of
datasets as well as the increasing available computational power of modern
computers lead to a steady growth in the complexity and size of DNN and CNN
models, and thus, to longer training times. Hence, various methods and attempts
have been developed to accelerate and parallelize the training of complex
network architectures. In this work, a novel CNN-DNN architecture is proposed
that naturally supports a model parallel training strategy and that is loosely
inspired by two-level domain decomposition methods (DDM). First, local CNN
models, that is, subnetworks, are defined that operate on overlapping or
nonoverlapping parts of the input data, for example, sub-images. The
subnetworks can be trained completely in parallel. Each subnetwork outputs a
local decision for the given machine learning problem which is exclusively
based on the respective local input data. Subsequently, an additional DNN model
is trained which evaluates the local decisions of the local subnetworks and
generates a final, global decision. With respect to the analogy to DDM, the DNN
can be interpreted as a coarse problem and hence, the new approach can be
interpreted as a two-level domain decomposition. In this paper, solely image
classification problems using CNNs are considered. Experimental results for
different 2D image classification problems are provided as well as a face
recognition problem, and a classification problem for 3D computer tomography
(CT) scans. The results show that the proposed approach can significantly
accelerate the required training time compared to the global model and,
additionally, can also help to improve the accuracy of the underlying
classification problem.
