text: Conventional Convolutional Neural Networks (CNNs) use either a linear or non-linear filter to extract features from an image patch (region) of spatial size @math (Typically, @math is small and is equal to @math , e.g., @math is 5 or 7). Generally, the size of the filter is equal to the size @math of the input patch. We argue that the representation ability of equal-size strategy is not strong enough. To overcome the drawback, we propose to use subpatch filter whose spatial size @math is smaller than @math . The proposed subpatch filter consists of two subsequent filters. The first one is a linear filter of spatial size @math and is aimed at extracting features from spatial domain. The second one is of spatial size @math and is used for strengthening the connection between different input feature channels and for reducing the number of parameters. The subpatch filter convolves with the input patch and the resulting network is called a subpatch network. Taking the output of one subpatch network as input, we further repeat constructing subpatch networks until the output contains only one neuron in spatial domain. These subpatch networks form a new network called Cascaded Subpatch Network (CSNet). The feature layer generated by CSNet is called csconv layer. For the whole input image, we construct a deep neural network by stacking a sequence of csconv layers. Experimental results on four benchmark datasets demonstrate the effectiveness and compactness of the proposed CSNet. For example, our CSNet reaches a test error of @math on the CIFAR10 dataset without model averaging. To the best of our knowledge, this is the best result ever obtained on the CIFAR10 dataset.
  @cite_8: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57 error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28 relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.
summary: We agree that both width and depth of networks are important for the tasks of visual recognition. However, larger capacity does not guarantee higher accuracy. Given a dataset with limited samples, when a network reaches its peak performance, it is difficult to further improve performance by simply adding more feature maps or stacking more convolutional layers. This means that the discriminability of networks does not increase infinitely with the size of networks. The performance comparison with ResNet110 @cite_8 and ResNet1202 @cite_8 on the CIFAR10 dataset supports our viewpoint. ResNet110 is a 110-layer CNN with 1.7M parameters, and ResNet1202 is a 1202-layer CNN with up to 19.4M parameters. However, ResNet110 achieves a test error of $ 6.43