text: Pixel-level semantic segmentation is a challenging task with a huge amount of computation, especially if the input sizes are large. In the segmentation network, apart from the pyramid backbone network, an extra decoder network is often employed to recover the spatial detail information. In this paper, we put forward a method for single-shot segmentation in a feature residual pyramid network (RPNet), which learns the coarse results and residuals of segmentations by decomposing the label at different levels of residual blocks. Specifically speaking, we use the residual features to learn the edges and details, and we also use the top-level feature to learn the coarse segmentation result. At the testing phase, the predicted residuals are used to enhance the details of the coarse segmentation result. Residual learning blocks split the network into several shallow sub-networks by level-wise training, which facilitates the gradient propagation in the RPNet. We then evaluate the proposed method and compare it with the recent state-of-the-art methods on CamVid and Cityscapes datasets. The proposed single-shot segmentation based on the RPNet achieves impressive results with high efficiency on the pixel-level segmentation task.
  @cite_14: In this work, we revisit atrous convolution, a powerful tool to explicitly adjust filter's field-of-view as well as control the resolution of feature responses computed by Deep Convolutional Neural Networks, in the application of semantic image segmentation. To handle the problem of segmenting objects at multiple scales, we design modules which employ atrous convolution in cascade or in parallel to capture multi-scale context by adopting multiple atrous rates. Furthermore, we propose to augment our previously proposed Atrous Spatial Pyramid Pooling module, which probes convolutional features at multiple scales, with image-level features encoding global context and further boost performance. We also elaborate on implementation details and share our experience on training our system. The proposed DeepLabv3' system significantly improves over our previous DeepLab versions without DenseCRF post-processing and attains comparable performance with other state-of-art models on the PASCAL VOC 2012 semantic image segmentation benchmark.
  @cite_26: 
  @cite_8: Deep convolutional neural networks achieve better than human level visual recognition accuracy, at the cost of high computational complexity. We propose to factorize the convolutional layers to improve their efficiency. In traditional convolutional layers, the 3D convolution can be considered as performing in-channel spatial convolution and linear channel projection simultaneously, leading to highly redundant computation. By unravelling them apart, the proposed layer only involves single in-channel convolution and linear channel projection. When stacking such layers together, we achieves similar accuracy with significantly less computation. Additionally, we propose a topological connection framework between the input channels and output channels that further improves the layer's efficiency. Our experiments demonstrate that the proposed method remarkably outperforms the standard convolutional layer with regard to accuracy complexity ratio. Our model achieves accuracy of GoogLeNet while consuming 3.4 times less computation.
  @cite_24: In this work we address the task of semantic image segmentation with Deep Learning and make three main contributions that are experimentally shown to have substantial practical merit. First , we highlight convolution with upsampled filters, or ‘atrous convolution’, as a powerful tool in dense prediction tasks. Atrous convolution allows us to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks. It also allows us to effectively enlarge the field of view of filters to incorporate larger context without increasing the number of parameters or the amount of computation. Second , we propose atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales. ASPP probes an incoming convolutional feature layer with filters at multiple sampling rates and effective fields-of-views, thus capturing objects as well as image context at multiple scales. Third , we improve the localization of object boundaries by combining methods from DCNNs and probabilistic graphical models. The commonly deployed combination of max-pooling and downsampling in DCNNs achieves invariance but has a toll on localization accuracy. We overcome this by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF), which is shown both qualitatively and quantitatively to improve localization performance. Our proposed “DeepLab” system sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 79.7 percent mIOU in the test set, and advances the results on three other datasets: PASCAL-Context, PASCAL-Person-Part, and Cityscapes. All of our code is made publicly available online.
  @cite_19: Semantic segmentation requires both rich spatial information and sizeable receptive field. However, modern approaches usually compromise spatial resolution to achieve real-time inference speed, which leads to poor performance. In this paper, we address this dilemma with a novel Bilateral Segmentation Network (BiSeNet). We first design a Spatial Path with a small stride to preserve the spatial information and generate high-resolution features. Meanwhile, a Context Path with a fast downsampling strategy is employed to obtain sufficient receptive field. On top of the two paths, we introduce a new Feature Fusion Module to combine features efficiently. The proposed architecture makes a right balance between the speed and segmentation performance on Cityscapes, CamVid, and COCO-Stuff datasets. Specifically, for a 2048 ( ) 1024 input, we achieve 68.4 Mean IOU on the Cityscapes test dataset with speed of 105 FPS on one NVIDIA Titan XP card, which is significantly faster than the existing methods with comparable performance.
  @cite_10: We introduce a fast and efficient convolutional neural network, ESPNet, for semantic segmentation of high resolution images under resource constraints. ESPNet is based on a new convolutional module, efficient spatial pyramid (ESP), which is efficient in terms of computation, memory, and power. ESPNet is 22 times faster (on a standard GPU) and 180 times smaller than the state-of-the-art semantic segmentation network PSPNet, while its category-wise accuracy is only 8 less. We evaluated ESPNet on a variety of semantic segmentation datasets including Cityscapes, PASCAL VOC, and a breast biopsy whole slide image dataset. Under the same constraints on memory and computation, ESPNet outperforms all the current efficient CNN networks such as MobileNet, ShuffleNet, and ENet on both standard metrics and our newly introduced performance metrics that measure efficiency on edge devices. Our network can process high resolution images at a rate of 112 and 9 frames per second on a standard GPU and edge device, respectively. Our code is open-source and available at https: sacmehta.github.io ESPNet .
summary: Another way is to optimize convolution blocks. ERFNet @cite_26 uses residual connections and factorized convolutions @cite_8 to maintain efficiency and accuracy. DeepLab v3+ @cite_24 proposes Atrous Separable Convolution to speed up standard convolution. ESPNet @cite_10 decomposes a standard convolution into a point-wise convolution and a spatial pyramid of dilated convolutions to improve efficiency. BiSeNet @cite_14 @cite_19 uses shallow Spatial Path and Context Paths to generate high-resolution feature and sufficient receptive field, and then combines the two paths to predict the target. This two-path structure can transfer the computation of depth to two sub-networks and improve the parallelism of sub-networks.