text: We introduce a novel strategy for learning to extract semantically meaningful features from aerial imagery. Instead of manually labeling the aerial imagery, we propose to predict (noisy) semantic features automatically extracted from co-located ground imagery. Our network architecture takes an aerial image as input, extracts features using a convolutional neural network, and then applies an adaptive transformation to map these features into the ground-level perspective. We use an end-to-end learning approach to minimize the difference between the semantic segmentation extracted directly from the ground image and the semantic segmentation predicted solely based on the aerial image. We show that a model learned using this strategy, with no additional training, is already capable of rough semantic labeling of aerial imagery. Furthermore, we demonstrate that by finetuning this model we can achieve more accurate semantic segmentation than two baseline initialization strategies. We use our network to address the task of estimating the geolocation and geoorientation of a ground image. Finally, we show how features extracted from an aerial image can be used to hallucinate a plausible ground-level panorama.
  @cite_5: 
  @cite_25: Convolutional Neural Networks define an exceptionally powerful class of models, but are still limited by the lack of ability to be spatially invariant to the input data in a computationally and parameter efficient manner. In this work we introduce a new learnable module, the Spatial Transformer, which explicitly allows the spatial manipulation of data within the network. This differentiable module can be inserted into existing convolutional architectures, giving neural networks the ability to actively spatially transform feature maps, conditional on the feature map itself, without any extra training supervision or modification to the optimisation process. We show that the use of spatial transformers results in models which learn invariance to translation, scale, rotation and more generic warping, resulting in state-of-the-art performance on several benchmarks, and for a number of classes of transformations.
  @cite_11: We address the problem of novel view synthesis: given an input image, synthesizing new images of the same object or scene observed from arbitrary viewpoints. We approach this as a learning task but, critically, instead of learning to synthesize pixels from scratch, we learn to copy them from the input image. Our approach exploits the observation that the visual appearance of different views of the same instance is highly correlated, and such correlation could be explicitly learned by training a convolutional neural network (CNN) to predict appearance flows -- 2-D coordinate vectors specifying which pixels in the input view could be used to reconstruct the target view. Furthermore, the proposed framework easily generalizes to multiple input views by learning how to optimally combine single-view predictions. We show that for both objects and scenes, our approach is able to synthesize novel views of higher perceptual quality than previous CNN-based techniques.
summary: Many methods have been proposed to represent the relationship between the appearance of two viewpoints. Seitz and Baker @cite_5 model image transformations using a space-variant linear filter, similar to a convolution but varying per-pixel. They highlight that a linear transformation of a vectorized representation of all the pixels in an image is very general; it can represent all standard parametric transformations, such as similarity, affine, perspective, and more. More recently, Jaderberg al @cite_25 describe an end-to-end learnable module for neural networks, the spatial transformer, which allows explicit spatial transformations ( scaling, cropping, rotation, non-rigid deformation) of feature maps within the network that are conditioned on individual data samples. Practically, including a spatial transformer allows a network to select regions of interest from an input and transform them to a canonical pose. Similarly, Tinghui al @cite_11 address the problem of novel view synthesis. They observe that the visual appearance of different views is highly correlated and propose a CNN architecture for estimating appearance flows, a representation of which pixels in the input image can be used for reconstruction.