text: Recovering the latent photorealistic faces from their artistic portraits aids human perception and facial analysis. However, recovering photorealistic faces from stylized portraits while preserving identity is challenging because the fine details of real faces can be distorted or lost in stylized images. In this paper, we present a new Identity-preserving Face Recovery from Portraits (IFRP) method to recover latent photorealistic faces from unaligned stylized portraits. Our IFRP method consists of two components: Style Removal Network (SRN) and Discriminative Network (DN). The SRN is designed to transfer feature maps of stylized images to the feature maps of the corresponding photorealistic faces. By embedding spatial transformer networks into the SRN, our method can compensate for misalignments of stylized faces automatically and output aligned realistic face images. The DN is used to enforce recovered face images to be similar to authentic faces. To ensure the identity preservation, we promote the recovered and ground-truth faces to share similar visual features via a distance measure which compares features of recovered and ground-truth faces extracted from a pre-trained VGG network. Our approach is evaluated on a large-scale synthesized dataset of real and stylized face pairs and outperforms the state-of-the-art methods. In addition, we demonstrate that our method can also recover photorealistic faces from unseen stylized portraits (unavailable in training) as well as original paintings.
  @cite_37: Face sketches are being used in eyewitness testimonies for about a century. These sketches are crucial in finding suspects when no photo is available, but a mental image in the eyewitness's mind. However, research shows that current procedures used for eyewitness testimonies have two main problems. First, they can significantly disturb the memories of the eyewitness. Second, in many cases, these procedures result in face images far from their target faces. These two problems are related to the plasticity of the human visual system and the differences between face perception in humans (holistic) and current methods of sketch production (piecemeal). In this paper, we present some insights for more realistic sketch to photo matching. We describe how to retrieve identity specific information from crude sketches, directly drawn by the non-artistic eyewitnesses. The sketches we used merely contain facial component outlines and facial marks (e.g. wrinkles and moles). We compare results of automatically matching two types sketches (trace-over and user-provided, 25 each) to four types of faces (original, locally exaggerated, configurally exaggerated, and globally exaggerated, 249 each), using two methods (PDM distance comparison and PCA classification). Based on our results, we argue that for automatic non-artistic sketch to photo matching, the algorithms should compare the user-provided sketches with globally exaggerated faces, with a soft constraint on facial marks, to achieve the best matching rates. This is because the user-provided sketch from the user's mental image, seems to be caricatured both locally and configurally.
  @cite_32: This paper presents a novel way to perform multi-modal face recognition. We use Partial Least Squares (PLS) to linearly map images in different modalities to a common linear subspace in which they are highly correlated. PLS has been previously used effectively for feature selection in face recognition. We show both theoretically and experimentally that PLS can be used effectively across modalities. We also formulate a generic intermediate subspace comparison framework for multi-modal recognition. Surprisingly, we achieve high performance using only pixel intensities as features. We experimentally demonstrate the highest published recognition rates on the pose variations in the PIE data set, and also show that PLS can be used to compare sketches to photos, and to compare images taken at different resolutions.
  @cite_25: This paper reports a human face image searching system using sketches. A two-phase method, namely, sketch-to-mug-shot matching and human face image searching using relevance feedback, is designed and developed. In the sketch-to-mug-shot matching phase, we have developed a facial feature matching algorithm using local and global features. A point distribution model is employed to represent local facial features while the global feature consists of a set of the geometrical relationship between facial features. It is found that the performance of the sketch-to-mug-shot matching is good if the sketch image looks like the mug shot image in the database. However, in some situations, it is hard to construct a sketch that looks like the photograph. To overcome this limitation, this paper makes use of the concept of ldquohuman-in-the-looprdquo and proposes a human face image searching algorithm using relevance feedback in the second phase. Positive and negative samples will be collected from the user. A feedback algorithm that employs subspace linear discriminant analysis for online learning of the optimal projection for face representation is then designed and developed. The proposed system has been evaluated using the FERET database and a Japanese database with hundreds of individuals. The results are encouraging.
  @cite_15: We propose a novel face photo retrieval system using sketch drawings. By transforming a photo image into a sketch, we reduce the difference between photo and sketch significantly, thus allow effective matching between the two. To improve the synthesis performance, we separate shape and texture information in a face photo, and conduct transformation on them respectively. Finally a Bayesian classifier is used to recognize the probing sketch from the synthesized pseudo-sketches. Experiments on a data set containing 606 people clearly demonstrate the efficacy of the algorithm.
summary: Moreover, there exist several methods to synthesize sketches from photographs (and vice versa) @cite_37 @cite_25 @cite_15 @cite_32 . While sketch-to-face synthesis is a related problem, our unified framework can work with various more complex styles.