Despite recent advances in face recognition using deep learning, severe accuracy drops are observed for large pose variations in unconstrained environments. Learning pose-invariant features is one solution, but needs expensively labeled large scale data and carefully designed feature learning algorithms. In this work, we focus on frontalizing faces in the wild under various head poses, including extreme profile views. We propose a novel deep 3D Morphable Model (3DMM) conditioned Face Frontalization Generative Adversarial Network (GAN), termed as FF-GAN, to generate neutral head pose face images. Our framework differs from both traditional GANs and 3DMM based modeling. Incorporating 3DMM into the GAN structure provides shape and appearance priors for fast convergence with less training data, while also supporting end-to-end training. FF-GAN employs not only the discriminator and generator loss but also a new masked symmetry loss to retain visual quality under occlusions, besides an identity loss to recover high frequency information. Experiments on face recognition, landmark localization and 3D reconstruction consistently show the advantage of our frontalization method on faces in the wild datasets.

Overview FF-GAN

Figure 1: The proposed FF-GAN framework. Given a non-frontal face image as input, the generator produces a high-quality frontal face. Learned 3DMM coefficients provide global pose and low frequency information, while the input image injects high frequency local information. A discriminator distinguishes generated faces against real ones, where high-quality frontal faces are considered real. A face recognition engine is used to preserve identity information. The output is a high quality frontal face that retains identity.

Face Frontalization on Multi-PIE

multipie

Figure 2: Face frontalization results on MultiPIE. Each example show 13 pose-variant inputs (odd rows) and the corresponding generated frontal outputs (even rows).

Face Frontalization on AFLW2000

aflw

Figure 3: Face frontalization results on AFLW. FF-GAN achieves very promising visual effects for faces with small, medium, large poses and under various lighting conditionis and expressions.

Face Frontalization on LFW

lfw

Figure 4: Face frontalization comparison results on LFW. (a) Input, (b) LFW-3D [1], (c) LFW-HPEN [2], (d) FF-GAN.

Face Frontalization on IJB-A

ijba

Figure 5: Face frontalization results on IJB-A.

LFW and AFLW2000 Datasets

You can download the LFW from here and AFLW200 from here.

References:

[1] Tal Hassner, Shai Harel, Eran Paz, and Roee Enbar. Effective face frontalization in unconstrained images. In CVPR, 2015

[2] Xiangyu Zhu, Zhen Lei, Junjie Yan, Dong Yi, and Stan Z. Li. High-fidelity pose and expression normalization for face recognition in the wild. In CVPR, 2015.

If you use the datasets, please cite to the papers:

Publications

  • Towards Large-Pose Face Frontalization in the Wild
    Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, Manmohan Chandraker
    In Proceeding of International Conference on Computer Vision (ICCV 2017), Venice, Italy, Oct. 2017
    Bibtex | PDF | arXiv | Supplemental | Poster
  • @inproceedings{ towards-large-pose-face-frontalization-in-the-wild,
      author = { Xi Yin and Xiang Yu and Kihyuk Sohn and Xiaoming Liu and Manmohan Chandraker },
      title = { Towards Large-Pose Face Frontalization in the Wild },
      booktitle = { In Proceeding of International Conference on Computer Vision },
      address = { Venice, Italy },
      month = { October },
      year = { 2017 },
    }