We present an algorithm for "unconstrained" 3D face reconstruction from a 2D photo collection of face images of a subject captured under a diverse variation of poses, expressions, and illuminations, without meta data about the cameras, timing, or light conditions. The output of our algorithm is a true 3D face surface model represented as a watertight triangulated surface with albedo data or texture information. This is certainly a very challending problem, as we do not have access to stereo imaging or video. Motivated by the success of the state-of-the-art method [2], we developed a novel photometric stereo-based method with two distinct novelties. First, working with a true 3D model allows us to enjoy the benefit of using images from all possible poses, including profile, without warping them to a frontal image. Second, by leveraging emerging face alignment techniques and our novel field-based Laplace editing, a combination of landmark constraints and photometric stereo-based normals drives our surface reconstruction.

Overview Face Recon

Figure 1: Given a photo collection of images with unknown pose, illumination, and expression, our goal is to reconstruct a detailed 3D model of the face.

Algorithm Summary

Motivated by the state-of-the-art results and ammendable limitation of [2], this project proposes a novel approach to 3D face reconstruction. Our algorithm is broken down into three major steps which we briefly describe. For full details, please refer to the bibliography on this topic. The first step is enabled by the recent explosion of face alignment techniques, where 2D landmark estimation has been substantially improved. Specifically, given a collection of unconstrained face images, we perform 2D landmark estimation and enhance a 3D template by deforming a generic 3D face template such that the projection of its 3D landmarks are consistent with the estimated 2D landmarks while maintaining the original surface normals. The second step is to estimate the person-specific face normals via photometric stereo. We take 2D face images at all poses and project them onto the enhanced 3D face template to establish a dense correspondence across the images. We then jointly estimate the lighting and surface normals via SVD. The third step is to deform the 3D shape so that its updated surface normals will be similar to the estimated ones, under the landmark constrain and an additional boundary constraint. The process iterates until convergence.

3D Face Reconstruction Binary Code

3D Face Reconstruction binary code may be downloaded from here.

If you use 3D Face Reconstruction code, please cite to the papers:

Publications

  • Adaptive 3D Face Reconstruction from Unconstrained Photo Collections
    Joseph Roth, Yiying Tong, Xiaoming Liu
    IEEE Transactions on Pattern Analysis and Machine Intelligence, Dec. 2016
    Bibtex | PDF | Project Webpage
  • @article{ adaptive-3d-face-reconstruction-from-unconstrained-photo-collections,
      author = { Joseph Roth and Yiying Tong and Xiaoming Liu },
      title = { Adaptive 3D Face Reconstruction from Unconstrained Photo Collections },
      booktitle = { IEEE Transactions on Pattern Analysis and Machine Intelligence },
      volume = { },
      month = { December },
      year = { 2016 },
    }
  • Adaptive 3D Face Reconstruction from Unconstrained Photo Collections
    Joseph Roth, Yiying Tong, Xiaoming Liu
    Proc. IEEE Computer Vision and Pattern Recognition (CVPR 2016), Las Vegas, NV, Jun. 2016
    Bibtex | PDF | Poster | Project Webpage
  • @inproceedings{ adaptive-3d-face-reconstruction-from-unconstrained-photo-collections,
      author = { Joseph Roth and Yiying Tong and Xiaoming Liu },
      title = { Adaptive 3D Face Reconstruction from Unconstrained Photo Collections },
      booktitle = { Proc. IEEE Computer Vision and Pattern Recognition },
      address = { Las Vegas, NV },
      month = { June },
      year = { 2016 },
    }
  • Unconstrained 3D Face Reconstruction
    Joseph Roth, Yiying Tong, Xiaoming Liu
    Proc. IEEE Computer Vision and Pattern Recognition (CVPR 2015), Bostan, MA, Jun. 2015 (Acceptance rate 28%)
    Bibtex | PDF | Poster | Project Webpage
  • @inproceedings{ unconstrained-3d-face-reconstruction,
      author = { Joseph Roth and Yiying Tong and Xiaoming Liu },
      title = { Unconstrained 3D Face Reconstruction },
      booktitle = { Proc. IEEE Computer Vision and Pattern Recognition },
      address = { Bostan, MA },
      month = { June },
      year = { 2015 },
    }