Recent research has shown that controllable image generation based on pre-trained GANs can benefit a wide range of computer vision tasks. However, less attention has been devoted to 3D vision tasks. In light of this, we propose a novel image-conditioned neural implicit field, which can leverage 2D supervisions from GAN-generated multi-view images and perform the single-view reconstruction of generic objects. Firstly, a novel offline StyleGAN-based generator is presented to generate plausible pseudo images with full control over the viewpoint. Then, we propose to utilize a neural implicit function, along with a differentiable renderer to learn 3D geometry from pseudo images with object masks and rough pose initializations. To further detect the unreliable supervisions, we introduce a novel uncertainty module to predict uncertainty maps, which remedy the negative effect of uncertain regions in pseudo images, leading to a better reconstruction performance. The effectiveness of our approach is demonstrated through superior single-view 3D reconstruction results of generic objects.

Introduction

Figure 1. Our approach leverages StyleGAN-generated multi-view pseudo images to learn a 3D model without 3D supervision, which can perform single-view 3D reconstruction for a variety of generic objects, e.g., airplanes, birds, cars, horses, motorbikes, potted plants, etc. In addition, our framework produces uncertainty maps, indicating the unreliable local areas in the pseudo images.

GANSVR Source Code

The source code can be downloaded from here

Publications

  • 2D GANs Meet Unsupervised Single-View 3D Reconstruction
    Feng Liu, Xiaoming Liu
    In Proceeding of European Conference on Computer Vision (ECCV 2022), Tel-Aviv, Israel, Oct. 2022
    Bibtex | PDF | arXiv | Supplemental | Code
  • @inproceedings{ 2d-gans-meet-unsupervised-single-view-3d-reconstruction,
      author = { Feng Liu and Xiaoming Liu },
      title = { 2D GANs Meet Unsupervised Single-View 3D Reconstruction },
      booktitle = { In Proceeding of European Conference on Computer Vision },
      address = { Tel-Aviv, Israel },
      month = { October },
      year = { 2022 },
    }