Depth completion involves estimating a dense depth image from sparse depth measurements, often guided by a color image. While linear upsampling is straight forward, it results in artifacts including depth pixels being interpolated in empty space across discontinuities between objects. Current methods use deep networks to upsample and "complete" the missing depth pixels. Nevertheless, depth smearing between objects remains a challenge. We propose a new representation for depth called Depth Coefficients (DC) to address this problem. It enables convolutions to more easily avoid inter-object depth mixing. We also show that the standard Mean Squared Error (MSE) loss function can promote depth mixing, and thus propose instead to use cross-entropy loss for DC. With quantitative and qualitative evaluation on benchmarks, we show that switching out sparse depth input and MSE loss with our DC representation and cross-entropy loss is a simple way to improve depth completion performance, and reduce pixel depth mixing, which leads to improved depth-based object detection.

Overview Depth Coefficients

Figure 1: Overview of DC Construction and Depth Reconstruction from Neural Network. Sparse depth is converted into Depth Coefficients with multiple channels, each channel holding information of a certain depth range. This, along with color, is input to the neural network. The output is a multi-channel dense depth density that is optimized using cross entropy with a ground-truth DC. The final depth is reconstructed based on the predicted density.

Video Demo

Depth Coefficients for Depth Completion Source Code

You can download the Depth Coefficients Source Code from here.

Publications

  • Depth Coefficients for Depth Completion
    Saif Imran, Yunfei Long, Xiaoming Liu, Daniel Morris
    In Proceeding of IEEE Computer Vision and Pattern Recognition (CVPR 2019), Long Beach, CA, Jan. 2019
    Bibtex | PDF | arXiv | Poster
  • @inproceedings{ depth-coefficients-for-depth-completion,
      author = { Saif Imran and Yunfei Long and Xiaoming Liu and Daniel Morris },
      title = { Depth Coefficients for Depth Completion },
      booktitle = { In Proceeding of IEEE Computer Vision and Pattern Recognition },
      address = { Long Beach, CA },
      month = { January },
      year = { 2019 },
    }