Most face relighting methods are able to handle diffuse shadows, but struggle to handle hard shadows, such as those cast by the nose. Methods that propose techniques for handling hard shadows often do not produce geometrically consistent shadows since they do not directly leverage the estimated face geometry while synthesizing them. We propose a novel differentiable algorithm for synthesizing hard shadows based on ray tracing, which we incorporate into training our face relighting model. Our proposed algorithm directly utilizes the estimated face geometry to synthesize geometrically consistent hard shadows. We demonstrate through quantitative and qualitative experiments on Multi-PIE and FFHQ that our method produces more geometrically consistent shadows than previous face relighting methods while also achieving state-of-the-art face relighting performance under directional lighting. In addition, we demonstrate that our differentiable hard shadow modeling improves the quality of the estimated face geometry over diffuse shading models.
Figure 1. We introduce a novel face relighting method that produces geometrically consistent shadows. By proposing a differentiable algorithm based on the principles of ray tracing that directly uses the face geometry for modeling hard shadows, our method produces physically correct hard shadows which the state-of-the-art face relighting method, Hou et al., cannot produce.
Figure 2. Given a single image It and target lighting direction ωt, our model generates a relit image Ip with geometrically consistent cast shadows. The geometric consistency is achieved thanks to our shadow mask estimation module, which estimates shadow mask Mshadow using depth map Dp (the face geometry). Mshadow incorporates non-diffuse cast shadows into our shading Sp.
Figure 3. We generate Mshadow using Dp, ωt, and the principles of ray tracing. For every point xi ∈ Dp , we sample points from Dp along the direction xi -> ωt . If there exists a sampled point whose distance to xi -> ωt is close to 0, then xi -> ωt intersects a surface (e.g. the nose) along its path and xi is under cast shadow. If there is no such point among the sampled points, then xi is not under cast shadow. We show 2 points x1 and x2, marked as green and red asterisks respectively. Among the sampled points for x1 (green points), there exists a point (marked by a yellow arrow) that intersects a surface (the nose) and thus x1 is under a cast shadow. For x2, none of the sampled points (red points) intersect a surface, so x2 is not under a cast shadow.
Figure 4. Across multiple in-the-wild subjects and target lightings, our model produces more geometrically consistent cast shadows than prior methods while achieving noticeably better visual quality. Best viewed if enlarged.
Figure 5. We show the average L1 error map between our relit images and the groundtruth test images of Multi-PIE for each lighting and compare with Hou et al. As shown in b), we have lower error in the shadowed regions, including shadows cast around the nose. Hou et al. has higher errors around the cast shadows, demonstrating that our method produces more geometrically consistent shadows across all subjects.