Uncertainty-Aware Deep Multi-View Photometric Stereo

ETH Zurich1
Google Research2
KU Lueven3

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022
New Orleans - Louisiana

Code (Coming Soon)


This paper presents a simple and effective solution to the longstanding classical multi-view photometric stereo (MVPS) problem. It is well-known that photometric stereo (PS) is excellent at recovering high-frequency surface details, whereas multi-view stereo (MVS) can help remove the low-frequency distortion due to PS and retain the global geometry of the shape. This paper proposes an approach that can effectively utilize such complementary strengths of PS and MVS. Our key idea is to combine them suitably while considering the per-pixel uncertainty of their estimates. To this end, we estimate per-pixel surface normals and depth using an uncertainty-aware deep-PS network and deep-MVS network, respectively. Uncertainty modeling helps select reliable surface normal and depth estimates at each pixel which then act as a true representative of the dense surface geometry. At each pixel, our approach either selects or discards deep-PS and deep-MVS network prediction depending on the prediction uncertainty measure. For dense, detailed, and precise inference of the object's surface profile, we propose to learn the implicit neural shape representation via a multilayer perceptron (MLP). Our approach encourages the MLP to converge to a natural zero-level set surface using the confident prediction from deep-PS and deep-MVS networks, providing superior dense surface reconstruction. Extensive experiments on the DiLiGenT-MV benchmark dataset show that our method provides high-quality shape recovery with a much lower memory footprint while outperforming almost all of the existing approaches.

Multi-View Stereo Reconstruction

Photometric Stereo Reconstruction

Our Approach

Reconstruction Results Comparison





Mesh Quality Comparison

We investigated the quality of the recovered meshes for DiLiGenT-MV objects. The visual results show that the distribution of the geometric primitives of SOTA MVPS methods is irregular and uneven. We also transfered the CVPR’22 logo texture on the local region of the mesh recovered using MVPS methods. It can be observed that the texture pattern on our recovered mesh is closer to ground-truth (notice the shift of the text). If the local topology is same, it must place the text at similar location as can be seen in ours. Overall, our method provides surfaces which are superior in quality, regular, hence more useful for geometry processing applications.


Berk Kaya

Suryansh Kumar

Carlos Oliveira

Vittorio Ferrari

Luc Van Gool


	title={Uncertainty-Aware Deep Multi-View Photometric Stereo},
	author={Kaya, Berk and Kumar, Suryansh and Oliveira, Carlos and Ferrari, Vittorio and Van Gool, Luc},
	booktitle={IEEE/CVF CVPR},


This work was funded by Focused Research Award from Google(CVL, ETH 2019-HE-318, 2019-HE-323, 2020-FS-351, 2020-HS-411). Suryansh Kumar's project is supported by "ETH Zurich Foundation and Google" for bringing together best academic and industrial research.