Uncertainty-Aware Deep Multi-View Photometric Stereo

Berk Kaya1
Suryansh Kumar1
Carlos Oliveira1
Vittorio Ferrari2
Luc Van Gool1,3
ETH Zurich1
Google Research2
KU Lueven3

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022
New Orleans - Louisiana

Paper
Supplementary
Code (Coming Soon)
Video


Abstract

This paper presents a simple and effective solution to the longstanding classical multi-view photometric stereo (MVPS) problem. It is well-known that photometric stereo (PS) is excellent at recovering high-frequency surface details, whereas multi-view stereo (MVS) can help remove the low-frequency distortion due to PS and retain the global geometry of the shape. This paper proposes an approach that can effectively utilize such complementary strengths of PS and MVS. Our key idea is to combine them suitably while considering the per-pixel uncertainty of their estimates. To this end, we estimate per-pixel surface normals and depth using an uncertainty-aware deep-PS network and deep-MVS network, respectively. Uncertainty modeling helps select reliable surface normal and depth estimates at each pixel which then act as a true representative of the dense surface geometry. At each pixel, our approach either selects or discards deep-PS and deep-MVS network prediction depending on the prediction uncertainty measure. For dense, detailed, and precise inference of the object's surface profile, we propose to learn the implicit neural shape representation via a multilayer perceptron (MLP). Our approach encourages the MLP to converge to a natural zero-level set surface using the confident prediction from deep-PS and deep-MVS networks, providing superior dense surface reconstruction. Extensive experiments on the DiLiGenT-MV benchmark dataset show that our method provides high-quality shape recovery with a much lower memory footprint while outperforming almost all of the existing approaches.



Multi-View Stereo Reconstruction


Photometric Stereo Reconstruction


Our Approach


Reconstruction Results Comparison



Buddha


Cow


Pot

Reading

Mesh Quality Comparison


We investigated the quality of the recovered meshes for DiLiGenT-MV objects. The visual results show that the distribution of the geometric primitives of SOTA MVPS methods is irregular and uneven. We also transfered the CVPR’22 logo texture on the local region of the mesh recovered using MVPS methods. It can be observed that the texture pattern on our recovered mesh is closer to ground-truth (notice the shift of the text). If the local topology is same, it must place the text at similar location as can be seen in ours. Overall, our method provides surfaces which are superior in quality, regular, hence more useful for geometry processing applications.




Authors



Berk Kaya

Suryansh Kumar

Carlos Oliveira

Vittorio Ferrari

Luc Van Gool

Citation 


@article{kaya2022uncertainty,
	title={Uncertainty-Aware Deep Multi-View Photometric Stereo},
	author={Kaya, Berk and Kumar, Suryansh and Oliveira, Carlos and Ferrari, Vittorio and Van Gool, Luc},
	booktitle={IEEE/CVF CVPR},
	year={2022}
}

Acknowledgements



This work was funded by Focused Research Award from Google(CVL, ETH 2019-HE-318, 2019-HE-323, 2020-FS-351, 2020-HS-411). Suryansh Kumar's project is supported by "ETH Zurich Foundation and Google" for bringing together best academic and industrial research.