Direkt zum Inhalt springen

# Technical University of Munich

#### Informatik IX Chair of Computer Vision & Artificial Intelligence

Boltzmannstrasse 3
85748 Garching info@vision.in.tum.de

CVG Group DVL Group

# Multi-View 3D Reconstruction

For a human, it is usually an easy task to get an idea of the 3D structure shown in an image. Due to the loss of one dimension in the projection process, the estimation of the true 3D geometry is difficult and a so called ill-posed problem, because usually infinitely many different 3D surfaces may produce the same set of images.

3D reconstruction of barley from 25 input images.

## 3D Reconstruction from multiple views

The goal of multiview 3D reconstruction is to infer geometrical structure of a scene captured by a collection of images. Usually the camera position and internal parameters are assumed to be known or they can be estimated from the set of images. By using multiple images, 3D information can be (partially) recovered by solving a pixel-wise correspondence problem. Since automatic correspondence estimation is usually ambiguous and incomplete further knowledge (prior knowledge) about the object is necessary. A typical prior is assume that the object surface is smooth.

Our research is focused on convex variational methods. The 3D reconstruction problem is formulated as an energy minimization problem. Due to the convexity of this energy, any (local) minimizer corresponds to the global minimum of this energy.

Minimizers of this energy are found with iterative numerical optimization methods which evolve the surface gradually from the initial surface to best one with respect to energy functional.
As a further consequence of the convexity, these methods are independent of the initialization. The initial surface can be of any shape, for example a simple box.

Two example reconstructions are shown below along with some of their corresponding input images.

## Spatio-Temporal 3D Reconstruction from multiple videos

Considering a dynamic scene that changes over time, 3D reconstruction can be applied to every time step independently. However, one can achieve temporally more consistent results by using the information from several time frames together, thus computing a spatio-temporal hyper-surface in 4D space.

The following video shows results of our latest publication on 4D reconstruction.

# Related publications

Export as PDF, XML, TEX or BIB

Sort Order:  by type by year
Book Chapters 2011 Book Chapters [] Convex Relaxation Techniques for Segmentation, Stereo and Multiview Reconstruction (D. Cremers, T. Pock, K. Kolev and A. Chambolle), Chapter in Markov Random Fields for Vision and Image Processing, MIT Press, 2011. Book Chapters [] Photometric Depth Super-Resolution (B. Haefner, S. Peng, A. Verma, Y. Quéau and D. Cremers), In IEEE Transactions on Pattern Analysis and Machine Intelligence, volume 42, 2020. [] A Non-invasive 3D Body Scanner and Software Tool towards Analysis of Scoliosis (S. Roy, A.T.D. Gruenwald, A. Alves-Pinto, R. Maier, D. Cremers, D. Pfeiffer and R. Lampe), In BioMed Research International (BMRI), 2019. ([pdf]) [bibtex] [] Fast Joint Estimation of Silhouettes and Dense 3D Geometry from Multiple Images (K. Kolev, T. Brox and D. Cremers), In IEEE Transactions on Pattern Analysis and Machine Intelligence, volume 34, 2012. [] A Variational Approach to Vesicle Membrane Reconstruction from Fluorescence Imaging (K. Kolev, N. Kirchgessner, S. Houben, A. Csiszar, W. Rubner, C. Palm, B. Eiben, R. Merkel and D. Cremers), In Pattern Recognition, volume 44, 2011. [] Multiview Stereo and Silhouette Consistency via Convex Functionals over Convex Domains (D. Cremers and K. Kolev), In IEEE Transactions on Pattern Analysis and Machine Intelligence, volume 33, 2011. [] Continuous Global Optimization in Multiview 3D Reconstruction (K. Kolev, M. Klodt, T. Brox and D. Cremers), In International Journal of Computer Vision, volume 84, 2009. [] Weighted Minimal Hypersurface Reconstruction (B. Goldluecke, I. Ihrke, C. Linz and M. Magnor), In IEEE Transactions on Pattern Analysis and Machine Intelligence, volume 29, 2007. Book Chapters [] Combining Depth Fusion and Photometric Stereo for Fine-Detailed 3D Models (E. Bylow, R. Maier, F. Kahl and C. Olsson), In Scandinavian Conference on Image Analysis (SCIA), 2019.  Oral Presentation, received the SCIA 2019 Honourable Mention award [] Fight ill-posedness with ill-posedness: Single-shot variational depth super-resolution from shading (B. Haefner, Y. Quéau, T. Möllenhoff and D. Cremers), In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.  Spotlight Presentation [] Depth Super-Resolution Meets Uncalibrated Photometric Stereo (S. Peng, B. Haefner, Y. Quéau and D. Cremers), In International Conference on Computer Vision Workshops (ICCVW), 2017.  Oral Presentation at ICCV Workshop on Color and Photometry in Computer Vision [] Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting (R. Maier, K. Kim, D. Cremers, J. Kautz and M. Niessner), In International Conference on Computer Vision (ICCV), 2017. [] Efficient Online Surface Correction for Real-time Large-Scale 3D Reconstruction (R. Maier, R. Schaller and D. Cremers), In British Machine Vision Conference (BMVC), 2017. [] De-noising, Stabilizing and Completing 3D Reconstructions On-the-go using Plane Priors (M. Dzitsiuk, J. Sturm, R. Maier, L. Ma and D. Cremers), In International Conference on Robotics and Automation (ICRA), 2017. ([video]) [] Generalized Connectivity Constraints for Spatio-temporal 3D Reconstruction (M. R. Oswald, J. Stühmer and D. Cremers), In European Conference on Computer Vision (ECCV), 2014. [] Surface Normal Integration for Convex Space-time Multi-view Reconstruction (M. R. Oswald and D. Cremers), In British Machine Vision Conference (BMVC), 2014. [] Spatial and Temporal Interpolation of Multi-View Image Sequences (T. Gurdan, M. R. Oswald, D. Gurdan and D. Cremers), In German Conference on Pattern Recognition (GCPR), volume 36, 2014. [] Volumetric 3D Mapping in Real-Time on a CPU (F. Steinbruecker, J. Sturm and D. Cremers), In International Conference on Robotics and Automation (ICRA), 2014. [] Large-Scale Multi-Resolution Surface Reconstruction from RGB-D Sequences (F. Steinbruecker, C. Kerl, J. Sturm and D. Cremers), In IEEE International Conference on Computer Vision (ICCV), 2013. [] A Convex Relaxation Approach to Space Time Multi-view 3D Reconstruction (M. R. Oswald and D. Cremers), In ICCV Workshop on Dynamic Shape Capture and Analysis (4DMOD), 2013. [] Fast and Accurate Large-scale Stereo Reconstruction using Variational Methods (G. Kuschk and D. Cremers), In ICCV Workshop on Big Data in 3D Computer Vision, 2013. [] Decoupling Photometry and Geometry in Dense Variational Camera Calibration (M. Aubry, K. Kolev, B. Goldluecke and D. Cremers), In IEEE International Conference on Computer Vision (ICCV), 2011. [] Parallel Generalized Thresholding Scheme for Live Dense Geometry from a Handheld Camera (J. Stühmer, S. Gumhold and D. Cremers), In ECCV Workshop on Computer Vision on GPUs (CVGPU), 2010. [] Real-Time Dense Geometry from a Handheld Camera (J. Stühmer, S. Gumhold and D. Cremers), In Pattern Recognition (Proc. DAGM), 2010. [] Anisotropic Minimal Surfaces Integrating Photoconsistency and Normal Information for Multiview Stereo (K. Kolev, T. Pock and D. Cremers), In European Conference on Computer Vision (ECCV), 2010. [] A Superresolution Framework for High-Accuracy Multiview Reconstruction (B. Goldluecke and D. Cremers), In Pattern Recognition (Proc. DAGM), 2009.  Received DAGM Best Paper Award [] Continuous Ratio Optimization via Convex Relaxation with Applications to Multiview 3D Reconstruction (K. Kolev and D. Cremers), In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009. [] An Experimental Comparison of Discrete and Continuous Shape Optimization Methods (M. Klodt, T. Schoenemann, K. Kolev, M. Schikora and D. Cremers), In European Conference on Computer Vision (ECCV), 2008. [] Integration of Multiview Stereo and Silhouettes via Convex Functionals on Convex Domains (K. Kolev and D. Cremers), In European Conference on Computer Vision (ECCV), 2008. [] Continuous Global Optimization in Multiview 3D Reconstruction (K. Kolev, M. Klodt, T. Brox, S. Esedoglu and D. Cremers), In Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR), Springer, volume 4679, 2007. [] Propagated Photoconsistency and Convexity in Variational Multiview 3D Reconstruction (K. Kolev, M. Klodt, T. Brox and D. Cremers), In Workshop on Photometric Analysis for Computer Vision, 2007. [] Robust variational segmentation of 3D objects from multiple views (K. Kolev, T. Brox and D. Cremers), In Pattern Recognition (Proc. DAGM) (K. Fet al., ed.), Springer, volume 4174, 2006. [] Reconstructing the Geometry of Flowing Water (I. Ihrke, B. Goldluecke and M. Magnor), In IEEE International Conference on Computer Vision (ICCV), IEEE, 2005. [] Spacetime-Continous Geometry Meshes from Multi-View Video Sequences (B. Goldluecke and M. Magnor), In Proceedings of the IEEE International Conference on Image Processing, IEEE Computer Society, 2005. [] Space-Time Isosurface Evolution for Temporally Coherent 3D Reconstruction (B. Goldluecke and M. Magnor), In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society, volume I, 2004. [] Weighted Minimal Hypersurfaces and Their Applications in Computer Vision (B. Goldluecke and M. Magnor), In European Conference on Computer Vision (ECCV), Springer, volume 3022, 2004. [] Spacetime-coherent Geometry Reconstruction from Multiple Video Streams (M. Magnor and B. Goldluecke), In 2nd International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT) (Y. Aloimonos, ed.), IEEE, 2004. [] Joint 3D Reconstruction and Background Separation in Multiple Views using Graph Cuts (B. Goldluecke and M. Magnor), In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society, volume I, 2003.