Reconstructing Building Interiors from Images

Reconstructing Building Interiors from ImagesYasutaka Furukawa Brian Curless Steven M. Seitz University of Washington, Seattle, USA

2011/01/16OutlineIntroductionGoalChallengesSystem pipelineAlgorithmic details (technical contribution)Experimental ResultsConclusion and future workReferenceOutlineIntroductionGoalChallengesSystem pipelineAlgorithmic details (technical contribution)Experimental ResultsConclusion and future workReferenceReconstruction and Visualization of Architectural ScenesSemi-automatic(Manual )Google Earth & Virtual EarthFaade : Buildingfacademade for use as a real-time video game engine environment.

Google Earth 4Virtual Earth

Aerial imagesAutomaticGround-level imagesAerial images: A projected image which is "floating in air", and cannot be viewed normally.

Reconstruction and Visualization of Architectural ScenesDifficultyLittle attention paid to indoor scenesIf you walk inside your home and take photographs, generating a compelling 3D reconstruction and visualization becomes much more difficult.

Google Earth 4

Aerial imagesVirtual Earth

????OutlineIntroductionGoalChallengesSystem pipelineAlgorithmic details (technical contribution)Experimental ResultsConclusion and future workReferenceGoalFully automatic system for interiors / outdoorsReconstructs a simple 3D model from imagesProvides real-time interactive visualization

OutlineIntroductionGoalChallengesSystem pipelineAlgorithmic details (technical contribution)Experimental ResultsConclusion and future workReferenceChallengesReconstructionMulti-view stereo (MVS) typically produces a dense modelWe want the model to beSimple for real-time interactive visualization of a large scene (e.g., a whole house)Accurate for high-quality image-based renderingSimple mode is effective for compelling visualizationChallengesIndoor Reconstruction

Texture-poor surfaces

Complicated visibility

Prevalence of thin structures(doors, walls, tables)OutlineIntroductionGoalChallengesSystem pipelineAlgorithmic details (technical contribution)Experimental ResultsConclusion and future workReferenceSystem pipeline3D reconstruction and visualization system for architectural scenes.

System pipelineImage-basedSFMMVSMWSMerging

MVS

Image-basedSFM

MWS

MergingSystem pipeline

Image-based

Image-based14

System pipelineStructure-from-MotionBundler by Noah SnavelyStructure from Motion for unordered image collectionsWEB: http://phototour.cs.washington.edu/bundler/

MVS

Image-basedSFM

MWS

Merging15System pipeline

PMVS by Yasutaka Furukawa and Jean PoncePatch-based Multi-View Stereo Software/Multi-view Stereo

MVS

Image-basedSFM

MWS

Merging16System pipeline

Manhattan-world Stereo

MVS

Image-basedSFM

MWS

Merging17System pipeline

MVS

Image-basedSFM

Manhattan-world Stereo

18System pipeline

Manhattan-world Stereo

Result

MVS

Image-basedSFM

MWS

Merging19System pipeline

Axis-aligned depth map merging(Paper contribution)

MVS

Image-basedSFM

MWS

Merging20OutlineIntroductionGoalChallengesSystem pipelineAlgorithmic details (technical contribution)Experimental ResultsConclusion and future workReferenceAxis-aligned Depth-map MergingBasic framework is similar to volumetric MRF

Hernandez is the name22

Axis-aligned Depth-map MergingBasic framework is similar to volumetric MRF

First explain the algorithm then differences from competing approaches.23Key Feature 1 - Penalty terms

Key Feature 1 - Penalty terms

Binary penalty

Binary encodes smoothness & dataKey Feature 1 - Penalty terms

Binary penalty

Binary encodes smoothness & dataUnary is often constant (inflation)Key Feature 1 - Penalty terms

Binary penalty

Binary encodes smoothness & dataUnary is often constant (inflation)Weak regularization at interesting placesFocus on a dense modelKey Feature 1 - Penalty terms

Binary penalty

Binary encodes smoothness & dataUnary is often constant (inflation)Weak regularization at interesting placesFocus on a dense model

We want a simple modelKey Feature 1 - Penalty terms

Binary penalty

Binary encodes smoothness & dataUnary is often constant (inflation)

Key Feature 1 - Penalty terms

Binary penalty

Binary encodes smoothness & dataUnary is often constant (inflation)Key Feature 1 - Penalty terms

Unary encodes dataBinary penalty

Binary encodes smoothness & dataUnary is often constant (inflation)Key Feature 1 - Penalty terms

Binary is smoothnessUnary encodes dataBinary penalty

Binary encodes smoothness & dataUnary is often constant (inflation)Binary is smoothness defined as neighboring voxels having the same label32Key Feature 1 - Penalty terms

Regularization becomes weakDense 3D model

Regularization is data-independentSimpler 3D modelBinary penalty

Axis-aligned Depth-map MergingAlign-voxel grid withthe dominant axesPut here how typical approaches do, and why they do not work

Talk about 4d neighborhood,

Put texts at the bottom of the figures34Axis-aligned Depth-map MergingAlign-voxel grid withthe dominant axesData term (unary)

Put here how typical approaches do, and why they do not work

Talk about 4d neighborhood,

Put texts at the bottom of the figures35Axis-aligned Depth-map MergingAlign voxel grid withthe dominant axesData term (unary)

Put here how typical approaches do, and why they do not work

Talk about 4d neighborhood,

Put texts at the bottom of the figures36Axis-aligned Depth-map MergingAlign voxel grid withthe dominant axesData term (unary)

Put here how typical approaches do, and why they do not work

Talk about 4d neighborhood,

Put texts at the bottom of the figures37Axis-aligned Depth-map MergingAlign voxel grid withthe dominant axesData term (unary)Smoothness (binary)

Put here how typical approaches do, and why they do not work

Talk about 4d neighborhood,

Put texts at the bottom of the figures38Axis-aligned Depth-map MergingAlign voxel grid withthe dominant axesData term (unary)Smoothness (binary)

Put here how typical approaches do, and why they do not work

Talk about 4d neighborhood,

Put texts at the bottom of the figures39Axis-aligned Depth-map MergingAlign voxel grid withthe dominant axesData term (unary)Smoothness (binary)Graph-cuts

Put here how typical approaches do, and why they do not work

Talk about 4d neighborhood,

Put texts at the bottom of the figures40Key Feature 2 RegularizationFor large scenes, data info are not complete

Typical volumetric MRFs bias to general minimal surface We bias to piece-wise planar axis-aligned for architectural scenes41

Key Feature 2 Regularization

Key Feature 2 Regularization

Key Feature 2 Regularization

Key Feature 2 RegularizationKey Feature 2 Regularization

Key Feature 2 RegularizationSame energy (ambiguous)

Key Feature 2 RegularizationSame energy (ambiguous)Data penalty: 0

Key Feature 2 RegularizationSame energy (ambiguous)Data penalty: 0 Smoothness penalty: Data penalty: 0 Smoothness penalty: 24Data penalty: 0

Key Feature 2 Regularizationshrinkage

Key Feature 2 RegularizationAxis-aligned neighborhood + Potts model

Ambiguous

Break ties with the minimum-volume solution

Piece-wise planar axis-aligned modelKey Feature 3 Sub-voxel accuracy

52Key Feature 3 Sub-voxel accuracy

53Key Feature 3 Sub-voxel accuracy

54Key Feature 3 Sub-voxel accuracy

55Summary of Depth-map MergingFor a simple and axis-aligned modelExplicit regularization in binaryAxis-aligned neighborhood system & minimum-volume solutionFor an accurate modelSub-voxel refinement56OutlineIntroductionGoalChallengesSystem pipelineAlgorithmic details (technical contribution)Experimental ResultsConclusion and future workReferenceExperimental resultsModel complexity control with parameter

Experimental resultsQualitative comparisons with a state-of-the-art MVS approach on hall with the number of faces in parentheses.

Experimental resultsEffects of the sub-voxel refinement procedure.

Experimental resultsEffects of the minimum volume constraint.

Experimental resultsEffects of the grid pruning on running time.

OutlineIntroductionGoalChallengesSystem pipelineAlgorithmic details (technical contribution)Experimental ResultsConclusion and future workReferenceConclusion & Future WorkConclusionFully automated 3D reconstruction/visualization system for architectural scenesNovel depth-map merging to produce piece-wise planar axis-aligned model with sub-voxel accuracyFuture workRelax Manhattan-world assumptionLarger scenes (e.g., a whole building)64OutlineIntroductionGoalChallengesSystem pipelineAlgorithmic details (technical contribution)Experimental ResultsConclusion and future workReferenceReferenceN. Cornelis, B. Leibe, K. Cornelis, and L. V. Gool. 3d urban scene modeling integrating recognition and reconstruction. IJCV, 78(2-3):121141, July 2008.L. Zebedin, J. Bauer, K. Karner, and H. Bischof. Fusion of feature- and area-based information for urban buildings modeling from aerial imagery. In ECCV, 2008.