Upload
erica-reynolds
View
214
Download
0
Embed Size (px)
Citation preview
Extracting Objects from Extracting Objects from Range and Radiance ImagesRange and Radiance Images
Extracting Objects from Extracting Objects from Range and Radiance ImagesRange and Radiance Images
Computer Science Division
University of California at Berkeley
Computer Science Division
University of California at Berkeley
Yizhou Yu Andras Ferencz Jitendra Malik
Image-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and Rendering
– Vary viewpoint
– Vary lighting
– Vary scene configuration
– Vary viewpoint
– Vary lighting
– Vary scene configuration
Recover Models of Real World Scenes and Make Possible Various Visual Interactions
Image-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and Rendering
• 1st Generation---- vary viewpoint but not lighting– Recover geometry ( explicit or implicit )
– Acquire photographs
– Facade, Plenoptic Modeling, View Morphing, Lumigraph, Layered Depth Images, Concentric Mosaics, Light Field Rendering etc.
• 1st Generation---- vary viewpoint but not lighting– Recover geometry ( explicit or implicit )
– Acquire photographs
– Facade, Plenoptic Modeling, View Morphing, Lumigraph, Layered Depth Images, Concentric Mosaics, Light Field Rendering etc.
Image-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and Rendering
• 2nd Generation---- vary viewpoint and lighting– Recover geometry & reflectance properties
– Render using light transport simulation or local shading
– [ Yu, Debevec, Malik & Hawkins 99 ], [ Yu & Malik 98 ], [ Sato, Wheeler & Ikeuchi 97 ]
• 2nd Generation---- vary viewpoint and lighting– Recover geometry & reflectance properties
– Render using light transport simulation or local shading
– [ Yu, Debevec, Malik & Hawkins 99 ], [ Yu & Malik 98 ], [ Sato, Wheeler & Ikeuchi 97 ]
Original Lighting & Viewpoint Novel Lighting & Viewpoint
Image-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and Rendering
• 3rd Generation--Vary spatial configurations in addition to viewpoint and lighting
• 3rd Generation--Vary spatial configurations in addition to viewpoint and lighting
Novel Viewpoint Novel Viewpoint & Configuration
Our FrameworkOur FrameworkOur FrameworkOur Framework
• Input– Multiple range scans of a
scene
– Multiple photographs of the same scene
• Input– Multiple range scans of a
scene
– Multiple photographs of the same scene
• Output– Geometric meshes of each
object in the scene
– Registered texture maps for objects
• Output– Geometric meshes of each
object in the scene
– Registered texture maps for objects
Overview & VideoOverview & VideoOverview & VideoOverview & Video
RangeImages
RadianceImages
PointCloud
PointGroups
Meshes SimplifiedMeshes
CalibratedImages
TextureMaps Objects
Registration Segmentation Reconstruction
PoseEstimation
Texture MapSynthesis
OutlineOutlineOutlineOutline
• Overview
• Range Data Segmentation
• Radiance Image Registration
• Meshing and Texture-Mapping
• Results
• Future Work
• Overview
• Range Data Segmentation
• Radiance Image Registration
• Meshing and Texture-Mapping
• Results
• Future Work
OverviewOverviewOverviewOverview
RangeImages
RadianceImages
PointCloud
PointGroups
Meshes SimplifiedMeshes
CalibratedImages
TextureMaps Objects
Registration Segmentation Reconstruction
PoseEstimation
Texture MapSynthesis
Range Image RegistrationRange Image RegistrationRange Image RegistrationRange Image Registration
• Registration with Calibration Objects– Cyra Technologies, Inc.
• Registration without Calibration Objects– [ Besl & McKay 92 ], [ Pulli 99 ]
• Registration with Calibration Objects– Cyra Technologies, Inc.
• Registration without Calibration Objects– [ Besl & McKay 92 ], [ Pulli 99 ]
OverviewOverviewOverviewOverview
RangeImages
RadianceImages
PointCloud
PointGroups
Meshes SimplifiedMeshes
CalibratedImages
TextureMaps Objects
Registration Segmentation Reconstruction
PoseEstimation
Texture MapSynthesis
Previous Work on Range Image Previous Work on Range Image SegmentationSegmentationPrevious Work on Range Image Previous Work on Range Image SegmentationSegmentation
• Local region growing with surface primitives– [ Hoffman & Jain 87 ], [ Besl & Jain 88 ], [ Newman
Flynn & Jain 93 ], [ Leonardis, Gupta & Bajcsy 95 ], [ Hoover et. al. 96 ]
• They don’t address general free-form shapes and albedo variations.
• Local region growing is suboptimal in finding object boundaries.
• Local region growing with surface primitives– [ Hoffman & Jain 87 ], [ Besl & Jain 88 ], [ Newman
Flynn & Jain 93 ], [ Leonardis, Gupta & Bajcsy 95 ], [ Hoover et. al. 96 ]
• They don’t address general free-form shapes and albedo variations.
• Local region growing is suboptimal in finding object boundaries.
Normalized Cut FrameworkNormalized Cut FrameworkNormalized Cut FrameworkNormalized Cut Framework
),(
),(
),(
),(),(
VBasso
BAcut
VAasso
BAcutBANcut
DyyWD )(
[ Shi & Malik 97 ], [ Malik, Belongie, Shi & Leung 99 ]
Recursive Binary Graph Partition by Minimizing
Approximate Solution
From 2D Image To 3D Point CloudFrom 2D Image To 3D Point CloudFrom 2D Image To 3D Point CloudFrom 2D Image To 3D Point Cloud
• Neighborhood– 3D spherical region
• Complexity– E.g. nineteen 800 by 800 scans
– Clustering
• Neighborhood– 3D spherical region
• Complexity– E.g. nineteen 800 by 800 scans
– Clustering
Point Cloud Segmentation: CuesPoint Cloud Segmentation: CuesPoint Cloud Segmentation: CuesPoint Cloud Segmentation: Cues
Normal Orientation
Returned Laser Intensity
Proximity
EuuEvudist pnu /||||||||),( 22
)),(),,((MAX),( vudistvudistvudist vu
nu
puu
v
Point Cloud Segmentation: Graph SetupPoint Cloud Segmentation: Graph SetupPoint Cloud Segmentation: Graph SetupPoint Cloud Segmentation: Graph Setup
Nodes: Clusters
Edges: Local + Random Long-range Connections
Weights: ),(),(),( vuPvuSvuw
)2/),(diff(exp),( 22svuvuS
)2/),(dist(exp),( 22pvuvuP
u
v
),( vuw
Point Cloud Segmentation: CriterionPoint Cloud Segmentation: CriterionPoint Cloud Segmentation: CriterionPoint Cloud Segmentation: Criterion
• Use Normalized Cut Criterion to Propose Cut
• Use Normalized Weighted Average Cut to Accept Cut
• Use Normalized Cut Criterion to Propose Cut
• Use Normalized Weighted Average Cut to Accept Cut
)),(
),(,),(
),((),(
BBWAasso
BAWAcut
AAWAasso
BAWAcutMINBANWAcut
),(
),(),(),(
vuP
vuPvuSBAWAcut
Point Cloud Segmentation: AlgorithmPoint Cloud Segmentation: AlgorithmPoint Cloud Segmentation: AlgorithmPoint Cloud Segmentation: Algorithm
• Coarse Segmentation– Clustering
– Cluster Segmentation
• Recursive segmentation based on normal continuity and proximity
• Recursive segmentation based on continuity in laser intensity and proximity
• Fine Segmentation– re-clustering and segmentation on each group from
coarse segmentation
• Coarse Segmentation– Clustering
– Cluster Segmentation
• Recursive segmentation based on normal continuity and proximity
• Recursive segmentation based on continuity in laser intensity and proximity
• Fine Segmentation– re-clustering and segmentation on each group from
coarse segmentation
OverviewOverviewOverviewOverview
RangeImages
RadianceImages
PointCloud
PointGroups
Meshes SimplifiedMeshes
CalibratedImages
TextureMaps Objects
Registration Segmentation Reconstruction
PoseEstimation
Texture MapSynthesis
Previous Work on Pose EstimationPrevious Work on Pose EstimationPrevious Work on Pose EstimationPrevious Work on Pose Estimation
• Mathematical Background– from 3 or more points [Hung et.al. ‘85],
[Haralick et.al. ‘89], Overview [Haralick et.al. ‘94]
– from points, lines and ellipse-circle pairs [Qiang et.al. ‘99]
• Automatic Techniques– Combinatorial search on automatically detected
features [Huttenlocher and Ulman ‘90],[Wunsch and Hirzinger ‘96]
– Hybrid approach (user initialization with finding silhouettes) [Neugebauer and Klein ‘99]
• Mathematical Background– from 3 or more points [Hung et.al. ‘85],
[Haralick et.al. ‘89], Overview [Haralick et.al. ‘94]
– from points, lines and ellipse-circle pairs [Qiang et.al. ‘99]
• Automatic Techniques– Combinatorial search on automatically detected
features [Huttenlocher and Ulman ‘90],[Wunsch and Hirzinger ‘96]
– Hybrid approach (user initialization with finding silhouettes) [Neugebauer and Klein ‘99]
Radiance Image Registration Radiance Image Registration Radiance Image Registration Radiance Image Registration
• Requirements– Automatic: many (50 - 200) images need to be
registered
– Accurate: Registration must be accurate to within one or two pixels for texture mapping
– General purpose: scene may be complicated, possibly without easily and uniquely identifiable features
• Solution– Place registration targets in scene
• Requirements– Automatic: many (50 - 200) images need to be
registered
– Accurate: Registration must be accurate to within one or two pixels for texture mapping
– General purpose: scene may be complicated, possibly without easily and uniquely identifiable features
• Solution– Place registration targets in scene
Finding Targets in ImagesFinding Targets in ImagesFinding Targets in ImagesFinding Targets in Images
• Pattern matching followed by ellipse fitting• Pattern matching followed by ellipse fitting
Combinatorial SearchCombinatorial SearchCombinatorial SearchCombinatorial Search• Brute force
– 4 correspondences each image, O(n4) time n targets
• Alternative– Use fitted ellipse parameters in addition to position to
estimate pose from 2 target matches, O(n2) time
• 20 seconds for scene with 100 targets
• Brute force– 4 correspondences each image, O(n4) time n targets
• Alternative– Use fitted ellipse parameters in addition to position to
estimate pose from 2 target matches, O(n2) time
• 20 seconds for scene with 100 targets
3D Targets
Registration SummaryRegistration SummaryRegistration SummaryRegistration Summary
• Place registration targets in scene before acquiring data
• Automatically detect targets in data
• Perform combinatorial search to match image targets to corresponding target in geometry
• Find camera pose from matched points
• Remove targets from images and fill in holes
• Place registration targets in scene before acquiring data
• Automatically detect targets in data
• Perform combinatorial search to match image targets to corresponding target in geometry
• Find camera pose from matched points
• Remove targets from images and fill in holes
Camera Pose ResultsCamera Pose ResultsCamera Pose ResultsCamera Pose Results
• Accuracy: consistently within 2 pixels
• Correctness: correct pose for 58 out of 62 images
• Accuracy: consistently within 2 pixels
• Correctness: correct pose for 58 out of 62 images
OverviewOverviewOverviewOverview
RangeImages
RadianceImages
PointCloud
PointGroups
Meshes SimplifiedMeshes
CalibratedImages
TextureMaps Objects
Registration Segmentation Reconstruction
PoseEstimation
Texture MapSynthesis
Mesh Reconstruction and SimplificationMesh Reconstruction and SimplificationMesh Reconstruction and SimplificationMesh Reconstruction and Simplification
• Meshing– Use the “crust” algorithm, [ Amenta, Bern &
Kamvysselis 98 ], for coarse geometry
– Use nearest-neighbor connections for detailed geometry
– Possible to use volumetric techniques, [ Curless & Levoy 96 ], to merge meshes
• Simplification– Use quadric error metric, [ Garland & Heckbert 97 ]
• Meshing– Use the “crust” algorithm, [ Amenta, Bern &
Kamvysselis 98 ], for coarse geometry
– Use nearest-neighbor connections for detailed geometry
– Possible to use volumetric techniques, [ Curless & Levoy 96 ], to merge meshes
• Simplification– Use quadric error metric, [ Garland & Heckbert 97 ]
Reconstructed Mesh with Camera Poses Reconstructed Mesh with Camera Poses and Calibration Targetsand Calibration TargetsReconstructed Mesh with Camera Poses Reconstructed Mesh with Camera Poses and Calibration Targetsand Calibration Targets
Models of Individual ObjectsModels of Individual ObjectsModels of Individual ObjectsModels of Individual Objects
OverviewOverviewOverviewOverview
RangeImages
RadianceImages
PointCloud
PointGroups
Meshes SimplifiedMeshes
CalibratedImages
TextureMaps Objects
Registration Segmentation Reconstruction
PoseEstimation
Texture MapSynthesis
Texture Map SynthesisTexture Map SynthesisTexture Map SynthesisTexture Map Synthesis
• Regular Texture-Mapping with Texture Coordinates– Compose a triangular texture
patch for each triangle.
– Allocate space from texture maps for each texture patch to assign texture coordinates.
– The size of each texture patch is determined by the amount of variations in photographs.
– The derivative of the Gaussian is used as a metric for variations.
• Regular Texture-Mapping with Texture Coordinates– Compose a triangular texture
patch for each triangle.
– Allocate space from texture maps for each texture patch to assign texture coordinates.
– The size of each texture patch is determined by the amount of variations in photographs.
– The derivative of the Gaussian is used as a metric for variations.
Photograph
Texture Map
3D Triangle
Texture-Mapping and Object Texture-Mapping and Object ManipulationManipulationTexture-Mapping and Object Texture-Mapping and Object ManipulationManipulation
Future WorkFuture WorkFuture WorkFuture Work
• Create watertight geometric models from sparse or incomplete data
• Improve mesh simplification techniques
• Texture Compression
• Create watertight geometric models from sparse or incomplete data
• Improve mesh simplification techniques
• Texture Compression
AcknowledgmentsAcknowledgmentsAcknowledgmentsAcknowledgments
• Thanks to Ben Kacyra, Mark Wheeler, Daniel Chudak, Jonathan Kung at Cyra Technologies, Inc., and Jianbo Shi at CMU.
• Supported by ONR BMDO, the California MICRO program, and Microsoft Graduate Fellowship.
• Thanks to Ben Kacyra, Mark Wheeler, Daniel Chudak, Jonathan Kung at Cyra Technologies, Inc., and Jianbo Shi at CMU.
• Supported by ONR BMDO, the California MICRO program, and Microsoft Graduate Fellowship.