33
Point Clouds From Calibration to Classification Norbert Pfeifer [email protected] Technische Universität Wien, Department for Geodesy and Geoinformation Vienna, Austria Contributions Gottfried Mandlburger (IfP Stuttgart, Germany), Philipp Glira (Siemens, Austria), Martin Pfennigbauer (Riegl, Austria), Konrad Wenzel (nframes, Germany), Tran Giang

Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

  • Upload
    others

  • View
    17

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Point CloudsFrom Calibration to Classification

Norbert Pfeifer

[email protected]

Technische Universität Wien, Department for Geodesy and GeoinformationVienna, Austria

ContributionsGottfried Mandlburger (IfP Stuttgart, Germany), Philipp Glira (Siemens, Austria), Martin Pfennigbauer (Riegl, Austria), Konrad Wenzel (nframes, Germany), Tran Giang

Page 2: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Point Cloud Calibration and Orientation

2

Dense Image Matching (RGB)Airborne LiDAR

• 3D color vs. mainly geometry• Differences: vegetation, crane• Piles of construction material• DIM: stereo occlucion • LiDAR: points between piles available

DIMLiDAR

How can these point cloudsbe „brought together“in order to understand their differences?

Mandlburger et al.

Page 3: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Point Cloud Change Detection

4 data sets taken at 4 different times Different modality

RGB, nIR, photo/lidar, GSD, point density, shadow, …

Classified Lidar point cloudsground, building, vegetation

32005 2011

Page 4: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Computational Geometry – Point Cloud Legacy

Design Process• Given: Point cloud pi ∈R3, i = 1, … n• Find: surface s(u,v) closely interpolating pi , i.e. minimize f ( s(ui,vi) - pi )• NURBS (…) surface, patch layout based on curvature, curvature defined in each point

5Image: learning alias

Page 5: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Computational Geometry – Point Cloud Legacy

Design Process• Given: Point cloud pi ∈R3, i = 1, … n• Find: surface s(u,v) closely interpolating pi , i.e. minimize f ( s(ui,vi) - pi )• NURBS (…) surface, patch layout based on curvature, curvature defined in each point

Quality Inspection• Given: CAD model of a object, discretized as points mi

• Given: point cloud of the manufactured object, data points dj

• Find: transformation of dj onto mi , but no exact correspondence• Replace exact with approximate, iteratively determined correspondence: ICP

6Image: learning alias

Page 6: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Computational Geometry – Point Cloud Legacy

Design Process• Given: Point cloud pi ∈R3, i = 1, … n• Find: surface s(u,v) closely interpolating pi , i.e. minimize f ( s(ui,vi) - pi )• NURBS (…) surface, patch layout based on curvature, curvature defined in each point

Quality Inspection• Given: CAD model of a object, discretized as points mi

• Given: point cloud of the manufactured object, data points dj

• Find: transformation of dj onto mi , but no exact correspondence• Replace exact with approximate, iteratively determined correspondence: ICP

Visualization• Given: Point cloud pi

• Find TIN with vertices pi that follows surface of object• No 2D solution, e.g. Delaunay criterion,

but localized processing independent of overall shape

7

Page 7: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Point clouds

Computational Geometry Object centered (often) Point pi ( xi , yi , zi ) Cartesian coordinate system No datum No attributes Error free (negligible errors) No measurement process

8

Geodesy and Geoinformation Billions of points Projected CS Random (0.5-30cm), systematic (up

to meter), and many gross errors Attributes: color, accuracy, echo ID,

FWF information, time, etc. Measurement process known Measurement along optical line of

sight

Page 8: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Point cloud research domain

Data structures kD-tree

appropriate for processing, nearestneighbor search, etc.

octreeappropriate for visualization (e.g. potree)

Data structure for manuel editing, processing and visualization of pointclouds?

9

Geodesy and Geoinformation Billions of points Projected CS Random (0.5-30cm), systematic (up

to meter), and many gross errors Attributes: color, accuracy, echo ID,

FWF information, time, etc. Measurement process known Measurement along optical line of

sight

Page 9: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Point cloud research domain

Data structures kD-tree

appropriate for processing, nearestneighbor search, etc.

octreeappropriate for visualization (e.g. potree)

Data structure for manuel editing, processing and visualization of pointclouds?

Processing strategiesparallelizationtile wise processing

Neighborhoodoptimal neighborhoods, parameterselection, etc.

10

Geodesy and Geoinformation Billions of points Projected CS Random (0.5-30cm), systematic (up

to meter), and many gross errors Attributes: color, accuracy, echo ID,

FWF information, time, etc. Measurement process known Measurement along optical line of

sight

Page 10: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Point cloud research domain

Calibration and orientation sensor modeling measurement process modeling

sensor + medium/media + object observation error models

12

Geodesy and Geoinformation Billions of points Projected CS Random (0.5-30cm), systematic (up

to meter), and many gross errors Attributes: color, accuracy, echo ID,

FWF information, time, etc. Measurement process known Measurement along optical line of

sight

Glira et al.Image: Riegl

Page 11: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Point cloud research domain

Applications land cover mapping topographic modeling indoor modeling engineering surveying deformation analysis change detection

13

Geodesy and Geoinformation Billions of points Projected CS Random (0.5-30cm), systematic (up

to meter), and many gross errors Attributes: color, accuracy, echo ID,

FWF information, time, etc. Measurement process known Measurement along optical line of

sight

Page 12: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Calibration of point clouds

Integrated calibration and orientation of lidar and image matching point clouds

Where‘s the problem? Lidar point clouds are good within surfaces and for vegetation,

not influenced by (sun) shadows Image matching point clouds are good on edges and have high resolution

physical explanation: aperture size of current laser scanners vs. photogrammetric cameras

Aimcombine the advantages and exploit the differences

Requirementprecise geo-referencing

Standard methodindependent bundle block adjustment and laser scanning strip adjustment

14

Page 13: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Flight block Melk (Austria)

15

Abbey and city of Melkaerial image

Overview map

Flight strips

Mandlburger et al.

Page 14: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Results: data acquisition Melk

17

Dense Image Matching3D RGB point cloud

nframes

Page 15: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Results: data acquisition Melk

18

Airborne LiDAR3D point cloud

Riegl

Page 16: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Elevation differences with standard method

19

Height differences „LiDAR minus DIM“ before integrated strip adjustment / AT

0 100 200 m

Mandlburger et al.

Page 17: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Integration of LiDAR strip adjustment and aerotriangulation

20

Equations• Direct georeferencing• Collinearity• Point to tangent plane ICP

correspondence

Glira et al.

Page 18: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Results: data acquisition Melk

22

Height differences „LiDAR minus DIM“ before integrated strip adjustment / AT

0 100 200 m

Mandlburger et al.

Page 19: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Results: data acquisition Melk

23

Height differences „LiDAR minus DIM“ after integrated strip adjustment / AT

vegetation

vegetation vegetation

Mandlburger et al.

Page 20: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Results: data acquisition Melk

24

Height differences „Lidar minus DIM“ Masked height differences „LiDAR minus DIM“ after hybrid adjustment

0 100 200 m

Mandlburger et al.

Page 21: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

DSM generation from LiDAR and DIM

25

LiDARDIM: Default settings DIM: Refinedsettings LiDAR+DIM

Precision (1σ):LiDAR: 2cmDIM: 10cm = 1.67 * GSD

Mandlburger et al.

Page 22: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

LiDAR/DIM data properties: vegetation penetration

Vegetation: DIM=top of grass surface, LiDAR=penetrates grass layer Impenetrable surfaces: negligible height differences between DIM and ALS

26

colorized DIM point cloud height difference: LiDAR-DIM

beach volleyball court

soccer fieldrunning trackA BMandlburger et al.

Page 23: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Classification of point clouds

Point cloud classification Principle: use measured and computed features to infer,

which class a point belongs to (features selection?)

28

Echo widthEchoRatio Sigma0 Tran et al.

Page 24: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Classification of point clouds

Point cloud classification Principle: use measured and computed features to infer,

which class a point belongs to (features selection?) Methods: manual decision trees, support vector machines,

random forests, Markov chains, deep convolutional neuralnetworks

Considerations: transferability of rules (across missions, scales, etc.), number of classes, run time, amout of trainingdata, representation of rarely occuring classes, classification accuracy

29

Point cloud

Nor<0.2Ground Y OffGroundN

GreenRatio<0.3393

RoadRoad

Y

Grassland

N Nor<=1.5 Buildings and TreeNLowtree+Car Y

GreenRatio<0.3393 Nor<=5.5

Car LowTree

SmallBuildings and Med Tree

Y

High Buildings and high TreeN

GreenRatio<0.348

Smallbuilding

Y

Medium Tree

N

_omnivariance>1

High Tree

Y

Tree mix building

N

Nor<17

High Tree

Y

High Building

N

Y N

Tran et al.

Page 25: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Change detection in point clouds

A1: classify point clouds of different epochs + compare labels (proximity?) A2: compute geometric point cloud difference + classify changed points Problem 1

change in classand change in geometry can beindependent

Problem 2different dataproperties

Problem 33D changes

30

Page 26: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

31

Cut tree

Ground change in height >0.5m

Reconstructed building

New buildings

Unchange

New tree

Ground to another landuse

2005

2011

PC CD envisaged

result

Tran et al.

Page 27: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

PC CD Method

Exploit the power of machine learning Features for class and features for change1. measurement process: echo ID, waveform attribute, etc.2. local neighborhood: normal vector, roughness, etc.3. approximate height (simple terrain model)4. change indicators

• distance to nearest point in PC of other epoch• point distribution feature

evaluated at location of epoch 1 pointswith points of epoch 2

32Tran et al.

Page 28: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

PC CD Method

Exploit the power of machine learning Features for class and features for change1. measurement process: echo ID, waveform attribute, etc.2. local neighborhood: normal vector, roughness, etc.3. approximate height (simple terrain model)4. change indicators

• distance to nearest point in PC of other epoch• point distribution feature

evaluated at location of epoch 1 pointswith points of epoch 2

Sample training data, learn model (e.g. random forests),

apply model to detect all changes,done (?)

33Tran et al.

Page 29: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

PC CD ML

34Tran et al.

Page 30: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

PC CD ML

35Tran et al.

Page 31: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

36Tran et al.

Page 32: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

Conclusions

Many open pointcloud researchquestions

37

Page 33: Point Clouds From Calibration to Classification · Calibration of point clouds. Integrated calibration and orientation of lidar and image matching point clouds Where‘s the problem?

References

Tran et al., 2018 (accepted): Classification of image matching point clouds over an urban area. International Journal of Remote Sensing.

Tran et al., 2018: Integrated Change Detection and Classification in Urban Areas Based on Airborne Laser Scanning Point Clouds. Sensors 18.

Mandlburger et al., 2017: Improved Topographic Models via Concurrent Airborne LIDAR and Dense Image Matching. ISPRS Annals IV-2/W4.

Glira et al., 2016: Rigorous Strip adjustment of UAV-based laserscanning data including time-dependent correction of trajectory errors. Photogrammetric Engineering & Remote Sensing 82.

Glira et al., 2015: A Correspondence Framework for ALS Strip Adjustments based on Variants of the ICP. Photogrammetrie, Fernerkundung, Geoinformation 2015.

Pfeifer et al., 2014: OPALS–A framework for Airborne Laser Scanning dataanalysis. Computers, Environment and Urban Systems 45.

Otepka et al., 2013: Georeferenced point clouds: A survey of features and point cloud management. ISPRS International Journal of Geo-Information 2.

38