How do computer vision technique work in drone applications?

Hyon Lim (임현) Co-founder & CEO UVify, Inc.

http://www.uvify.com

1

2016 컴퓨터비전 및 패턴인식 워크샵 CVPR 기술과 국내 벤처 기업 동향

Corporate Introduction• Short presentation (another file)

2

Why vision?• Vast information

• Extremely low Size, Weight, and Power (SWaP) footprint

• Cheap and easy to use

• Passive sensor

• GPS is prone to jamming & signal lossand not working in indoor environment

3

Challenges• Real-time performance is required

• Full 6-DoF motion of MAVs

• Fast vehicle dynamics

• Limited payload ( < 1kg)

• Real-time performance under limited sensing and computation power

• Complex environments

• Modeling unknown environments, environment changes4

Research problems considered

5

Image-based Localization Visual SLAM Optical-flow navigation

Long-term, large-scale Short-term, small-scale

Part 1 Part 3Part 2

Part I . Image-based Localization

6

Hyon Lim, SN Sinha, et al. “Real-time image-based 6-dof localization in large-scale environments”, CVPR 2012. Hyon Lim, SN Sinha, et al. “Real-time monocular image-based 6-dof localization”, IJRR 2014.

Approaches of localization using images

7

Query image

Geo-tagged image database

Approximated location (place recognition)

Query imageRetrieved image

Retrieved 6-DoF pose

Geo-registered 3D map

Exact location/orientation (our goal)

Goal of image-based localization

8

Scene

Query image

Find exact position and orientation (pose) of given image in 3D model

Robot’s view

System flow

9

Proposed Method

Structure from Motion

Camera clustering

Multi‐scale feature extraction

Database (kd‐tree)

Offline pipeline

Interest point detection

ExtractBRIEF

Pose Estimation

Kalman Filter Update

Online pipeline

ExtractDAISY

2D‐2DTracking

2D‐3DMatching

Image-based localization offline steps

10

3D points + calibrated cameras

Camera is clustered based on visibility of SfM 3D points Overlapping clusters used for coarse localization

Multi-scale DAISY feature extraction

Structure from

Motion

Image-based localization online steps

11

1 2 3 4

Feature tracking

Matching

Pose estimation

Filtering

Overall pipeline

12

2D-2D Feature tracking

… … … …

3D-2D matching Camera pose estimationP3P + RANSACKd-Tree + Priority matching

Global and guided matching

Binary descriptor + Harris

Experimental result (outdoor)

13

Input video 2D-2D tracking

Synthesizedpoint-cloud

view

Statistics

Global point-cloud

view

Experimental result (indoor MAV)

14

Application: semantic localization

15

Part 2 . Real-time Visual SLAM

16

Hyon Lim, Jongwoo Lim, H. Jin Kim, “Real-time 6-dof monocular visual SLAM in a large-scale environment”, ICRA 2014.

System demo

17

Part 2-1. RGB-D Real-time Visual SLAM18

Experimental result

19

Proposed method Existing method

Result : new keyframe selection

20

Result : feature-based loop closing

21

Direct 2D-3D localization result

Online pose estimation resultNote:

Once direct 2D-3D localization succeeded, map uncertainty is also reduced.

Direct 2D-3D localization result

Online pose estimation resultNote:

Once direct 2D-3D localization succeeded, map uncertainty is also reduced.

Result : feature-based loop closing

22

Conclusion

26

• Precise real-time vision-based 6-DoF estimation methods are developed

• Image-based localization : Real-time image-based localization in large-scale environment

• Visual SLAM : Monocular and RGB-D sensor with a new keyframe selection and loop closing method

• Light-weight algorithm for low-level stabilization of MAVs for short-term, small-scale motion.

Future work• Life-long visual navigation

• Map saving after visual SLAM and Image-based Localization afterwards

• Map enhancement during IBL pipeline

• 365-day mapping & localization

• Large-scale collaborative multi-vehicle mapping and localization based on IBL + VSLAM

• Extend to mobile device mapping and localization

27

Thank you for the attention

The end

28