Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
IVSIAC 產學小聯盟成果發表會
Advanced ADAS Technology
Prof. Jiun-In GuoDept. of Electronics Engineering, National Chiao Tung University
Hsinchu, Taiwan
June 26th, 2015
1
Outline
• Design trends
• State-of-the-art technology
• Overview on NCTU iVS intelligent vision for automobiles
• Advanced ADAS technology
• Some application scenarios
• Conclusion
2
Outline
• Design trends
• State-of-the-art technology
• Overview on NCTU iVS intelligent vision for automobiles
• Advanced ADAS technology
• Some application scenarios
• Conclusion
3
Design Trend for Car Safety: From Passive to Active
4
Euro NCAP Regulation
5
6
Technology for Self Driving
Car Tech News in CES 2015
• Self-driving cars, Apple CarPlay and Android Auto take over CES 2015
• Delphi's computer chauffeur drives me around Las Vegas at CES 2015
• CES 2015: Cruise control gains self-steering with Valeo'sCruise4U
• Volkswagen's Trained Parking is a robotic valet for your home
• CNET's Connected Car panel explores the future of transportation at CES 2015
• Hands off with the Volkswagen Golf R Touch at CES 2015
• Cars of the future won't need drivers: CES 2015
7
Outline
• Design trends
• State-of-the-art technology
• Overview on NCTU iVS intelligent vision for automobiles
• Advanced ADAS technology
• Some application scenarios
• Conclusion
8
Mobileye Technology
Features:1. Headway Monitoring and Warning2. Intelligent High-Beam Control
3. Daylight Pedestrian Collision Warning,
including Bicycle Detection
Vision SoC for Safety Driving
4. Forward Collision Warning,
both in Highway and Urban areas,
including Motorcycle Detection
5. Lane Departure Warning
6. Traffic light recognition
7. Speed limit indication
9
TI TDA2x SoC for Advanced Driver Assistance Systems (ADAS)
1010
TI’s Automotive Technologies
Advanced ADAS based on Machine Learning
• Nvidia Drive PX
11
Outline
• Design trends
• State-of-the-art technology
• Overview on NCTU iVS intelligent vision for automobiles
• Advanced ADAS technology
• Some application scenarios
• Conclusion
12
Research Team
13
Dr. Jiun-In Guo
Dr. Ching-Lung Su
Dr. Ching-Wei Yeh
Dr. Kuan-Hung Chen
NCTUProfessor and Director of Institute of Electronics Department of Electronics Engineering
CCUProfessor
Department of Electronics Engineering
FCUProfessor
Department of Electronics Engineering
YuntechProfessor Department & Graduate school of Electronics Engineering
ADAS Technology We Developed
14
Blind Spot Detection
Auto headlight control
Auto High Beam Control
Auto Windshield Wiper Control
Animated Car-Backing Track
Hands-free Trunk Lift gate
Inclement Weather Processing
Wide-view Video Stitching
HDR Night Vision(Single-CAM, Dual-CAM, Micro Camera Array)
Traffic Light Detection
Speed Limit Detection(Taiwan)
USA Driver Dangerous Behavior Detection
Car License Plate Detection
Pedestrian and Scooter Detection
(On IMX6)
Lane Departure Warning (Day, Night, Rain)
Forward Collision Warning(Day, Night, Rain)
LDWS+FCWS(Day, Night, Rain)
Stop & Go
2D/3D Hand Tracking (Single-CAM and Dual-CAM)
Hand Writing
ADAS Technology Developed in NCTU
15
Performance Summary
16
Function Feature
Performance
CPU Resolution FPS
Lane Departure Warning System (LDWS)
• Support variety of road lanes(Solid, dashed, cat-eyes)
• Support variety of weather conditions(Sunny, rainy, foggy days, nights, etc.)
ARM Cortex-A9 1GHz D1 30
Forward Collision Warning System (FCWS)
• Detection distance at day time: 0~50m• Detection distance at night time: 0~40m
ARM Cortex-A9 1GHz D1 30
Stop & Go System (working with FCWS)
• Working distance at day: 0~10m ARM Cortex-A9 1GHz D1 30
Pedestrian and Motorcycle Detection
System
• Obstacle type: pedestrian, motorcycle, bicycle
• Multiple ROI detection• Moving Object Detection IP
ARM Cortex-A9 1GHz D1 30
AMD A10-7850K GPU
D1 30
Accelerator (200MHz in TSMC 90nm)
D1 30
Blind Spot Warning System (BSWS)
• Detection region: outward 0~3m, backward 0~20m
• Object type: vehicle, motorcycleARM Cortex A9 1GHz VGA 30
Inclement Weather Processing Technology
• Dynamic local contrast enhancement • Automatic harsh environment judgment• Support a variety of inclement weather
conditions (nights, rainy, cloudy, foggy days, sandstorm)
ARM Cortex A9 1GHz D1 30
Performance Summary
17
Function FeaturePerformance
CPU Resolution FPS
Traffic Light Detection System
• Vehicle speed: 0~100 km/h ARM Cortex A9 1GHz D1 30
Speed Limit Detection System
• Vehicle speed: 0~150 km/h• Support speed limit sign of North America,
Taiwan, Europe, and China
Intel Atom 1.6GHz (USA)
D1 50
ARM Cortex A9 1GHz (Taiwan, EU)
D1 15
Wide-view Video Stitching System
• Support variety of weather conditions (Sunny, rainy, cloudy, nights, etc.)
• Extending Panoramic to 360 degree surround view
ASIC4ch HD720@
178Hz 30
ASIC4ch HD1080@
202MHz30
Single-camera HDR Night Vision System
• Good for scenes with high variation in lighting (Tunnel, night view with strong light, etc. )
ARM Cortex-A9 1 GHz VGA 13
Dual-Camera HDR Night Vision System
• Good for scenes with high variation in lighting (Tunnel, night view with strong light, etc. )
ASIC HD1080 30
Driver Dangerous Behavior Detection
• To detect eating, phone calling and dozingTREK-668 (Intel Atom
1.6GHz)360x240 15
2D/3D Hand Tracking for Gesture Control
• Single-camera(2D)/Dual-camera(3D) vision based system
Intel Core i7-2600 3.40GHz (3D)
VGA 24
Outline
• Design trends
• State-of-the-art technology
• Overview on NCTU iVS intelligent vision for automobiles
• Advanced ADAS technology
• Some application scenarios
• Conclusion
18
Highlight on Advanced ADAS Technology
19
Speed Limit Detection(Taiwan)
USA
Pedestrian and Scooter Detection
Lane Departure Warning
Forward Collision Warning
Cross Traffic Detection
Vision-based system
Support resolution: Up to 720x480
Support variety of weather conditions(Sunny, rainy, foggy days, nights, etc.)
Support variety of road lanes(Solid, dashed, cat-eyes)
Lane Departure Warning System (LDWS)
20
Demo Video (Day, Night, Rain)
100 fps D1@ Intel Atom 1.6GHz (using one core)30 fps D1@ Cortex-A9 (Freescale iMX6 1GHz using one core)
Detected rate: 97.43%False alarm rate: 7.89%
LDWS Flow chart
21
Image Capture
Adaptivethreshold
Integral Image
Line-thinning
Houghtransform
Departure
Right laneFSM
Left laneFSM
Line Collection
Lane detectprocess
line detectprocess
Patent Comparison: LDWSMobileye
WO0180068A1、US6704621、US8254635、US2013141580A1、US2012069185A1
NCTU iVS LAB(Patent pending)
Process
Steps Methods Steps Methods
XImage pre-processing
ROI setting
Detect lane boundaries
(3-D)
1. Road skeleton estimation system
(1) Roadway is modeled as a circular/parabolic arc parallel to XZ plane in 3-D space
(2) Two-phase model(3) Value cost function
determine the line (Warped image)
Lane detection(2-D)
Line detection1. Pixel-based Adaptive Threshold
(PAT): dynamically distribute image into lane and non-lane part.
2. Sobel and Randomized Hough transform (RHT)
Lane definition1. Slope2. Vanishing point distance3. Previous frame lane
information4. Lane information finite state
machineFind road curvature XVehicle relationship
to laneX
Time to lane crossing (TLC)
Wheel-to-lane distanceDeparture warning
Image center to Lanes center distance
22
Vision-based system
Detection distance at day time: 0~50m
Detection distance at night time: 0~40m
Support resolution: Up to 720x480
Forward Collision Warning System (FCWS)
23
Demo Video (Day, Night, Rain)
30 fps D1@ Intel Atom 1.6GHz (using one core) with LDWS30 fps D1 @ Cortex-A9 (Freescale iMX6 1GHz) with LDWS
Detected rate: 90.24%False alarm rate: 5.40%
FCWS Flow chart
ROI setting
Image capture
Day or Night
Edge feature
Shadow Feature
Combination
Tracking
Distance estimation
Light feature
Edge feature
FCWSDay
FCWSNight
Patent Comparison: FCWS
Mobileye (Note) NCTU-IVS
Detect Vehicle
Generate candidate regions of interest:1. Rectangular shaped region for all
position and all size2. Filter out candidates lack of texture
and in-compliance with perspective constrains on range and size.
Single frame classification:1. 2-stage classification algorithm.2. SVM + Adaboost.Multi-frame Approval Process:1. Additional information collected
over a number of frames are used in system.
Generate candidate regions of interest:1. Noon or Night judgement.2. Noon : Shadow feature extraction.3. Night : Light-pair feature extraction.Single frame classification:1. Vertical feature verification for noon
and night.Candidate Vehicle buffer:1. Accumulate feature frequency
according to spatial information.2. Select maximum temporal frequency
candidate as vehicle.Tracking state:1. Finite state machine and kalman filter.
Range estimation
Pinhole model and position-based geometry.
Horizontal line + position-based geometry.
Time to collision
Estimate TTC by Scale change and frame rate.
X
25
Note: Itay Gat, Meny Benady, and Amnon Shashua, “A Monocular Vision Advance Warning System for the Automotive Aftermarket,” SAE Technical Paper 2005-01-1470, 2005, doi:10.4271/2005-01-1470.
Vision-based system using machine learning
Support resolution: Up to 720x480
30 fps D1 @ Intel Atom 1.6GHz (using single core)30 fps D1 @ Cortex-A9 (Freescale iMX6 1GHz)30 fps D1 @ AMD A10-7850K GPU Realization30 fps D1 @ Accelerator (200MHz in TSMC 90nm)
Obstacle type: pedestrian, motorcycle, bicycle
Multiple ROI detection
Pedestrian and Motorcycle Detection System
26
Demo Video (PC)
Demo Video (IMX6)
Detected rate: 85 %False alarm rate: 3.6 %
Detection distance: 0~40m (D1)
Proposed Adaboost-based Detection Algorithm
27
Y
Y
Feature value
Calculation
Stage Sum
> thresholdReject
Accumulate
Weighting Value
for Stage Sum
Final Stage?
Choosing
“Left”or“Right”Weighting Value
Adjusted Threshold =
Variance * Feature Threshold
Start a New
Stage
Accept
Feature
Coordinate
Decoding
Integral & Square
Integral Image
Calculation
N
N
3m~5m (360x65)
1m~3m(180x55)
0.5m~1m (180x80)
ROI
Inner
Down-sample
Image
Original
Image
Choose Source
Image
No More
Feature
Y
N
Stage Number
from Classifier
Node number
from Classifier
1. Outside of 3m
2. Within 3m
(full-length)
3. Within 3m
(half-length)
Classifiers
Stage Threshold
from Classifier
Next stage
ROI
Dynamic setup
1. Classifier type
2. Classifier stage
Search size : 14x28~40x80
Next node
Off-line
14x28
40x80
Classifier Stages
Feature Node
12 types
Moving Object Detection IP
Pedestrian and Motorcycle Detection in ASIC
28
Specification
Technology
Gate Count
On-Chip Memory
Capability D1@30fps
TSMC 90nm CMOS
173K Gates
35.7K Bytes
Memory Interface AXI 32bit Bus
Max Frequency 200MHz
Video Format YUV 4:2:0
Detection Rate 85% (based on classifiers)
SRAM (17.4KB)
Search
Window
Buffer
(14.5KB)
Mesh &
Integral pixel
Calculator
(48x88x21bit)
(48x88x8bit)
Feature
Coordinate
Calculator
Classifier 2
Classifier 1
Coordinate
Buffer
Object
Recognition
Engine
AXI Master AXI Slave
Register
Files
(feature number,
feature type)
Parameter Bus
Candidate
Window
DRAM
RISCSource Images
Unique Features
· Supporting pedestrian/motorcyclist detection
· Supporting close-range detection
· Dynamic ROI region
· Fully HW computation
Verification
· RTL simulation pass
· Altera Stratix4 FPGA testing pass
· ESL Aldec FPGA emulation pass
Vision-based system
Support resolution: HD720/D1/VGA
Vehicle speed: 0~150 km/h
D1 23fps@Intel Atom 1.6GHz (Taiwan, EU)D1 50fps@Intel Atom 1.6GHz (USA)
Support speed limit sign of different countries
Speed Limit Detection System
29
Demo Video (Taiwan)
Demo Video (USA)
Detected rate: 94.34 % (USA), 94.52% (Taiwan, EU)False alarm rate: 5.66 % (USA), 5.48% (Taiwan, EU)
Proposed Algorithm for USA Speed Limit Detection
1. Rectangle Detector
2. Achromatic Decomposition
3. Adaptive Threshold
4. Digit Segmentation
5. Digit Normalization
6. Digit Recognition
30
• Behavior analysis• Eating detection
• Detect the distance between hands and mouth
• Phone calling detection
• Detect if hand tracking blob locate beside the face
• Dozing off detection
• The system can not detect eyes for more than 15 frames in the face region
• Dangerous driving warning• Eating detection
• Phone calling detection
• Dozing off detection
31
Driver Behavior Analysis System (DBAS)
Alley View for Cross Traffic Detection
• Goal: • To show the left/right sides of view through view
transformation on a fish-eye camera
• Could be combined with object detection
32
Input: Fish-eye view Output: Alley view
Multiple People Detection, Tracking, and Behavior Analysis
33
• Major Functions• People Detection
• People Tracking
• Behavior Analysis
• Challenges• Lightness variation
• Strong lightness variation by turn on/off the light indoors
• People keeping static• People will be updated to
background if they keep static
• People crossing• The labels often exchange
when people switch
• Moving objects• When objects move(like chair),
it will be detected
Functions
Behavior Analysis
Video Sequence
PeopleTracking
Falling Down
Detection
People detection
People Labeling
Posture Analysis
Applications
Virtual Fence
Trajectory Recording
Aware!
Falling down detection
Virtual fence
Demo Video for Multiple People Detection, Tracking, and Behavior Analysis
34
Outline
• Design trends
• State-of-the-art technology
• Overview on NCTU iVS intelligent vision for automobiles
• Advanced ADAS technology
• Some application scenarios
• Conclusion
35
ADAS for Fleet Management
• Concept
36
ADAS Algorithms We Adopted
Lane Departure Warning System (LDWS)
Forward Collision Warning System (FCWS)
Driver Behavior Analysis System (DBAS)
Speed Limit Detection (SLD)
37
ADAS for Fleet Management
• Dangerous driving behaviors
38
車道偏移警示
前車防撞警示
速限標示偵測
危險駕駛行為警示
左車道偏移
右車道偏移
前方車距
道路速限
手部偵測
臉部偵測
五官偵測
蛇行單位時間內,訊號交互
系統 功能 時間資訊 車速資訊 危險行為
單位時間內,維持短距 高速 逼車
<當前車速 超速
使用手機
進食
打瞌睡
左車道偏移警示
前車過近警示
方向燈訊息
右車道偏移警示
無左方向燈
無右方向燈
ADAS for Fleet Management
•危險行為回傳資料
39
危險行為
回傳資料
車輛ID 時間 地點(GPS) 當前車速 規定速限 偏移資訊 距離資訊 駕駛行為
蛇行 ● ● ● ●
車道偏移 ● ● ● ●
逼車 ● ● ● ● ●
前車過近 ● ● ● ● ●
超速 ● ● ● ● ●
使用手機 ● ● ● ●
進食 ● ● ● ●
打瞌睡 ● ● ● ●
Multiple ADAS Functions in Freescale i.MX6 with Multithreading
• Approaches• Multiple threads
are created• Captured thread
• Worker threads• LDWS, FCWS, SLD,
DBAS
• Display thread
• Thread pipelining• Synchronized by
FIFO in data transfer
40
pth_DLCE
pth_LDWS
pth_PDS
Write data packet to display
thread
Read data packet from the last
worker thread
Draw the boundaries of driving
lane and the marking windows of
detected pedestrians
Capture thread
Display thread
pth_ADAS
working loop
initialization
Create worker threads and FIFOs
working pipeline
pth_FCWS
Pth_LDWS
Pth_FCWS
Pth_SLD
Pth_DBAS
Performance Summary
41
Freescale iMX6 Quad Core Cortex A9 (BD-SL-i.MX6)
(Automotive grade)
系統 平台 CPU 解析度 效能
A. 車道偏移警示
Freescale i.MX6 Cortex A9 1 GHz D1(720x480)
30fps
B. 前車防撞警示 30fps
C. 速限標誌偵測 24fps
D. 駕駛行為分析 15fps
E. A+B 30fps
F. A+B+C 24fps
G. A+B+C+D 15fps
System Integration
42
• ADAS box (IMX6) + IDU-300 (3G) + Cloud Server
Client Side
43
車隊管理 I.MX6 demo (camera on road)功能: FCWS + LDWS + SLD
系統: 40km/h + (-1~+1)km/h
資訊: 統計五分鐘內偏移次數、未保持安全車距、超速等行為
Server Side
• GCP(receive message from IDU-300)
• GCP_AP(decode G2A data)
44
GCP
GCP_AP
Our waring message
Vision Radar for ADAS
• LDWS+FCWS in traffic map
45
System Initial
Parameter Setting
Line model Vehicle model
Image capture
Not initial Initial already
Lane departure warning system(Multiple lanes detection)
Forward collision warning system(Multiple vehicles detection)
Lane width estimation
Locate vehicle x position
Build traffic map(vehicle position& departure
judgement)
Outline
• Design trends
• State-of-the-art technology
• Overview on NCTU iVS intelligent vision for automobiles
• Advanced ADAS technology
• Some application scenarios
• Conclusion
46
Conclusion - On-going MOST Project (ViDAR) (2014/8~2017/7)
47
IVVS: Intelligent Vehicle ViDAR System
Multiple Object Identification Multiple Object Recognition Multiple Object Ranging Multiple Object Tracking Multiple Object Behavior Analysis HDR Night Vision 360/540 Degree Surround View
Multiple Object Identification
Multiple Object RecognitionMultiple Object Ranging
Multiple Object Tracking
Multiple Object Behavior Analysis
HDR Night Vision
360/540 Degree Surround View
Conclusion - Roadmap
• Intelligent Vision Technology for Advanced ADAS Applications
48
ADAS (2013-2015) Advanced ADAS (ViDAR) with deep learning(2015-2017)
Advanced ADAS (ViDAR) with deep learning and 360 degree
surround view by SPADscanning
(2017-2019)
Conclusion
• Intelligent Vision is an emerging topic in both academia and industry.• ADAS has become an attractive feature in the vehicles
developed by the major automobile companies in the world.
• Machine learning is also a hot topic in image/vision content analysis.
• Advanced ADAS is the trend for automotive applications.
• Sensor fusion is required (Vision + mmwave + Laser (SPAD))
• More and more intelligence is required and will be invented in the near future.• Technology always begins from Human.
49
50
Thank you very much for your attention !
http://ivs.ee.nctu.edu.tw/iac/
Our Vision, Your Intelligence !
Q&A