Using Model-Based Design to develop high quality · Using Model-Based Design to develop high quality ... Example of Analysis and Re-simulation Tool 1. Gather data - Instrument Toolox

1

2© 2016 The MathWorks, Inc.

Using Model-Based Design to develop high quality

and reliable ADAS & Automated Driving Systems

김종헌차장

Senior Application Engineer

MathWorks Korea

3

Electronics & Active Safety: Helping to increase Road Safety

A reduction of 66% between 1991-2013.

Innovation of automotive safety & ADAS solutions made this possible.

76230

25938

?How much were road fatalities reduced over the last 20 years?

Evolution of Road Fatalities in EU (1991-2013)Source: Website European Commission

4

Challenges in ADAS & AD Development

Algorithm

– Complicate algorithm to analyze the environment

surrounding the car and make a decision in real-time

Perception

Localization

Situation analysis & Driving Behavior

Path Planning, Actuator Control

Process

– Vehicle prototype is difficlt for testing in early

engineering stage

– Limited flexibility to vary and explore different design

directions with real prototype

Test

– Vast amount of operating scenarios and data to test

(Millions of test km required)

5

Agenda

Challenges to automate the driving

Example workflow for ADAS algorithm development

– Managing & analyzing big data from vehicle fleet test

– Algorithms MATLAB/Simulink provide to support ADAS development

– Verifying and implementing ADAS algorithms using MBD

ROS Interface for Automated Driving system development

Q & A

6

MATLAB Gives Design Engineers the Environment for

Algorithm Design, Exploration, and Verification

Regression

Test

C Code

Algorithm

Interactive

Exploration

Data

7

Developing models for ADAS Algorithms

Regression

Test

C Code

Algorithm

Interactive

Exploration

Data

Regression

Test

C Code

Algorithm

Interactive

Exploration

Data

8

Example of Analysis and Re-simulation Tool

1. Gather data- Instrument Toolox- Vehicle Network Toolbox- Image Acquisition Toolbox

Model-Based Approach to Resource-Efficient

Object Fusion for an Autonomous Braking System

Jonny Andersson, Scania,

MathWorks Automotive Conference 2015

2. Re-simulate- MATLAB/Simulink- Parallel Computing Toolbox- MDCS

9

Example of Visualization and Algorithm development Tool

• Productivity

Data visualization

Multi-modal data analysis

• Allow Engineers Rapidly

Build and Share Tools

Easy-to-use plot functions

Build-and-deploy

Acquisition of Heterogeneous

Data (CAN & Image)

Video play

Vision/Radar data analysis

Sensor fusion algorithm development

Birds-Eye

View

10

Example of Analysis and Re-simulation Tool

1. Gather data- Instrument Toolox- Vehicle Network Toolbox- Image Acquisition Toolbox

2. Re-simulate- MATLAB/Simulink- Parallel Computing Toolbox- MDCS

3. Find situations- Various Data Analytics Tools

4. Classify and analyze- Various Data Analytics Tools- MATLAB Compiler- Report GeneratorModel-Based Approach to Resource-Efficient

Object Fusion for an Autonomous Braking System

Jonny Andersson, Scania,

MathWorks Automotive Conference 2015

11

MathWorks Tools Help Automate Ground Truth Labeling

Tools for

– Sensor input and visualization

– Video annotation

– Tracking and machine learning for

automation

Leverage parallel computing to

speed up training

Scale to big data architectures for

processing and searching

12

13

14

15

Developing Vision Using MathWorks Tools

Continental built their

tooling using MATLAB

16


Regression

Test

C Code

Algorithm

Interactive

Exploration

Data

Regression

Test

C Code

Algorithm

Interactive

Exploration

Data

17

MathWorks Products for Vision Applications Development

Image Processing

Toolbox™

Contrast adjustment

Geometric transformations

Various filters

Segmentation

Object analysis

Computer Vision System Toolbox™

High-speed video I/O

Point Cloud processing

Tracking

Stereovision

Image Acquisition Toolbox™

Image capture from standard H/W

Analog, Camera Link, DCAM,

GigE Vision, USB camera, etc

Microsoft Kinect Support

Statistics and Machine Learning

Toolbox™

Multivariate statistics

Probability distribution

Machine learning

Experimental design

Statistical process control

18

Cascade DetectorComputer Vision System Toolbox - 4 Easy Steps to Create a Car Detector

Collect Car Image Data

(Training Images)

Label Objects of Interest

Train Object Detector

Test Detector Performance

19

Example of 3D Vision Algorithm DevelopmentComputer Vision System Toolbox

Stereo Calibration App.

Feature Detection/Extraction

Point Tracker

Feature Matching

Kd-tree based

Point Cloud

Object

Point Cloud

Visualization

Point Cloud

Matching

21

Example of Lidar Data Processing (Fitting and Removing Surface)Computer Vision System Toolbox

radar

vison

if(params.ptCloud.pcfitplane.enable)

if(params.ptCloud.pcfitplane.lowerHeight>0)

[~,~,outlierIndices] = pcfitplane(ptCloud,params.ptCloud.pcfitplane.lowerHeight,[0,0,1]);

ptCloud = select(ptCloud,outlierIndices);

end

if(params.ptCloud.pcfitplane.upperHeight>0)

[~,inlierIndices,~] = pcfitplane(ptCloud,params.ptCloud.pcfitplane.upperHeight,[0,0,1]);

ptCloud = select(ptCloud,inlierIndices);

end

end

22

Deep Learning for Object Recognition Neural Network Toolbox

Addressing Challenges in Deep Learning for Computer Vision

Challenge

Managing large sets of

labeled images

Resizing, Data augmentation

Background in neural

networks (deep learning)

Computation intensive task

(requires GPU)

Solution

imageSet or imageDataStore to

handle large sets of images

imresize, imcrop, imadjust,

imageInputLayer, etc.

Intuitive interfaces, well-documented

architectures and examples

Training supported on GPUs

No GPU expertise is required

Automate. Offload computations to a

cluster and test multiple architectures

CNN Object Detector

23

MathWorks Products for Radar Development

PA

LNA

DSP

Determination of waveforms

(range, resolution)

Phased Array System Toolbox

Signal Processing Toolbox

Instrument Control Toolbox

Algorithms for data analysis


DSP System Toolbox

Channel modeling (interference, noise)

Communications System Toolbox

Modeling of antenna arrays (no of elements, position)


Modeling of RF Impairments

(noise, non-linearity, frequency dependency)

SimRF

24


Waveform

GeneratorTransmitter

Transmit

Array

Signal

ProcessingReceiver

Receive

Array

Environment,

Targets, and

Interference

Waveforms

Pulse, LFM, FMCW, etc.

Tx Antenna Arrays

ULA, URA, etc.

Rx Antenna Arrays

ULA, URA, etc.

Beamforming, Matched

Filtering, Detection, CFAR,

STAP, etc.

Transmitter

Monostatic and Bistatic

Receiver

Monostatic and Bistatic

Environment effects,

impairments, interference

Algorithms

Simulation Tools

25

Phased Array System Toolbox Algorithms

Direction of ArrivalSpace-Time Adaptive

ProcessingDetection

28

• Calculate Ground Speed

• Object classification

• Filtering

• Offset Compensation

Data

Pre-processing

Zoning

Path

Estimation

Path

Estimation

How about Sensor fusion algorithm for FCW

Vision

Object

Radar

Object

Vision

LD

Vehicle

CAN

Sensor

Fusion

Kalman

Filter

Sensor Fusion

& Tracking

MIO: Most-Important Object

Threat

Assessment

Risk

Assessment

Maneuver

Analysis

Zoning

Find

MIOFCW

Kalman

Filter

29

• Calculate Ground Speed

• Object classification

• Filtering

• Offset Compensation

Data

Pre-processing

Zoning

Path

Estimation

Path

Estimation

Sensor fusion algorithm for FCW

Vision

Object

Radar

Object

Vision

LD

Vehicle

CAN

Sensor

Fusion

Kalman

Filter

Sensor Fusion

& Tracking

MIO: Most-Important Object

Threat

Assessment

Risk

Assessment

Maneuver

Analysis

Zoning

Find

MIOFCW

Kalman

Filter

30

Sensor Fusion

Made Easy by MATLAB CVST

Vision

Object

Radar

Object

Radar

VisionR1 R2 Rm

V1

V2

Vn

Radar

costMatrix

Assignments

V1 + R2

V2 + R1

Vn + Rm

Fusion

𝑓(𝑉1) + 𝑓(𝑅2)𝑓(𝑉2) + 𝑓(𝑅1)

𝑓(𝑉𝑛) + 𝑓(𝑅𝑚)

Fused Object List


[assignments, unassignedVisions, unassignedRadars] = ...assignDetectionsToTracks(costMatrix, param.costOfNonAssignment);

Pairs of visions and

associated radars

31

Kalman Filter

0x̂

0P

Initial state

& covariance

1ˆ

kx

1kP

Previous state

& covariance

Pk

APk1A

TQ

(1) Predict state based on physical model and previous state

(2) Predict error covariance matrix

Time Update (“Predict”)

K k Pk

H

T(HPk

H

TR)

1(1) Compute Kalman gain

(2) Update estimate state with measurement

Measurement Update (“Correct”)

?x k ?x kKk(zk H?x k

)

(3) Update the error covariance matrix

Pk (IK kH) Pk

kkk vHxz

Measurementkx̂

kP

Output of

updated state

1 kk

Current becomes previous

R : Sensor noise covariance matrix (measurement error)

K : Kalman gain

uw

][ TE wwQ

: Control variable matrix

: Process (state) noise

][T

kkk E eeP kkk xxe ˆ

: Process (state)

covariance matrix

(estimation error)

: Process noise

covariance matrix

v

H

: Measurement noise

k minimize P

A : State matrix relates the state at the

previous, k-1 to the state at the current, k

: Output matrix relates the state to the

measurement

kkkk wBuxAx

1ˆˆ

From sensor spec or experiment

32

Kalman Filter Made Easy by MATLAB CVST

0x̂

0P

Initial state

& covariance

1ˆ

kx

1kP

Previous state

& covariance Time Update (“Predict”)

Measurement Update (“Correct”) Current Measurement

Output of

updated state

1 kk

Current becomes previous

[z_pred,x_pred,P_pred] = predict(obj)

z_pred : prediction of measurementx_pred : prediction of stateP_pred : state estimation error covariance

at the next time step

[z_corr,x_corr,P_corr] = correct(obj,z)

z_corr : correction of measurementx_corr : correction of stateP_corr : state estimation error covariance

z

x_corrP_corr

Predicted state

x_pred

kalmanFilterSysObj = vision.KalmanFilter(A,H,'ProcessNoise',Q,'MeasurementNoise',R)


33

Regression

Test


Interactive

Exploration

Data

AlgorithmRegression

TestAlgorithm

Interactive

Exploration

Data

C Code

34

Create components in MATLAB and reuse them in Simulink

35

Automate regression testing with Simulink Test

Specify tests and Interact with results

in the Test Manager

Generate a report to share results

36

Generate C code for your algorithm with MATLAB Coder

Software in the loop &

Processor in the loop

enabled by

Embedded Coder

MATLAB

C

37

Enable coverage of MATLAB and C code with

Simulink Verification and Validation

Collects coverage on MATLAB code

in “Normal” mode

Collects coverage on generated C

code in “Software in the loop” mode

38

39

Prototype on hardware with Simulink Real-Timeas seen in today’s Test drive your ADAS algorithms presentation

Algorithm Models

Vehicle and

Environment

Models

Forward

Collision

Warning

Autonomous

Emergency

Braking

40

41

User Stories

Ground Detection using Backward Camera

11

Dynamic Alignment of Radar Sensor

42

Agenda

Challenges to automate the driving

Example workflow for ADAS algorithm development

– Managing & analyzing big data from vehicle fleet test

– Algorithms MATLAB/Simulink provide to support ADAS development

– Verifying and implementing ADAS algorithms using MBD

ROS Interface for Automated Driving system development

Q & A

43

Ethernet

CPU2

Main CPU

Challenges of Doing It All by Yourself

Enc

IMU

LIDAR

Kinematics& Control

Remote

machine

Your

algorithm

Image pre-

processing

Camera

Localization &

Mapping

GUI

Visual SLAM

Motor

Controller

Map server

Local

Planner

Global

Planner

You may need…

• Reliable distributed architecture

• Algorithms for autonomy

• Path planning

• Collision avoidance

• Sensor processing

• Sensor data acquisition

• Simulation environments

WiFi

44

Automated Driving at BMW

45

Potential Frameworks @ BMW

In-house SW Commercial SW Open source

46

Potential Frameworks @ BMW

In-house SW Commercial SW Open sourceIn-house SW Commercial SW Open source

47

What is ROS (Robot Operating System)?

• An architecture for distributed inter-process

communication

• Packages for common algorithms and drivers

• Multilanguage interface (C++, Python, Lua, Java

and MATLAB)

Benefit for Automated Driving

• Sensor data acquisition

• Reliable distributed architecture

• Lots of “off the shelf” algorithms for autonomy Path planning / Collision avoidance /Sensor processing

• Simplified component compatibility through

standalone interfaces

• Integration with simulation environments

48

ROS Trends in Robotics Development

http://rosindustrial.org/ric-americas/

• #1 middleware for robotics

applications development

• Popular in research and

gaining great momentum in

industry

http://rosindustrial.org/ric-americas/

49

LIDAR RADAR GPS/IMUCamera

Autonomous Car as an Advanced Robotics System

Steering Actuator

Actuator ECUs

GAS/Brake Actuator

Motion Controllers

Planning

Localization Obstacle

avoidance

Global Map

Motion controlNODE

NODE

NODE

NODE

NODE

NODE

NODE

NODE NODE NODE

ROS: communication framework

and stack of libraries

50

With MATLAB/Simulink?

Custom C

C/C++ Code

Early Idea

MATLAB Code Simulink Model

Convert to ROS Node

by Hand

Generate

Code

ROS

Need to learn ROS and Linux

Should be familiar with C++

Difficult to get started

51

With RST

Custom C

Early Idea


ROS

Workflow with Robotics System Toolbox

Without RST

Custom C

C/C++ Code

Early Idea


Convert to ROS Node

by Hand

Code

Generation

with

Interface

Merge to Simulink

Operation

connecting

with ROS

Generate

Code

ROS

Need to learn ROS and Linux

Should be familiar with C++

Difficult to get started

52

What can be done with the Robotics System Toolbox?

ROSMATLAB Code

SM Models

Built-in

algorithms

Robot

ROS node

Simulation

environment

Networking

MATLAB on PC

53

Co-simulation with ROS

54


55


ROSMATLAB Code

SM Models

Built-in

algorithms

Robot

ROS node

Simulation

environment

Networking

MATLAB on PC

56


Robot URI

Ex) 192.168.204.150

Simulator URI

Ex)192.168.204.144

57

Implementation

58


ROSMATLAB Code

SM Models

Built-in

algorithms

Robot

ROS node

Simulation

environment

Networking

Code Generation

MATLAB on PC

Generate standalone ROS node from Simulink

59

Automated Driving at BMW

60

Automated Driving with ROS at BMW

** MICHAEL AEBERHARD, BMW, ROSCON 2015

61

The MATLAB environment helps engineers …

Gain insight by visualizing and analyzing data– Plotting helps you understand sensor behavior and “give life to the data”

– Creating Apps helps you simplify the analysis process

Speed up algorithm design to implementation iterations by

generating code– Generating C code helps the algorithm and implementation stay synchronized

– Automating regression testing helps algorithm and software engineers collaborate

Have a rapid interactive environment leveraging the ROS

communication framework and stack of libraries.

62

Q & A

Thank you for your attention!

Questions?

Documents

Using Model-Based Design to develop high quality · Using Model-Based Design to develop high quality ... Example of Analysis and Re-simulation Tool 1. Gather data - Instrument Toolox