Learn from IoT: Pedestrian Detection and Intention ...gsolmaz/publications/MobiCom19... · Smartphone to vehicle delay ~0.6sec From smartphone to cloud and lastly in-vehicle IoT platform

Learn from IoT: Pedestrian Detection and Intention Prediction for Autonomous Driving

Gürkan Solmaz, Everton Luis Berz, Marzieh Farahani Dolatabadi*, Samet Aytac, Jonathan Fürst, Bin Cheng, Jos den Ouden*

IoT Research Group

NEC Laboratories Europe

* Eindhoven University of Technology (TU/e)

1st ACM Workshop on Emerging Smart Technologies and Infrastructures for Smart Mobility and Sustainability (SMAS), ACM MobiCom 2019

Los Cabos, Mexico, October 21, 2019

This work has received funding from the European Union’s Horizon 2020 research and innovation programme within the project AUTOPILOT (Automated Driving Progressed by Internet Of Things) under the grant agreement No 731993. Responsibility for the information and views set out in this document lies entirely with the authors.

2 © NEC Laboratories Europe GmbH 2019

Motivation

▌Why “learn from IoT”?Uber accident (2018): Both car sensors

and driver can easily fail

Mobile sensing, vast amount of sensors, RSUs, V2V, V2X communication, 5G, edge computing

IoT can support traffic safety

▌Major advancements in ML due to increased availability of data, computation, and software TensorFlow, PyTorch

▌IoT expansion to many domainsSmart cities and smart mobility

IoT platforms: Azure, AWS IoT

Large-scale tesbeds

▌Aim: Combine IoT and ML for enhanced AD safety


Proposed approach

▌ System learning from data collected from

1) in-vehicle sensors (e.g., cameras)

2) external IoT sources such as mobile devices of VRUs

▌ Components included in the system setup:

1) autonomous car

2) mobile devices with the mobility app

3) in-vehicle IoT platform

4) VRU detection and intention prediction components

5) cloud IoT platform

▌ ML leverages data sources for training

Data collected by the vehicle at the TU/e campus

Smartphone apps of TU/e and NEC for pedestrian movement tracking

Mobile ITS-G5 location devices

▌ We propose methods for accurate VRU detection and pedestrian intention prediction

1) creating a world model (WM) by combining the vehicle and mobile device data and performing VRU detection and localization

2) using the latest trajectory data (in-vehicle or mobile device) to predict the pedestrians’ intended movement steps based on historical measurements

▌ Petri Net models to combine VRU detection and intention prediction

Decide under which conditions which inputs can be combined for the autonomous decision-making


VRU detection through WM

▌WM contains the vehicle itself and objects around

▌The formalism adopted is WIRE where the

WM aims to track semantic objects such as VRUs http://wiki.ros.org/wire

(World Information for Robot Environments framework by TU/e)

▌Multiple Hypothesis Tracker (MHT) algorithm [2] combines evidences to a common world representation dynamically

▌Objects’ attributes, classification, and prior knowledge are associated in the hypotheses treeEvery hypothesis contains a list of anchors and has a correctness probability

Each anchor contains an individual symbol, a set of measurements and a probabilistic signature that consists out of a mixture of probability density functions generated by a set of behavior models

The predicate attribute space represents predicate grounding relations that link attribute values and predicate symbols [2]

http://wiki.ros.org/wire


Pedestrian intention prediction method

▌This method predicts the next location of pedestrians based on historical data and the current position

▌ML model uses the mobile device data

Speed of pedestrian

GPS trajectory values

▌The model predicts nf future locations

▌The representation of pedestrian trajectory is inspired by the model in [8]

the path modeling with an

example pedestrian

trajectory during the TU/e

experiments


Intention prediction ML model

▌ Three input features, three concatenation layers, and nf output features

▌ Adam algorithm [5] for the optimization process with ReLU activation function

▌ Embed encoder: Map the inputs into vectors and then forward to the (intermediary) concatenation layers

▌ Intermediary layers: Concatenate all outputs of the feature encoders and pass the concatenation through fully connected layers

▌ The data is randomly partitioned into training (70%), validation (20%), and testing (10%) subsets.

We defined np = 10 and nf = 5

Approximately equivalent to 11m

and 5.5m in a straight-walking

distance


Combining VRU detection and intention prediction

▌ Combining the inputs using stochastic priority Petri Nets from the two previous parts to support the autonomous decision-making

States/places: big circles, transitions: rectangles

Probabilistic var. for transitions: λ

Priorities: Curly-braced numbers

# tokens: Numbers w/o curly braces

▌ The model has two types of VRUs

1. users of mobile devices and our app (w/ IoT data)

2. people w/o the app

▌ Three possible cases exist during the objects come to vicinity of each other

1) both the mobile device and vehicle sensor data,

2) only mobile device data

3) only vehicle sensor (camera) data available in the in-vehicle platform


TU/e experiments

▌ We conduct the pilot tests mainly at the TU/e campus with 2km road network, speed limit 15km/h

▌ Three pilot tests: Each ~1-2 weeks long 21 controlled experiments (total of 70 runs)

Predefined pedestrian movements

Autonomous driving behaviors are mostly uncontrolled

▌ Custom-built autonomous car prototype (Toyota Prius)

▌ The car has an ITS-G5 device connected to the in-vehicle IoT platform Two pedestrians also carry mobile ITS-G5 devices

▌ IoT gateway connects the in-vehicle platform with two cloud IoT plaftormsvia MQTT and HTTP using cellular 4G

▌ Robot operating system (ROS) operates on the in-vehicle IoT platform


Message delay and WM

▌Smartphone to vehicle delay ~0.6secFrom smartphone to cloud and lastly in-vehicle IoT platform (ROS timestamps)

▌In the WM, vehicle and mobile devices together has more consistency compared to only vehicleVehicle receives the global location of a pedestrian a few seconds before the

camera detection


Intention detection results

▌ Ludwig and TensorFlow to train and validate the model

▌ Each step is a consecutive data point in the range of [0.5,1] sec Most predictions for 1st step is <1m

▌ Controlled scenarios: Walking-straight has the best accuracy Max error of 1.9m in the furthest prediction step

▌ 18% and 48% decrease in the error compared to the NARX and Dead Reckoning approaches 0.2m better on average than [8]


Conclusion & future work

▌We propose learning from IoT data to improve the safety of autonomous driving

Key problems: pedestrian detection and pedestrian intention prediction

▌The proposed system can complement existing safety solutions

▌Future work:

extracting features from other data sources such as OpenStreetMap and traffic operation center data (e.g., traffic lights)

Old problems also still exist


This work has received funding from the European Union’s Horizon 2020 research and innovation programme within the project AUTOPILOT (Automated Driving Progressed by Internet Of Things) under the grant agreement No 731993. Responsibility for the information and views set out in this document lies entirely with the authors.

Questions?

[email protected]

mailto:[email protected]

Documents

Learn from IoT: Pedestrian Detection and Intention ...gsolmaz/publications/MobiCom19... · Smartphone to vehicle delay ~0.6sec From smartphone to cloud and lastly in-vehicle IoT platform