PDR Challenge in Warehouse Picking and Virtual Mapping Party

国立研究開発法人

PDR Challenge in Warehouse Picking and Virtual Mapping Party

Takeshi Kurata12

1AIST, 2Univ. of Tsukuba


PDR（Pedestrian Dead-Reckoning）Estimates velocity vector, relative altitude, and action type by measurements from a wearable sensor module.

Wearing a sensor module on waist (2D SHS (Steps and Heading Systems) PDR) Easy to wear and maintain Easy to measure data for action recognition Relatively easily apply for handheld setting compared to shoe-mounted PDR

(3D-INS (Inertial Navigation System) PDR)

2Handheld PDR From PDR to PDRplus

10-axis sensors• Accelerometers• Magnetic sensors• Gyro sensors• Barometer

Shoe-mounted PDR

Waist-worn PDR


ISMAR2003 Image registration + PDR

PLANS2010 PDRplus (PDR + Action recognition)

PLANS2014 Handheld PDR (Walking Direction Estimation)

2015- 2015-

PDR Module

ICAT2006 PDR + GPS + RFIDG-spatial EXPO2010 PDR demo on iPhone 4

ICServ2013 Best Paper: Computer-Supported QC Circle (CSQCC) with PDR

IPIN2015, 2016 10-axes PDR, Time-series sub-optimization in SDF

2011-

Docomo map navi (As of Sep, 2016, 480 areas including subways and underground shopping arcades in Japan)

2014-

産総研技術移転ベンチャー

PDR Benchmark Standardization Committee

Established in 2014 (Chair: T. Kurata, 33 organizations in Japan as of April, 2016)

Overview: History of PDR in AIST

AIST Start-up

Acad

emia

Indu

stry

PDR: Pedestrian Dead ReckoningSDF: Sensor Data Fusion (Hybrid Positioning)RFID: Radio Frequency IdentifierGPS: Global Positioning System


AR by PDR + Image registration (1999-2003)

Panorama-based Annotation: IWAR1999, ISWC2001,

ISMAR2003

G

Environmental mapA

B C D

E

A

B

C

F

Input frames

Position at whicha panorama is taken

PositionDirection

235 [deg]

5 [deg]From the user’s camera

Located Orientated

4





2015- 2015-

PDR Module




2011-


2014-





AIST Start-up

Acad

emia

Indu

stry



In the year of 2010• iPhone 4: the first popular consumer mobile device equipped

with 9-axis sensors including accelerometers, magnetic sensors, and gyro sensors

6

G-spatial EXPO 2010:Handheld PDR on iPhone 4(Worldʼs first-ever live demo)

PLANS2010, PLANS2014





2015- 2015-

PDR Module




2011-


2014-





AIST Start-up

Acad

emia

Indu

stry



Frontier of PDR: Walking direction estimation

8• Tutorial: Personal Navigation with Handheld Devices by Valerie Renaudin, IPIN 2015.


Frontier of PDR: Walking direction estimation

9

• Tutorial: Personal Navigation with Handheld Devices by Valerie Renaudin, IPIN 2015.• Long Paper: Christophe Combettes, Valerie Renaudin, Comparison of Misalignment

Estimation Techniques Between Handheld Device and Walking Directions, IPIN 2015.• FIS was proposed by Kourogi and Kurata in PLANS 2014.

“Globally, the FIS method provides better results than the other two methods.” by IFSTTAR

Frequency analysis of Inertial Signals

Forward and Lateral Acc. Modeling

Principal Component Analysis





2015- 2015-

PDR Module




2011-


2014-





AIST Start-up

Acad

emia

Indu

stry



Global Trend on PDRPDR R&D players have rapidly indicated their presence all over the world on and after 2010.

Movea (France)

Sensor Platforms (USA)

CSR (UK)

TRX (USA)

Trusted Positioning (Canada)

11

Acquired by QualcommAcquired by InvenSenseAcquired by InvenSense

Acquired by Audience

Indoo.rs (USA)

SFO

Acquired by TDK?





2015- 2015-

PDR Module




2011-


2014-





AIST Start-up

Acad

emia

Indu

stry



Standardization on PDR Benchmarking• PDR related R&D is highly active worldwide: Necessity for sharing

common measures.• Description of the performance should be unified in spec sheets

and scientific papers.• Different measures from absolute positioning methods such as

GNSS, Wi-Fi, and BLE are required for PDR, which is a method of relative positioning.

• PDR Benchmark Standardization Committee was established in 2014 as a platform of the grassroots activity.

13https://www.facebook.com/pdr.bms



Scene in data collection

15


Multi-Algorithm On-Site Evaluation System• Evaluates the accuracy of each PDR algorithm automatically as often

as sensor data is uploaded to the server• Provides trajectory images so that participants can compare their PDR• algorithms in real time.

16

http://pdrsv.hasc.jpK. Kaji, K. Kanagu, K. Murao, N. Nishio, K. Urano, H. Iida, N. Kawaguch, Multi-Algorithm On-Site Evaluation System for PDR Challenge, ICMU2016.


UbiComp/ISWC 2015 PDR Challenge Corpus• Is now open to the public. (http://hub.hasc.jp/)

17

Routes 5

Devices 7

Subjects 93

# of pedestrian sensing data 241

# of pedestrian sensing data with calibration data 230

# of pedestrian sensing data with LIDAR data 10

Avg. of walking time [sec] 101

Avg. of moving distance [m] 115

Avg. of angular change [°] 606

K. Kaji, M. Abe, W. Wang, K. Hiroi, and N. Kawaguchi, UbiComp/ISWC 2015 PDR challenge corpus, HASCA2016 (UbiComp2016 Proceedings: Adjunct), pp.696-704

Statistics of the corpus

Detailed route statistics of pedestrian sensing data with calibration data


Open Data Contest in Logistics &PDR Challenge in Warehouse Picking

• Open data contest in logistics by Frameworx– Submission: 2016/4/18-

2016/7/18– Award ceremony: 2016/9/12

• PDR Challenge in Warehouse Picking– Will be held as an international

contest in IPIN 2017

18


PDR Challenge Series• Ubicomp/ISWC 2015 PDR Challenge

– Scenario: Indoor Navigation– On-site– Continuous walking while keeping watching the navigation

screen by holding the smartphone– Several minutes per trial

• 2017 PDR Challenge in Warehouse Picking– Scenario: Picking work in a warehouse– Off-site– Not only walking but various actions including picking and

carrying– Several hours per trial– Will be held in IPIN 2017

19


Examples of picking workersʼ trajectories estimated by PDR + WMS (Warehouse Management System)

20


How to design benchmark Indicators?

• Other aspects to be considered– Reliability: Different measures from absolute positioning

methods are required for PDR– Efficiency: Power consumption– Repeatability: Temperature Hysteresis, Magnetic field, etc.– Representativeness: How to hold, Route shape, etc.

21

Benchmark indicators of vision-based spatial registration and tracking for MAR (ISO/IEC WD 18520) (PEVO: Projection Error of Virtual Objects)


How to compare and visualize?

22

Easy Difficult

Method 1

Easy Difficult

Method 2


How to compare and visualize?

23

Easy Difficult

Met

hod

1M

etho

d 2


Competitions: IPIN and the others(cf. EvAAL presentation in IPIN 2105 etc.)

24

IPIN year EvAAL, IPSN, UbiComp/ISWC

Zurich, Switzerland 2010 universAAL is launchedGuimaraes, Portugal 2011 EvAAL: indoor localization

Sidney, Australia 2012 EvAAL: + activity recognitionMontbeliard, France 2013 EvAAL: same as 2012

Busan, Korea1st IPIN competition 2014 EvAAL: 3 floors, smartphone

IPSN: infrastruc. based + free

Banff, CanadaEvAAL-ETRI comp. 2015

EvAAL-ETRI: 6 floors, on/off-siteIPSN: infrastruc. based + free

UbiComp/ISWC: 2 floors, smartphone PDR, 90 subjects

Madrid, SpainIndoor Localization

Competition2016 IPIN: smartphone (on/off-site), PDR, Robot

IPSN: infrastruc. based + free, 2D/3D


IPIN2017

25


Virtual Mapping Party

26

which enables the participants to contribute to the accessible information collection for visually impaired people from anywhere and at any time.


Characteristics of each mapping work

27

Type of activities location Time Remarks

Conventionalmapping party On-site Sync.

Face to face communicationDeep understanding of real conditionsMandatory skill for organizing events Up to weather

Mapping party utilizing smartphones app.

On-site Any time(Async.)

Mapping while commutingEasy to contributeDeep understanding of real conditions Position of contents depending on localization methods

Mapping partyutilizing crowdsourcingimage sharing service

Anywhere(Off-site)

Any time(Async.)

CrowdsourcingRemote mappingEasy to contribute anytime and anywhereDepend on shared dataLimited understanding of real conditions

Virtual mapping party

Anywhere(Off-site)

Any time(Async.)

CrowdsourcingRemote mappingEasy to contribute anytime and anywhere Easy to measure contentsʼ positionEasy to verify registered contentsMandatory pre-recording


How to decide POI/POR position

28

Using intersection of line of sight and

the ground

Triangulating with plural panoramas.

POI: Point Of Interest (Landmark such as Store, restaurant, hospital, facilities, etc.)POR: Point Of Reference (specific point location the existence of which is easily recognized for confirming routes such as characteristic shape and material of ground (steps, stairs, sloop, door), sound/noise, and scent/odor.)


Screenshots for the virtual mapping interface.

29


Desktop vs. Smartphone VR

30

• The number of registered POI/PORs in WSs held at Miraikan : 598 (42 participant, 6 one-hour Workshops)

• Sense of Immersion: Desktop << Cardboard HMD • Registration efficiency: Desktop/Smartphone=1.43

ＰＯＩ/POR/Request on OSM


Feedback from WS participants

31

Categories of feedback Positive feedback Negative feedback/ Suggestions

About VR experience with omnidirectionalimages/movies and

3D sound

I like the function for 3D sounds.3D sounds seem to be very useful, since visually impaired people can confirm amount of traffic on roads.I could more realistically experience the VR scene by omnidirectional movies than still omnidirectional images.

Estimating direction of sound sources was difficult.The quality of the images/movies was not perfect.

About devices for VR experience

I like instant HMD, since we can experience VR with what I have.I like VR experience with Oculus VR HMD since I can realistically experience by movies.I like Samsung's Gear VR HMD, because the image quality looked good and it was confortable for wearing.

It took a while to get used to HMD, and I got tired when I wore HMD.Mapping with PC is better in terms of degree of fatigue.An instant HMD was not so comfortable for wearing.I thought wearing HMD on glass was difficult.

About user interfaces I like the function for pointing in first personʼs view not map view.

PC is the easiest platform for inputting POR/POI.

About AR Tactile map I like the function for sending request by visually impaired people.

The accuracy of gesture recognition for AR tactile map needs to be improved.

About POR/POIsThere are so many POR/POIs in the display. I think it becomes more clear if the displayed contents are limited to nearby contents.I found empty POR/POIs without detailed information. I think filtering of the registered contents are required.

Other suggestionsI wondered if the system could support communication between participants.I would like to regularly contribute virtual mapping parties from my home.It was the most beneficial application of VR I have ever experienced.


Usage of the AR tactile map for virtual mapping party

• Allowing the visually-impaired people to join the mapping party by gesture – Search: Confirming POR/POI on the tactile map with

sound for telling existence of POR/POI where user touches

– Tap: Confirming POR/POIwith Text-to-Sound whenuser taps the specific point

– Double Tap: Requestingthe POR/POI registration forspecific points of the map


AR tactile maps with HP Sprout


Automatic identification and tracking of tactile maps

34

• ORB Feature point detector/local feature descriptor is used for identifying tactile map with RGB image

• Estimating homography matrix between rectified image templates and input image

Engineering

PDR Challenge in Warehouse Picking and Virtual Mapping Party