PDR for lab-forming field and its benchmarking

Takeshi Kurata12, Ryosuke Ichikari1,Tatsuro Myokan12, Masakatsu Kourogi1, Katsuhiko Kaji3, and Nobuo Kawaguchi4

1AIST, 2Univ. of Tsukuba, 3Aichi institute of technology, 4Nagoya Univ.

ISMAR 2016 Workshop: Standards for Mixed and Augmented Reality (2016/9/23)

Lab-forming Field & Field-forming Lab

• Borrowing from “Terraforming”• Lab-forming Field: Transforming a real field into a lab-like place. • Field-forming Lab: Transforming a laboratory into a field-like place.

2

Big Data on Result/Behavior/Environment for Lab-forming Field

3

Case studyin Japanese Restaurant “Ganko”

• Objectives1. (for AIST) to test the CSQCC

(Computer-Supported QCC) suites in a real service field.

2. (for the restaurant) to observe effects of process improvement planned by CSQCC.

• Place– Japanese cuisine restaurant

GANKO Ginza 4-chome (Tokyo)

• Term– 1st term

• January 12 to 18, 2011– 2nd term

• February 3 to 9, 2011

4

Dining area Course dishes

1st term(Jan. 12-18, 2011)

for observing ordinary operations

QC circlefor making improvement plans

2nd term (Feb. 3-9, 2011)for observing improved operations

5 B2

B1

Dinning Area

Kitchen

Office room

Pantry

During Discussion in CSQCC6

Trajectory of a wait staff in lunch time: 12:00-14:00

Fact: Going in and out of the kitchen/office to no small extent.Possible result: Difficulty in concentrating on guest service.Cause: Cell phone everywhere, but reservation book only in the office room.Possible improvement: e-reservation book

Dinning Area

Kitchen

Office room

T. Fukuhara, R. Tenmoku, T. Okuma, R. Ueoka, M. Takehara, and T. Kurata, "Improving Service Processes based on Visualization of Human-behavior and POS data: a Case Study in a Japanese Restaurant“, ICServ2013, pp.1-8.

PDR（Pedestrian Dead-Reckoning）Estimates velocity vector, relative altitude, and action type by measurements from a wearable sensor module.

Wearing a sensor module on waist (2D SHS (Steps and Heading Systems) PDR) Easy to wear and maintain Easy to measure data for action recognition Relatively easily apply for handheld setting compared to shoe-mounted PDR

(3D-INS (Inertial Navigation System) PDR)

7

Handheld PDR From PDR to PDRplus

10-axis sensors• Accelerometers• Magnetic sensors• Gyro sensors• Barometer

Shoe-mounted PDR

Waist-worn PDR

PDR Business Launched in Japan

Collaboration among ＮＴＴ Docomo, ZDC, AKM, and AIST.

A PDR driven Indoor navigation mobile app was released in Apr, 2015. (As of Sep, 2016,

440 areas including subways and underground shopping arcades all across

Japan)

Collaboration between MegaChips and AIST.

Sensor Hub LSI “Frizz” was released in Jan, 2015 and MegaChips plans to sell 10 million Frizz products in 2015. （Power-aware motion

coprocessor optimized for PDR)

8

• M. Kourogi and T. Kurata, Personal positioning based on walking locomotion analysis with self-contained sensors and a wearable camera, ISMAR2003, pp. 103–112, (2003).

AR by PDR + Image registration(1999-2003)

Panorama-based Annotation: IWAR1999, ISWC2001,

ISMAR2003

G

Environmental mapA

B C D

E

A

B

C

F

Input frames

Position at whicha panorama is taken

PositionDirection

235 [deg]

5 [deg]From the user’s camera

Located Orientated

9

Frontier of PDR: Walking direction estimation

10

• Tutorial: Personal Navigation with Handheld Devices by Valerie Renaudin, IPIN 2015.

Frontier of PDR: Walking direction estimation

11

• Tutorial: Personal Navigation with Handheld Devices by Valerie Renaudin, IPIN 2015.• Long Paper: Christophe Combettes, Valerie Renaudin, Comparison of Misalignment

Estimation Techniques Between Handheld Device and Walking Directions, IPIN 2015.• FIS was proposed by Kourogi and Kurata in PLANS 2014.

“Globally, the FIS method provides better results than the other two methods.”

Frequency analysis of Inertial Signals

Forward and Lateral Acc. Modeling

Principal Component Analysis

Overview: History of our PDR12

ISWC2001

IWAR1999

ISMAR2003

PLANS2014

PLANS2010 ICServ2013

Docomo map navi

Image registration + Gyro

Panorama-based annotation (Image-registration-based positioning)

Image registration + PDR

PDRplus (PDR + Action recognition)

Handheld PDR(Walking-direction estimation)

2015- 2015-

PDR module

2011-

Academia

Industry

Before PDR

ICAT2006 PDR + GPS + RFID

In the year of 2010• iPhone 4: the first popular consumer mobile device

equipped with 9-axis sensors including accelerometers, magnetic sensors, and gyro sensors

13

G-spatial EXPO 2010:Handheld PDR on iPhone 4(World’s first-ever live demo)

PLANS2010, PLANS2014

Global Trend on PDRPDR R&D players have rapidly indicated their presence all over the world on and after 2010.

Movea (France)

Sensor Platforms (USA)

CSR (UK)

TRX (USA)

Trusted Positioning (Canada)

14

Acquired by QualcommAcquired by InvenSenseAcquired by InvenSense

Acquired by Audience

Indoo.rs (USA)

SFO

Standardization on PDR Benchmarking• PDR related R&D is highly active worldwide: Necessity for sharing

common measures.• Description of the performance should be unified in spec sheets

and scientific papers.• Different measures from absolute positioning methods such as

GNSS, Wi-Fi, and BLE are required for PDR, which is a method of relative positioning.

• PDR Benchmark Standardization Committee was established in 2014 as a platform of the grassroots activity.

15

https://www.facebook.com/pdr.bms

Support Organizations• Asahi Kasei Corporation, Asia Air Survey Co., Ltd. (Y. Minami), INTEC Inc.,

MTI Ltd., KDDI R&D Laboratories, Inc., KOKUSAI KOGYO CO., LTD.,SHIBUYA KOGYO CO., LTD., Koozyt, Inc., SITESENSING, inc., Sharp Corporation, Sugihara Software and Electron Industry Co., Ltd. (SSEI), Information Services International-Dentsu, Ltd. (ISID), Hitachi, Ltd.,Frameworx, Inc. (S. Watanabe), MULTISOUP CO.,LTD., Milldea, LLC, Murata Manufacturing Co., Ltd., MegaChips Corporation, Recruit Lifestyle Co., Ltd. (K. Ushida), RICOH COMPANY, LTD., Rei-Frontier Inc.,

• Aichi Institute of Technology (K. Kaji), Akashi National College of Technology (I. Arai), Kanagawa Institute of Technology (H. Tanaka), Keio University (S. Haruyama, N. Kohtake, M. Nakajima), University of Tsukuba (T. Kurata), Tokyo Institute of Technology (S. Okada), Nagoya University (N. Kawaguchi), Niigata University (H. Makino), Ritsumeikan University (N. Nishio), National Institute of Advanced Industrial Science and Technology (AIST) (T. Kurata, M. Kourogi), Human Activity Sensing Consortium (HASC), Location Information Service Research Agency (LISRA)

• 33 organizations in Japan as of April, 2016

16

Activity Examples in Japan17

1st meeting of the committee (2014/05/22)

1st seminar in LBJ (2014/06/11)

2nd meeting of the committee (2015/05/19)

2nd seminar in LBJ (2015/06/13)

Organized session in HCG symposium (2014/12/17)

3rd seminar in LBJ (2016/06/10) LBJ: Location Business JapanHCG: Human Communication Group

Organized session in HCG symposium (2015/12/16)

3rd meeting of the committee (2015/05/19)

Scene in data collection19

Multi-Algorithm On-Site Evaluation System• Evaluates the accuracy of each PDR algorithm automatically as often

as sensor data is uploaded to the server• Provides trajectory images so that participants can compare their PDR• algorithms in real time.

20

http://pdrsv.hasc.jpK. Kaji, K. Kanagu, K. Murao, N. Nishio, K. Urano, H. Iida, N. Kawaguch, Multi-Algorithm On-Site Evaluation System for PDR Challenge, ICMU2016, (to appear)

UbiComp/ISWC 2015 PDR Challenge Corpus• Is now open to the public. (http://hub.hasc.jp/)

21

Routes 5

Devices 7

Subjects 93

# of pedestrian sensing data 241

# of pedestrian sensing data with calibration data 230

# of pedestrian sensing data with LIDAR data 10

Avg. of walking time [sec] 101

Avg. of moving distance [m] 115

Avg. of angular change [°] 606

K. Kaji, M. Abe, W. Wang, K. Hiroi, and N. Kawaguchi, UbiComp/ISWC 2015 PDR challenge corpus, HASCA2016 (UbiComp2016 Proceedings: Adjunct), pp.696-704

Statistics of the corpus

Detailed route statistics of pedestrian sensing data with calibration data

Open Data Contest in Logistics &PDR Challenge in Warehouse

• Open data contest in logistics by Frameworx– Submission: 2016/4/18-2016/7/18– Award ceremony: 2016/9/12

• PDR Challenge in Warehouse– Now planning– Will be held as a Japanese

domestic contest in 2016– Will be held as an international

contest in IPIN 2017

22

PDR Challenge Series• Ubicomp/ISWC 2015 PDR Challenge

– Scenario: Indoor Navigation– On-site– Continuous walking while keeping watching the navigation

screen by holding the smartphone– Several minutes per trial

• 2016/2017 PDR Challenge in Warehouse (tentative)– Scenario: Picking work in a warehouse– Off-site– Not only walking but various actions including picking and

carrying– Several hours per trial– Will be held in IPIN 2017

23

Examples of picking workers’ trajectories estimated by PDR + WMS

(Warehouse Management System)

24

How to design benchmark Indicators?

• Other aspects to be considered– Reliability: Different measures from absolute positioning methods

are required for PDR– Efficiency: Power consumption– Repeatability: Temperature Hysteresis, Magnetic field, etc.– Representativeness: How to hold, Route shape, etc.

25

Benchmark indicators of vision-based spatial registration and tracking for MAR (ISO/IEC WD 18520)

How to compare and visualize?26

Easy Difficult

Method 1

Easy Difficult

Method 2

How to compare and visualize?27

Easy Difficult

Met

hod

1M

etho

d 2

Competitions: IPIN and the others(cf. EvAAL presentation in IPIN 2105 etc.)

28

IPIN year EvAAL, IPSN, UbiComp/ISWC

Zurich, Switzerland 2010 universAAL is launchedGuimaraes, Portugal 2011 EvAAL: indoor localization

Sidney, Australia 2012 EvAAL: + activity recognitionMontbeliard, France 2013 EvAAL: same as 2012

Busan, Korea1st IPIN competition 2014 EvAAL: 3 floors, smartphone

IPSN: infrastruc. based + free

Banff, CanadaEvAAL-ETRI comp. 2015

EvAAL-ETRI: 6 floors, on/off-siteIPSN: infrastruc. based + free

UbiComp/ISWC: 2 floors, smartphone PDR, 90 subjects

Madrid, SpainIndoor Localization

Competition2016 IPIN: smartphone (on/off-site), PDR, Robot

IPSN: infrastruc. based + free, 2D/3D

IPIN2017

29

ISMAR2015

Thank you!• AIST is now hiring for Tenure-track, Postdoc, and

RA (PhD) positions at Tsukuba, Japan.• Target research fields are ↓ ↓ ↓

30