国立研究開発法人
PDR Challenge in Warehouse Picking and Virtual Mapping Party
Takeshi Kurata12
1AIST, 2Univ. of Tsukuba
国立研究開発法人
PDR(Pedestrian Dead-Reckoning)Estimates velocity vector, relative altitude, and action type by measurements from a wearable sensor module.
Wearing a sensor module on waist (2D SHS (Steps and Heading Systems) PDR) Easy to wear and maintain Easy to measure data for action recognition Relatively easily apply for handheld setting compared to shoe-mounted PDR
(3D-INS (Inertial Navigation System) PDR)
2Handheld PDR From PDR to PDRplus
10-axis sensors• Accelerometers• Magnetic sensors• Gyro sensors• Barometer
Shoe-mounted PDR
Waist-worn PDR
国立研究開発法人
ISMAR2003 Image registration + PDR
PLANS2010 PDRplus (PDR + Action recognition)
PLANS2014 Handheld PDR (Walking Direction Estimation)
2015- 2015-
PDR Module
ICAT2006 PDR + GPS + RFIDG-spatial EXPO2010 PDR demo on iPhone 4
ICServ2013 Best Paper: Computer-Supported QC Circle (CSQCC) with PDR
IPIN2015, 2016 10-axes PDR, Time-series sub-optimization in SDF
2011-
Docomo map navi (As of Sep, 2016, 480 areas including subways and underground shopping arcades in Japan)
2014-
産総研技術移転ベンチャー
PDR Benchmark Standardization Committee
Established in 2014 (Chair: T. Kurata, 33 organizations in Japan as of April, 2016)
Overview: History of PDR in AIST
AIST Start-up
Acad
emia
Indu
stry
PDR: Pedestrian Dead ReckoningSDF: Sensor Data Fusion (Hybrid Positioning)RFID: Radio Frequency IdentifierGPS: Global Positioning System
国立研究開発法人
AR by PDR + Image registration (1999-2003)
Panorama-based Annotation: IWAR1999, ISWC2001,
ISMAR2003
G
Environmental mapA
B C D
E
A
B
C
F
Input frames
Position at whicha panorama is taken
PositionDirection
235 [deg]
5 [deg]From the user’s camera
Located Orientated
4
国立研究開発法人
ISMAR2003 Image registration + PDR
PLANS2010 PDRplus (PDR + Action recognition)
PLANS2014 Handheld PDR (Walking Direction Estimation)
2015- 2015-
PDR Module
ICAT2006 PDR + GPS + RFIDG-spatial EXPO2010 PDR demo on iPhone 4
ICServ2013 Best Paper: Computer-Supported QC Circle (CSQCC) with PDR
IPIN2015, 2016 10-axes PDR, Time-series sub-optimization in SDF
2011-
Docomo map navi (As of Sep, 2016, 480 areas including subways and underground shopping arcades in Japan)
2014-
産総研技術移転ベンチャー
PDR Benchmark Standardization Committee
Established in 2014 (Chair: T. Kurata, 33 organizations in Japan as of April, 2016)
Overview: History of PDR in AIST
AIST Start-up
Acad
emia
Indu
stry
PDR: Pedestrian Dead ReckoningSDF: Sensor Data Fusion (Hybrid Positioning)RFID: Radio Frequency IdentifierGPS: Global Positioning System
国立研究開発法人
In the year of 2010• iPhone 4: the first popular consumer mobile device equipped
with 9-axis sensors including accelerometers, magnetic sensors, and gyro sensors
6
G-spatial EXPO 2010:Handheld PDR on iPhone 4(Worldʼs first-ever live demo)
PLANS2010, PLANS2014
国立研究開発法人
ISMAR2003 Image registration + PDR
PLANS2010 PDRplus (PDR + Action recognition)
PLANS2014 Handheld PDR (Walking Direction Estimation)
2015- 2015-
PDR Module
ICAT2006 PDR + GPS + RFIDG-spatial EXPO2010 PDR demo on iPhone 4
ICServ2013 Best Paper: Computer-Supported QC Circle (CSQCC) with PDR
IPIN2015, 2016 10-axes PDR, Time-series sub-optimization in SDF
2011-
Docomo map navi (As of Sep, 2016, 480 areas including subways and underground shopping arcades in Japan)
2014-
産総研技術移転ベンチャー
PDR Benchmark Standardization Committee
Established in 2014 (Chair: T. Kurata, 33 organizations in Japan as of April, 2016)
Overview: History of PDR in AIST
AIST Start-up
Acad
emia
Indu
stry
PDR: Pedestrian Dead ReckoningSDF: Sensor Data Fusion (Hybrid Positioning)RFID: Radio Frequency IdentifierGPS: Global Positioning System
国立研究開発法人
Frontier of PDR: Walking direction estimation
8• Tutorial: Personal Navigation with Handheld Devices by Valerie Renaudin, IPIN 2015.
国立研究開発法人
Frontier of PDR: Walking direction estimation
9
• Tutorial: Personal Navigation with Handheld Devices by Valerie Renaudin, IPIN 2015.• Long Paper: Christophe Combettes, Valerie Renaudin, Comparison of Misalignment
Estimation Techniques Between Handheld Device and Walking Directions, IPIN 2015.• FIS was proposed by Kourogi and Kurata in PLANS 2014.
“Globally, the FIS method provides better results than the other two methods.” by IFSTTAR
Frequency analysis of Inertial Signals
Forward and Lateral Acc. Modeling
Principal Component Analysis
国立研究開発法人
ISMAR2003 Image registration + PDR
PLANS2010 PDRplus (PDR + Action recognition)
PLANS2014 Handheld PDR (Walking Direction Estimation)
2015- 2015-
PDR Module
ICAT2006 PDR + GPS + RFIDG-spatial EXPO2010 PDR demo on iPhone 4
ICServ2013 Best Paper: Computer-Supported QC Circle (CSQCC) with PDR
IPIN2015, 2016 10-axes PDR, Time-series sub-optimization in SDF
2011-
Docomo map navi (As of Sep, 2016, 480 areas including subways and underground shopping arcades in Japan)
2014-
産総研技術移転ベンチャー
PDR Benchmark Standardization Committee
Established in 2014 (Chair: T. Kurata, 33 organizations in Japan as of April, 2016)
Overview: History of PDR in AIST
AIST Start-up
Acad
emia
Indu
stry
PDR: Pedestrian Dead ReckoningSDF: Sensor Data Fusion (Hybrid Positioning)RFID: Radio Frequency IdentifierGPS: Global Positioning System
国立研究開発法人
Global Trend on PDRPDR R&D players have rapidly indicated their presence all over the world on and after 2010.
Movea (France)
Sensor Platforms (USA)
CSR (UK)
TRX (USA)
Trusted Positioning (Canada)
11
Acquired by QualcommAcquired by InvenSenseAcquired by InvenSense
Acquired by Audience
Indoo.rs (USA)
SFO
Acquired by TDK?
国立研究開発法人
ISMAR2003 Image registration + PDR
PLANS2010 PDRplus (PDR + Action recognition)
PLANS2014 Handheld PDR (Walking Direction Estimation)
2015- 2015-
PDR Module
ICAT2006 PDR + GPS + RFIDG-spatial EXPO2010 PDR demo on iPhone 4
ICServ2013 Best Paper: Computer-Supported QC Circle (CSQCC) with PDR
IPIN2015, 2016 10-axes PDR, Time-series sub-optimization in SDF
2011-
Docomo map navi (As of Sep, 2016, 480 areas including subways and underground shopping arcades in Japan)
2014-
産総研技術移転ベンチャー
PDR Benchmark Standardization Committee
Established in 2014 (Chair: T. Kurata, 33 organizations in Japan as of April, 2016)
Overview: History of PDR in AIST
AIST Start-up
Acad
emia
Indu
stry
PDR: Pedestrian Dead ReckoningSDF: Sensor Data Fusion (Hybrid Positioning)RFID: Radio Frequency IdentifierGPS: Global Positioning System
国立研究開発法人
Standardization on PDR Benchmarking• PDR related R&D is highly active worldwide: Necessity for sharing
common measures.• Description of the performance should be unified in spec sheets
and scientific papers.• Different measures from absolute positioning methods such as
GNSS, Wi-Fi, and BLE are required for PDR, which is a method of relative positioning.
• PDR Benchmark Standardization Committee was established in 2014 as a platform of the grassroots activity.
13https://www.facebook.com/pdr.bms
国立研究開発法人
国立研究開発法人
Scene in data collection
15
国立研究開発法人
Multi-Algorithm On-Site Evaluation System• Evaluates the accuracy of each PDR algorithm automatically as often
as sensor data is uploaded to the server• Provides trajectory images so that participants can compare their PDR• algorithms in real time.
16
http://pdrsv.hasc.jpK. Kaji, K. Kanagu, K. Murao, N. Nishio, K. Urano, H. Iida, N. Kawaguch, Multi-Algorithm On-Site Evaluation System for PDR Challenge, ICMU2016.
国立研究開発法人
UbiComp/ISWC 2015 PDR Challenge Corpus• Is now open to the public. (http://hub.hasc.jp/)
17
Routes 5
Devices 7
Subjects 93
# of pedestrian sensing data 241
# of pedestrian sensing data with calibration data 230
# of pedestrian sensing data with LIDAR data 10
Avg. of walking time [sec] 101
Avg. of moving distance [m] 115
Avg. of angular change [°] 606
K. Kaji, M. Abe, W. Wang, K. Hiroi, and N. Kawaguchi, UbiComp/ISWC 2015 PDR challenge corpus, HASCA2016 (UbiComp2016 Proceedings: Adjunct), pp.696-704
Statistics of the corpus
Detailed route statistics of pedestrian sensing data with calibration data
国立研究開発法人
Open Data Contest in Logistics &PDR Challenge in Warehouse Picking
• Open data contest in logistics by Frameworx– Submission: 2016/4/18-
2016/7/18– Award ceremony: 2016/9/12
• PDR Challenge in Warehouse Picking– Will be held as an international
contest in IPIN 2017
18
国立研究開発法人
PDR Challenge Series• Ubicomp/ISWC 2015 PDR Challenge
– Scenario: Indoor Navigation– On-site– Continuous walking while keeping watching the navigation
screen by holding the smartphone– Several minutes per trial
• 2017 PDR Challenge in Warehouse Picking– Scenario: Picking work in a warehouse– Off-site– Not only walking but various actions including picking and
carrying– Several hours per trial– Will be held in IPIN 2017
19
国立研究開発法人
Examples of picking workersʼ trajectories estimated by PDR + WMS (Warehouse Management System)
20
国立研究開発法人
How to design benchmark Indicators?
• Other aspects to be considered– Reliability: Different measures from absolute positioning
methods are required for PDR– Efficiency: Power consumption– Repeatability: Temperature Hysteresis, Magnetic field, etc.– Representativeness: How to hold, Route shape, etc.
21
Benchmark indicators of vision-based spatial registration and tracking for MAR (ISO/IEC WD 18520) (PEVO: Projection Error of Virtual Objects)
国立研究開発法人
How to compare and visualize?
22
Easy Difficult
Method 1
Easy Difficult
Method 2
国立研究開発法人
How to compare and visualize?
23
Easy Difficult
Met
hod
1M
etho
d 2
国立研究開発法人
Competitions: IPIN and the others(cf. EvAAL presentation in IPIN 2105 etc.)
24
IPIN year EvAAL, IPSN, UbiComp/ISWC
Zurich, Switzerland 2010 universAAL is launchedGuimaraes, Portugal 2011 EvAAL: indoor localization
Sidney, Australia 2012 EvAAL: + activity recognitionMontbeliard, France 2013 EvAAL: same as 2012
Busan, Korea1st IPIN competition 2014 EvAAL: 3 floors, smartphone
IPSN: infrastruc. based + free
Banff, CanadaEvAAL-ETRI comp. 2015
EvAAL-ETRI: 6 floors, on/off-siteIPSN: infrastruc. based + free
UbiComp/ISWC: 2 floors, smartphone PDR, 90 subjects
Madrid, SpainIndoor Localization
Competition2016 IPIN: smartphone (on/off-site), PDR, Robot
IPSN: infrastruc. based + free, 2D/3D
国立研究開発法人
IPIN2017
25
国立研究開発法人
Virtual Mapping Party
26
which enables the participants to contribute to the accessible information collection for visually impaired people from anywhere and at any time.
国立研究開発法人
Characteristics of each mapping work
27
Type of activities location Time Remarks
Conventionalmapping party On-site Sync.
Face to face communicationDeep understanding of real conditionsMandatory skill for organizing events Up to weather
Mapping party utilizing smartphones app.
On-site Any time(Async.)
Mapping while commutingEasy to contributeDeep understanding of real conditions Position of contents depending on localization methods
Mapping partyutilizing crowdsourcingimage sharing service
Anywhere(Off-site)
Any time(Async.)
CrowdsourcingRemote mappingEasy to contribute anytime and anywhereDepend on shared dataLimited understanding of real conditions
Virtual mapping party
Anywhere(Off-site)
Any time(Async.)
CrowdsourcingRemote mappingEasy to contribute anytime and anywhere Easy to measure contentsʼ positionEasy to verify registered contentsMandatory pre-recording
国立研究開発法人
How to decide POI/POR position
28
Using intersection of line of sight and
the ground
Triangulating with plural panoramas.
POI: Point Of Interest (Landmark such as Store, restaurant, hospital, facilities, etc.)POR: Point Of Reference (specific point location the existence of which is easily recognized for confirming routes such as characteristic shape and material of ground (steps, stairs, sloop, door), sound/noise, and scent/odor.)
国立研究開発法人
Screenshots for the virtual mapping interface.
29
国立研究開発法人
Desktop vs. Smartphone VR
30
• The number of registered POI/PORs in WSs held at Miraikan : 598 (42 participant, 6 one-hour Workshops)
• Sense of Immersion: Desktop << Cardboard HMD • Registration efficiency: Desktop/Smartphone=1.43
POI/POR/Request on OSM
国立研究開発法人
Feedback from WS participants
31
Categories of feedback Positive feedback Negative feedback/ Suggestions
About VR experience with omnidirectionalimages/movies and
3D sound
I like the function for 3D sounds.3D sounds seem to be very useful, since visually impaired people can confirm amount of traffic on roads.I could more realistically experience the VR scene by omnidirectional movies than still omnidirectional images.
Estimating direction of sound sources was difficult.The quality of the images/movies was not perfect.
About devices for VR experience
I like instant HMD, since we can experience VR with what I have.I like VR experience with Oculus VR HMD since I can realistically experience by movies.I like Samsung's Gear VR HMD, because the image quality looked good and it was confortable for wearing.
It took a while to get used to HMD, and I got tired when I wore HMD.Mapping with PC is better in terms of degree of fatigue.An instant HMD was not so comfortable for wearing.I thought wearing HMD on glass was difficult.
About user interfaces I like the function for pointing in first personʼs view not map view.
PC is the easiest platform for inputting POR/POI.
About AR Tactile map I like the function for sending request by visually impaired people.
The accuracy of gesture recognition for AR tactile map needs to be improved.
About POR/POIsThere are so many POR/POIs in the display. I think it becomes more clear if the displayed contents are limited to nearby contents.I found empty POR/POIs without detailed information. I think filtering of the registered contents are required.
Other suggestionsI wondered if the system could support communication between participants.I would like to regularly contribute virtual mapping parties from my home.It was the most beneficial application of VR I have ever experienced.
国立研究開発法人
Usage of the AR tactile map for virtual mapping party
• Allowing the visually-impaired people to join the mapping party by gesture – Search: Confirming POR/POI on the tactile map with
sound for telling existence of POR/POI where user touches
– Tap: Confirming POR/POIwith Text-to-Sound whenuser taps the specific point
– Double Tap: Requestingthe POR/POI registration forspecific points of the map
国立研究開発法人
AR tactile maps with HP Sprout
国立研究開発法人
Automatic identification and tracking of tactile maps
34
• ORB Feature point detector/local feature descriptor is used for identifying tactile map with RGB image
• Estimating homography matrix between rectified image templates and input image