Lab-Forming Fields and Field-Forming Labs

国立研究開発法人

Lab-Forming Fields (LFF)and

Field-Forming Labs (FFL)Takeshi Kurata1, 2

1 Human Informatics Research Institute, AIST, Japan

2University of Tsukuba, JapanE-mail: t.kurata@aist.go.jp

1

Takeshi Kurata, Ph.D.• Position: 

– Research Group Leader, Service Sensing, Assimilation, and Modeling Research Group, Human Informatics Research Institute, AIST

– Professor (Cooperative Graduate School Program), Faculty of Engineering, Information and Systems, University of Tsukuba

• Professional Experience:– 2011‐2014 Doctoral co‐supervisor, Joseph Fourier University, UJF‐

Grenoble 1, France– 2012‐ ISO/IEC JTC 1/SC 24 Member– 2003‐2005 Visiting Scholar, HIT Lab, University of Washington

• Education:– 2007 Ph.D. (Eng.) from Doctoral Program in Graduate School of 

Systems and Information Engineering, University of Tsukuba– 1996 M.E. from Doctoral Program in Engineering, University of Tsukuba

• Research Interests:– Service Research, Assistive technology, Wearable/Pervasive Computing, 

Mixed and Augmented Reality, Computer Vision2

AIST http://www.aist.go.jp/

PresidentDr. Ryoji Chubachi

AIST, Tsukuba1 h drive from Tokyo

National Institute of Advanced Industrial Science and Technology

• One of the largest national institute in Japan– The independent agency of the Ministry of Economy, Trade and

Industry– The mission of AIST is advanced research and development for

industry– Over 2,300 permanent researchers– Over 50 research units cover various research fields

3

Research Fields and Staffs of AIST

4

Human Informatics Research Inst.

• History– Established in April, 2015– Main department is located in Tsukuba

• Organization– 85 permanent researchers– 10 research groups

• Brain science• Human factors engineering• Digital human modeling• Service engineering

5

Framework of Human Informatics

Human and Society Service

Presentation

AnalysisUnderstanding

Sensing

ParticipationSocial cognition

HealthcareWellness

SafetyComfort

Deep Data (High Quality Reference Data)

Big Data

ITIoT/Cyber-Physical

6

国立研究開発法人

Comb Data: Big + Deep in LFF & FFL

7

SFS

Dollhouse VR

CCE Lite

WearableRGB-D sensing

PDR

Handheld AR

Result/Behavior/Environment and LFF/FFL8

国立研究開発法人

Lab-Forming Fields & Field-Forming Labs

• Borrowing from “Terraforming”• Lab-forming Field: Transforming a real

field into a lab-like place. (IoT/G-IoT)• Field-forming Lab: Transforming a

laboratory into a field-like place. (VR)9

国立研究開発法人

Service design loop

10

国立研究開発法人 11

測って 図るHakatte Hakaru

MeasureWeighSurvey

PlanDesignAttempt

国立研究開発法人

測って図るHakatte Hakaru

12

ASPR Technologies for Multi-Stakeholders

13

国立研究開発法人

Efficient interactive label attaching for supervised Service Operation Estimation

14

So many kinds of positioning methods

15

PDR（Pedestrian Dead-Reckoning）Estimates velocity vector, relative altitude, and action type by measurements from a wearable sensor module.

Wearing a sensor module on waist (2D SHS (Steps and Heading Systems) PDR) Easy to wear and maintain Easy to measure data for action recognition Relatively easily apply for handheld setting compared to shoe-mounted PDR

(3D-INS (Inertial Navigation System) PDR)

16

Handheld PDR From PDR to PDRplus

10-axis sensors• Accelerometers• Magnetic sensors• Gyro sensors• Barometer

Shoe-mounted PDR

Waist-worn PDR

AR by PDR + Image registration(1999-2003)

Panorama-based Annotation: IWAR1999, ISWC2001,

ISMAR2003

G

Environmental mapA

B C D

E

A

B

C

F

Input frames

Position at whicha panorama is taken

PositionDirection

235 [deg]

5 [deg]From the user’s camera

Located Orientated

17

Frontier of PDR: Walking direction estimation

18

• Tutorial: Personal Navigation with Handheld Devices by Valerie Renaudin, IPIN 2015.

Frontier of PDR: Walking direction estimation

19

• Tutorial: Personal Navigation with Handheld Devices by Valerie Renaudin, IPIN 2015.• Long Paper: Christophe Combettes, Valerie Renaudin, Comparison of Misalignment

Estimation Techniques Between Handheld Device and Walking Directions, IPIN 2015.• FIS was proposed by Kourogi and Kurata in PLANS 2014.

“Globally, the FIS method provides better results than the other two methods.”

Frequency analysis of Inertial Signals

Forward and Lateral Acc. Modeling

Principal Component Analysis

Overview: History of our PDR20

ISWC2001

IWAR1999

ISMAR2003

PLANS2014

PLANS2010 ICServ2013

Docomo map navi(500 areas as of March, 2017))

Image registration + Gyro

Panorama-based annotation (Image-registration-based positioning)

Image registration + PDR

PDRplus (PDR + Action recognition)

Handheld PDR(Walking-direction estimation)

2015- 2015-

PDR module

2011-

Academia

Industry

Before PDR

ICAT2006 PDR + GPS + RFID

Global Trend on PDRPDR R&D players have rapidly indicated their presence all over the world on and after 2010.

Movea (France)

Sensor Platforms (USA)

CSR (UK)

TRX (USA)

Trusted Positioning (Canada)

21

Acquired by QualcommAcquired by InvenSenseAcquired by InvenSense

Acquired by Audience

Indoo.rs (USA)

SFO

Standardization on PDR Benchmarking• PDR related R&D is highly active worldwide: Necessity for sharing

common measures.• Description of the performance should be unified in spec sheets

and scientific papers.• Different measures from absolute positioning methods such as

GNSS, Wi-Fi, and BLE are required for PDR, which is a method of relative positioning.

• PDR Benchmark Standardization Committee was established in 2014 as a platform of the grassroots activity.

22

https://www.facebook.com/pdr.bms

Support Organizations• Asahi Kasei Corporation, Asia Air Survey Co., Ltd. (Y. Minami), INTEC Inc.,

MTI Ltd., KDDI R&D Laboratories, Inc., KOKUSAI KOGYO CO., LTD.,SHIBUYA KOGYO CO., LTD., Koozyt, Inc., GOV Co., Ltd., SITESENSING, inc., Sharp Corporation, Sugihara Software and Electron Industry Co., Ltd. (SSEI), ZENRIN DataCom CO., LTD., Information Services International-Dentsu, Ltd. (ISID), Hitachi, Ltd., IBM Japan, Ltd., Frameworx, Inc. (S. Watanabe), MULTISOUP CO.,LTD., Milldea, LLC, Murata Manufacturing Co., Ltd., MegaChips Corporation, Recruit Lifestyle Co., Ltd. (K. Ushida), RICOH COMPANY, LTD., Rei-Frontier Inc.,

• Aichi Institute of Technology (K. Kaji), NARA Institute of Science and Technology (NAIST) (I. Arai), Kanagawa Institute of Technology (H. Tanaka), Keio University (S. Haruyama, N. Kohtake, M. Nakajima), University of Tsukuba (T. Kurata), Tokyo Institute of Technology (S. Okada), Nagoya University (N. Kawaguchi), Niigata University (H. Makino), Ritsumeikan University (N. Nishio), National Institute of Advanced Industrial Science and Technology (AIST) (T. Kurata, M. Kourogi), Human Activity Sensing Consortium (HASC), Location Information Service Research Agency (LISRA)

• 36 organizations in Japan as of March, 2017

23

Scene in data collection25

PDR Challenge Series• Ubicomp/ISWC 2015 PDR Challenge

– Scenario: Indoor Navigation– On-site– Continuous walking while keeping watching the

navigation screen by holding the smartphone– Several minutes per trial

• IPIN 2017 PDR Challenge in Warehouse Picking– Scenario: Picking work in a warehouse– Off-site– Not only walking but various actions including picking

and carrying– Several hours per trial

26

IPIN2017

27

• User Requirements• Hybrid IMU Pedestrian Navigation & Foot Mounted

Navigation• Human Motion Monitoring• High Sensitivity GNSS, Indoor GNSS, Pseudolites• RTK GNSS with handheld devices• Mitigating GNSS errors prior to moving indoors• Self-contained sensors• Signal Strength Based Methods, Fingerprinting• UWB (Ultra-wideband)• Passive & Active RFID• Optical Systems• Ultrasound Systems• TOF, TDOA based Localization• Localization, Algorithms for Wireless Sensor Networks• Frameworks for Hybrid Positioning• Industrial Metrology & Geodetic Systems, iGPS• Radar Systems• Mapping, SLAM• Indoor Spatial Data Model & Indoor Mobile Mapping• Novel uses of maps and 3D building models• Magnetic Localization• Innovative Systems• Location Privacy• Applications of Location Awareness & Context

Detection• Health and Wellness ApplicationsRegular Papers Due: April 30, 2017

Integrated Positioning (SDF)

28

Sub-meter indoor positioning: Visible Light Communication (VLC) & PDR

• Less density of infrastructure installation by SDF combining VLC and PDR

• Reduction of initial/running cost of sensing by Replacement demand of lighting

29 Collaboration with Panasonic

RGBD (Depth) sensor & PDR• Error compensation of PDR with precise

trajectories obtained from surveillance (RGBD) cameras

• Coverage compensation of surveillance cameras with continuous measurement of PDR

30

国立研究開発法人

VDR (Vehicle/Vibration-based DR)

31

国立研究開発法人

Whole-body posture estimation and precise positioning

32

Many sensors for heterogeneous and more precise real-world capturing (position, orientation, posture, physical load, etc.) and deep-data gathering

CSQCC (Computer-supported QC Circle)

33 Staying-time rate at each dinning area per personSales at each dinning area per employee

Visualization tool combining human-behavioral and accounting history

Employee taking order while cleaning up the

guest room

Icons showing the number of customers at each table

POS data log

Service Characteristics1. Intangible2. Heterogeneous3. Inseparable4. Perishable

Alleviate the issues due to IHIP

QCC in manufacturing industryPurpose: Productivity improvement

Conventional QCC in service industryPurpose: Productivity improvement

Subjective QCC in service industryPurpose: Improvement of CS/ES

w/ reasonable ways to gather objective data in plants

In 1980s, applying QCC for service industry

w/o reasonable ways to gather objective data in service fields

In 1990s, Service industry lost interest in QCC

In 2000

QCC in the Service Industry in Japan

34

Computer-supported QCC (CSQCC)Purpose: Productivity improvement

In 2010

CSQCC in the futureProductivity improvementImprovement of CS/ES

w/ reasonable ways to gather subjective data continuouslyw/ reasonable ways to

gather objective data in service fields

1950~ Deming Award

3rd CSQCC for newly open

35

Yamano Aiko-tei, Shinjuku: Mansion style restaurant

(2014.12.23)

2014.10.18 (Sat) 2014.11.08 (Sat)

Case studyin Japanese Restaurant “Ganko”

• Objectives1. (for AIST) to test the CSQCC

(Computer-Supported QCC) suites in a real service field.

2. (for the restaurant) to observe effects of process improvement planned by CSQCC.

• Place– Japanese cuisine restaurant

GANKO Ginza 4-chome (Tokyo)

• Term– 1st term

• January 12 to 18, 2011– 2nd term

• February 3 to 9, 201136

Dining area Course dishes

1st term(Jan. 12-18, 2011)

for observing ordinary operations

QC circlefor making improvement plans

2nd term (Feb. 3-9, 2011)for observing improved operations

37 B2

B1

Dinning Area

Kitchen

Office room

Pantry

During Discussion in CSQCC38

Trajectory of a wait staff in lunch time: 12:00-14:00

Fact: Going in and out of the kitchen/office to no small extent.Possible result: Difficulty in concentrating on guest service.Cause: Cell phone everywhere, but reservation book only in the office room.Possible improvement: e-reservation book

Dinning Area

Kitchen

Office room

T. Fukuhara, R. Tenmoku, T. Okuma, R. Ueoka, M. Takehara, and T. Kurata, "Improving Service Processes based on Visualization of Human-behavior and POS data: a Case Study in a Japanese Restaurant“, ICServ2013, pp.1-8.

Summary of 1st CSQCC for Wait Staff

39

Grasp of actual condition Shorter stay in dinning area than the manager assumed

Kaizen plan development (1) Re-composition of service processes (SP)(2) Thoroughly obeying each division’s roll, (3) Guts

Direct effect Stay ratio in dinning area at dinner time: UP ↑Spillover effect Number of additional orders at dinner time: UP ↑

Side effect(Trade-off)

(1) Work load (walking distance): No difference →(2) Number of additional orders at 3pm: No difference →

Stay ratio in dinning areas

30%

35%

40%

45%

50%

55%

11 12 13 14 15 16 17 18 19 20 21 22

Walking Distance [m]

1,000

1,500

2,000

2,500

11 12 13 14 15 16 17 18 19 20 21 22Num. of additional orders per customer

0.0

0.4

0.8

1.2

11 12 13 14 15 16 17 18 19 20 21 22Hour Hour Hour

BeforeAfter

Down: Due to SP re-comp. for preparation

of dinner/partyUP: Much more than time

decreased in Tea hour

No diff.: Due to no SP re-comp.

No diff.: Despite SP re-comp. for preparation of dinner/party

UP: due to reduction of opportunity loss

No diff. on workload

Lunch Tea Dinner Lunch Tea Dinner Lunch Tea Dinner

Walk distance of waiting staff per customer (meters / hour / person)

40

***

* p < .05, ** p < .01, *** p < .001

******

They were able to reduce walking distance while not reducing staying time in the dining area!

Indicators for position keeping

41

B2

B1

Zone Dedication Rate＝Orange／RedZone Order Defense Rate ＝Orange／Blue

All of orders in the staffʼs zone

# of accepted orders by a staff in the staffʼs zone

The total # of accepted orders by the staff

Relation between skill level andZone Defense/Dedication

42

IV. ExpertThey take all of orders in their zone while taking orders in other zone for helping others.

II. Fully occupiedThey take orders in his/her own zone but it is not enough for covering the zone. Support by other staffs is needed.

III. Well organizedThey take all of orders in his/her zone, but they don’t help other zones.

I. PurposelessThey fail to take orders in his/her zone and take orders in other zones. Training is required.

Zone

ord

er d

efen

se ra

tio (Z

OD

):Th

e ra

tio o

f # o

f acc

epte

d or

ders

by

a st

aff i

n hi

s/he

r ow

n zo

ne o

ut o

f all

of o

rder

s in

the

zone

Zone dedication ratio (ZD):The ratio of # of accepted orders by a staff in his/her own zoneout of the total # of accepted orders by the staff

Precision

individual skill Teamworkperformance

Before

43

Precision

After

44

Improved coverage of each zone by each staff

Less need for helping other staffs (zones)

Precision

Pre-evaluation of Kaizen PlanConsidering Efficiency and Employee Satisfaction

by Simulation Using Data Assimilation

45

Sensing ModelingPicking work model of employee

Action

HT

WMSSimulation

・Analyze・Visualize

EmployeeCart

Receiver

VL with IDVLC

Evaluation

Kaizen Support FrameworkSimulator

Planning

・Man-hour Productivity・Worktime・Time to spare ・Evenness of work rate

traffic line

Subject Extraction

Kaizen of the kaizen activity

46

Simulation To-Be

Pre-evaluation

Sensing analyzevisualize As-Is

Understand current status

KaizenPlan C

KaizenPlan B

KaizenPlan A

Action

It is possible to quantitatively decide Kaizen planand to apply KSF to several warehouses

Overview of the measurement field

25m

50 meters

54 meters

D ABC

47

Wide passage

Narrow passage

25m

48

When many employees conduct picking work,Zone A become crowded.

Items in A zone were picked frequently.

Overview of the measurement field

P1P2

P3HT

Measurement method ：Warehouse Management System (WMS)

WMS manages items and provides information. When employees pick an ordered item, they are required to scan a barcode with a hand terminal.We can estimate positions from scan data 49

Measurement method : Visible Light Communication System

Receiver

50

Measure and record the positions of the employee and carts that are equipped with a receiver

P1P2

P3

HT

Measurement method ：Warehouse Management System

and Visible Light Communication System

Receiver

Estimate route during pickings for combination with timestamp

51

Picking work model constructed and verification of reproduction

AB

C

Order

HT

52

CartEmployee confirm the orders using hand terminalMove toward shelf and into the wide passage with cart

Place cart near shelf on wide passage and leave and enter narrow passage

Pick up items and read a barcode with hand terminalReturn to cart and place picked items on the cart

In this model essentially handles plural orders by repeating these steps

Sorting place

2. Distribute Shelves equally for every zone

1. Divide one floor with some zones

Zone Picking

3. Employee takes charge of only one zone

5. Processes a zone package and brings it to the sorting place

4. Created by a combination of the same zone’s sub‐orders.

53

54

Simulated trajectoriesActual method (Single picking) Kaizen plan (Zone picking)

Trajectory distinguished by color for each employee

Actual method(Single picking)

Kaizen plan(Zone picking)

# of zone -

# of employeesN-7

3-4

4-4

5-5

6-7

7-7

EFMan-hour productivity M H H H H H

Work time as a team M H M M L L

ES Time to spare L L L L M M

Evenness of workload M M L L M L

The result of the best combinations of efficiency and employee satisfaction

EF (Efficiency), ES (Employee Satisfaction)

55

Interview with FPV

Passage of Time

+ Over 50% cost reduction on labor cost and preparation time compared with existing time studies+ Consideration of customer privacy by not using cameras+ FPV with less motion sickness+ Effective in episodic memory retrieval for retrospective interviews considering bounded rationality

Worker’s trajectory

3D model built from a set of photos

First-person view (FPV)

CCE (Cognitive Chrono-Ethnography) Lite

Japanese-style hotel at Kinosaki Onsen (hot spring)

56

国立研究開発法人

キャビンアテンダントのおもてなし分析

57

東⼤・ANA総研の共同研究、及び東⼤・産総研の共同研究の事例⽇経情報ストラテジー12⽉号

• 飛行中の機内でＣＡの動線を計測• PDR+BLE+マップマッチング• BLEは機内持ち込み荷物の中でラピッド設置・撤去

国立研究開発法人

PDR+BLEを⽤いた達⼈CAと新⼈ＣＡの⽐較

58

CA1業務内容

CA2業務内容

行き 帰り 計

達人CA 15分 6分 21分

新人CA 17分 3分 20分

• ドリンク提供の帰り時間を多く作る• おかわりを申告してもらいやすい• 乗客の変化へ対応がしやすい

1. 乗客の変化に気づき対応するという受動的な行動

2. 乗客の申告を促す能動的な行動

2種類の行動メカニズムの存在の示唆

得られた知見

[日経情報ストラテジー2015年12月号より]

Service Field Simulator•Supporting service design using VR technology

– Evaluating service environment and its process in advance by sensing and analyzing human behavior in virtual environment

Risk reduction by evaluation of the new service in advancecomparison between • current layout and new layout plan• current process and new process

Acquiring more detail and reliable data• Various sensors are available because of

limited sensing area• Easy to control the condition

As is New plan

With EEG

With Eye-Tracker

59

Continued improvementSFS Ver. 1.0• Low resolving power: 0.2• Short of vertical FOV

SFS Ver. 2.0• 24 Full-HD 27-inch LCD: Resolving

power is improved to 0.7

SFS Ver. 2.1• 40 Full-HD 24-inch LCD: Vertical

FOV is improved (Upper 35°, Lower 58.5°)

60

Case studies for verifying efficiency•Gaze point analysis using combination of eye-

tracking device and SFS

– Hypothesis•we can do the same investigation using an eye-tracker and the SFS

as real in-store marketing

in-store marketing experienced person（subjective opinion）："the motion of the gazed point in the virtual environment is similar to that in the real store especially from the entrance to in front of the shelf where target products are laid out"

61

Case studies for verifying efficiency•Investigation for a method for measuring human interest

using EEG and the SFS

62

Example of Analysis and Future Work

63

To compare the shopping behavior in detail, we made heat-map visualization of thestay time for each 50 cm grid in the real and virtual store. The read area indicatessubjects spent longer time than other area. Because position data of the real storesituation is recorded by hand, we only have the discrete position and timestampdata. Therefore, we could not compare both of them strictly, but we found out wecould get the similar results.

Comparison of heat-map visualization of stay

Virtual store in SFSReal store

Dollhouse VR: An Asymmetric Collaborative System for Architectural-scale Space Design

64 提供︓慶応⼤ 杉浦先⽣

Collaborative system for multi-stakeholders

国立研究開発法人

Research cases on LFF and FFL in AIST

65

国立研究開発法人

Thank You!!

66

SFS

Dollhouse VR

CCE Lite

WearableRGB-D sensing

PDR

Handheld AR