Benchmarking of vision-based registration and tracking for MAR

Benchmarking of Vision-based Spatial Registration and

Tracking Methods for MAR(ISO/IEC NP 18520)

Takeshi Kurata12, Koji Makita13, Takafumi Taketomi4, Hideaki Uchiyama5, Shohei Mori6, Tomotsugu Kondo7, Fumihisa Shibata8

1AIST, 2Univ. of Tsukuba, 3Canon, 4NAIST, 5Kyushu Univ., 6Keio Univ., 1The Open Univ. of Japan, 1Ritsumeikan Univ.

ISMAR 2016 Workshop: Standards for Mixed and Augmented Reality (2016/9/23)

ISO/IEC WD (Working Draft) 18520• Main Body

– Terms and Definitions– Benchmarking framework – Benchmark Indicators– Trial set for benchmarking

2

Benchmark Indicators ＋

Benchmarking Framework

Trial Set(Dataset)＋

• Annex A: Benchmarking organizations and activities

• Annex B: Tracking competitions in ISMAR

Benchmarking framework

vSRT: Vision-based spatial registration and tracking 3

Example of stakeholders and their roles

4Technology developer


5Technology supplier

Example of stakeholders and their rolesBenchmarking

service provider

6


7

Benchmark provider


8

Technology user


9

Benchmark provider

Technology supplier

Benchmarking service provider

Technology developer

Technology user

Benchmark Indicators


Off-site On-site

Reliability

• PEVO• Reprojection error of image

features• Position and posture errors

of a camera


features• Position and posture errors of a

camera• Completeness of a trial (kind of

Robustness?)

Temporality • Latency• Frequency • Time for trial completion

Variety

• Number of datasets used for benchmarking

• Variety on properties of datasets used for benchmarking

• Number of trials conducted for benchmarking



PEVO: Projection error of virtual objects, which is the most direct and intuitive indicator for vSRT methods for MARvSRT: Vision-based spatial registration and tracking 11

Off-site On-site

Reliability


features• Position and posture errors

of a camera


features• Position and posture errors of a

camera• Completeness of a trial

Temporality • Latency• Frequency • Time for trial completion

Variety

• Number of datasets used for benchmarking


• Number of trials conducted for benchmarking



PEVO: Projection error of virtual objects, which is the most direct and intuitive indicator for vSRT methods for MARvSRT: Vision-based spatial registration and tracking 12

ISMAR 2015 Tracking competition

Trial set for benchmarking


Off-site On-site

Dat

aset C

onte

nts

• Image sequences• Ground truth of intrinsic/extrinsic

parameters of one or more cameras • Optional contents

• 3D model data for the target objects in image sequences

• 3D model data for virtual objects in image sequences

• Depth image• Self-contained sensor data, etc.

• Ground truth of challenge points• 3D models for the target objects• 3D models for virtual objects

overlaid in benchmarking

Met

adat

a • Scenario• Camera motion type• Camera configuration• Image quality

• Scenario

Physical object

instances

• Easily available or deliverable physical objects

• Information on how to find the physical objects

• Physical objects


14

Off-site On-site

Dat

aset C

onte

nts








Met

adat


• Scenario

Physical object

instances





TrakMark

15

Off-site On-site

Dat

aset C

onte

nts








Met

adat


• Scenario

Physical object

instances





Metaio

16

Off-site On-site

Dat

aset C

onte

nts








Met

adat


• Scenario

Physical object

instances





The City of Sights:An Augmented Reality Stage Set

17

Off-site On-site

Dat

aset C

onte

nts








Met

adat


• Scenario

Physical object

instances






18

Off-site On-site

Dat

aset C

onte

nts








Met

adat


• Scenario

Physical object

instances






19

Off-site On-site

Dat

aset C

onte

nts








Met

adat


• Scenario

Physical object

instances






20

IT project performance benchmarking framework (ISO/IEC 29155 series)

MAR Reference Model(ISO/IEC CD 18039)

Benchmarking for MAR

Benchmarking of vision-based geometric registration and tracking methods for MAR

ISO/IEC WD 18520

Venn diagram on conceptual relationship between ISO/IEC 29155 series, ISO/IEC CD 18039, and ISO/IEC WD 18520

21

IT product, system, and service

vSRT methods for MAR

IT projectBenchmarking of vision-based geometric

registration and tracking methods for MAR(ISO/IEC WD 18520)

Create, Improve

IT project performance benchmarking

Conduct Benchmarking

IT project performance benchmarking framework(ISO/IEC 29155 series)

Target of benchmarking:Performance of

vSRT methods for MAR

Target of benchmarking:IT project performance

Layered structure between ISO/IEC 29155 series and ISO/IEC WD 18520

22

Thank you!• AIST is now hiring for Tenure-track, Postdoc, and

RA (PhD) positions at Tsukuba, Japan.• Target research fields are ↓ ↓ ↓

23

Engineering

Benchmarking of vision-based registration and tracking for MAR