CAADFutures2017: Integration of a Structure from Motion into Virtual and Augmented Reality for Architectural and Urban Simulation

Integration of a Structure from Motion

into Virtual and Augmented Reality

for Architectural and Urban Simulation

July 13 2017

Tomohiro Fukuda1 Hideki Nada2 Haruo Adachi2

Shunta Shimizu3 Chikako Takei3 Yusuke Sato1

Nobuyoshi Yabuki1 Ali Motamedi1

1

Demonstrated in Real Architectural and Urban Projects

1 Osaka University, Osaka, Japan

2 Sakaiminato City Office, Tottori, Japan

3 Forum8 Co., Ltd., Tokyo, Japan

Contents1. Introduction

2. SfM-based Modeling for VR

3. Using Camera Parameters of SfM for Marker-less AR

4. Conclusions

2

Virtual and Augmented Reality

Constructs 3D past and future virtual spaces with real-time andinteractive operation.

Considerable amount of time and expense is entailed on anurban scale to construct and integrate 3D objects for all virtualspaces.

3

RealEnv.

AugmentedReality (AR)

AugmentedVirtuality (AV)

Virtual Env. = VR

Mixed Reality (MR)

Simplified representation of a "virtuality continuum“ by Milgram and Kishino (1994)

VR Azuchi Castle prj.

CAAD Futures 2015, Sao Paulo

Housing Design prj.

eCAADe 2015, Vienna

Virtual and Augmented Reality

Overlays 3D virtual objects on real world images

Can help visualize full-scale design projects on a planning site

Must track the geometric registration between live video and 3Dobjects to accurately render the outdoor AR

4

RealEnv.

AugmentedReality (AR)

AugmentedVirtuality (AV)

Virtual Env. = VR

Mixed Reality (MR)

Simplified representation of a "virtuality continuum“ by Milgram and Kishino (1994)

Sensor-based AReCAADe2006, Volos

Marker-based AR2015 Int’l WS on Computing in Civil Engineering, Texas

17 22 27℃

SfM (Structure from Motion)

Estimates 3D structures from 2D sequential images that are coupled to local motion signals

Requires only familiar equipment (Smart phones), free or low-cost software

5Google Maps

Target Building

Shooting routes and directions

Camera parameter

Point clouds点群

Dense point clouds

1,335,958 points Camera Parameter3,194,585 faces

1,598568 vertices

Objective

Integration of SfM into VR and AR:

A 3D modeling method by SfM are explored for real-time

rendering to realize less modeling cost in large-scale VR

Reduce a large number of generated point clouds and meshes

for the frame rate of real-time rendering

Camera parameters of SfM are explored for a marker-less AR

to realize registration and tracking accuracy to visualize full-

scaled architectural design projects

Apply SfM-based AR to real architectural design projects to

examine the uses and limitations

6




4. Conclusions

7

Urban Design Project:

Mizuki Shigeru Road Renewal

4

Challenges Aging

Barrier-free

Schematic renovation plan

Creating a hospitable and entertaining road

that everyone would like to visit

Shigeru Mizuki MuseumJR Sakaiminato st.800ｍ

水木マンガの世界

森にすむ妖怪たち

神仏･吉凶を司る妖怪たち身近なところにひそむ妖怪たち

川や水辺野や里山海辺町家に棲む妖怪たち

Creation of a one-way and S-shaped roadway

Creation of a wider sidewalk space

Re-accommodation of the bronze statues within the new

zone concept

Improvement of cityscape with a sense of unity

Night landscape design

15

16

VR production

Challenges 3D digital modeling of 153 bronze statues

Possess different shapes and included complicated shapes with holes

Conventional method Modeling using 3DCG software demands enormous time and cost

resources

17

Construction design study and public meeting to build

consensus

Representation of a Human’s Motion and Direction by 2D images in VR

(Fukuda et al., Int’l J. of Arch. Comput. 7(2), 2009)

Photography for SfMConditions

Shoot on cloudy days To generate a shadow-less texture at 3D

modeling by SfM to ensure optical consistency

between VR and 3D objects by SfM

Shoot at times of less tourists Not to shoot noise elements

Role sharing

Sakaiminato city office staffs took 10,000

photos of 153 statues in a month VR experts worked away from the project site

18

Photographs for SfM

19

SfM-based Statues Modeling

20

High4286 polygons

Medium1025 polygons

Low298 polygons

Software: PhotoScan, Remake,UC-win/Road SfM plug-in

Shooting time: 10-20 min./statue

Production time: Automatically generated 30 min. Editing 30min.

SfM-based Statues Modeling

153 bronze statues are ranked

into 3 categories:

High 27 Med 46 Low 80

21

OS Windows7 Professional 64-bit

CPU Intel Corei7-3770 3.40GHz

RAM 16GB

GPU NVIDIA GeForce GTX 670 VRAM 4GB)

Res. 1920x1080

100 statues

Accuracy fps

High 16.4

Med. 24.1

Low 27.1

When all the high-accuracy statues are

installed, realizing real-time rendering is

difficult.

Results: 3D model by SfM

22

河童

ねずみ男ガラッパ小豆洗い

鬼太郎と目玉おやじ

かみきりと一反木綿

袖引小僧龍水木しげる先生執筆中

(N=27) (N=46) (N=80)

SfM-based Roadside Building Modeling

23

Conventional method

Polygons 19

Vertices 48

120 buildings along the roadside▶ Difficult to realize real-time

rendering using SfM-based 3D model

SfM (e.g. PhotoScan)

Polygons 2.4 million

Vertices 1.2 million

SfM-based Roadside Building Modeling

24

Conventional method

Editing work still involves a considerable amount of time.

The roadside buildings were finally created by using the

conventional method.

SfM-Photoscan SfM-Remake

Lack of content: Elements to be modeled are not

modeled by SfM (roof)

Inadequate content: Geometry is not correct

although it is modeled by SfM (signs, arcade)

Surplus content: Unnecessary objects are

modeled (wire contains sky elements)

25

3D modelNumber of polygons 965,472

Number of vertices 929,325

Texture

Number of textures 885

Number of texture pixels 65,530,368

Data amount 1.5 GB

VRAM use when running VR application 667.4 MB

TotalData amount 1.35 GB

Frame rate 30-50 fps

26

27

28水木しげるロードリニューアルイメージ映像




4. Conclusions

29

Marker-less AR: System Flow

30


31

Photographs from viewpoints


32

Store position and orientation data of all camera

3D point cloud reconstructed by SfM


33

Define relative coordinates of 3D

design objects using SfM model


34

Start video capture (OpenCV 2.4)


35

Extract features of the live video image

Compare them with features in DB

using SURF (Speeded-up Robust Features)

Extract the camera’s position and

orientation of the most similar image


36

Render 3D virtual objects precisely in AR

using referred position and orientation data


37

Motion vectors for tracking are calculated by the optical flow technique.

Positions of points on the screen and the corresponding world coordinates of those

points are calculated in real time.

Application in Building Design prj.

38

To study the facade and approach of the new office building at the preliminary design stage (Jan – Mar 2016)

Demolish the existing office building and build a new three-story office building in Osaka, Japan

0 20m

Interview survey

Advantage

The size of the building could be intuitively comprehended at

the construction site

It was possible to obtain deeper communication in the

preliminary design stage.

Disadvantage

Difficulty of viewing the exterior as well as the building model

through the tablet LCD display.

Walking freely by using a video see-through type HMD is difficult

because the surrounding environment is not adequately visible.

Using an optical see-through HMD, a problem exists wherein it is

difficult to view the AR due to the effect of sunlight.

39

Interview survey

Although the size of the rendered 3D model is accurate, an

opinion persisted that this was not possible.

The validity of the size of the designed building can be

explained by comparing it with the size of the adjacent

building in the experiment.

40




4. Conclusions

41

Conclusions Integrating SfM into VR and AR is demonstrated for

architectural and urban visual simulations.

A modeling method using SfM is developed to model a number

of 3D objects under real-time rendering conditions. The total

data amount of bronze statues is suppressed by setting three

reproduction ranks. A large-scale VR application can be

constructed with decreasing the modeling cost.

An AR system using SfM camera parameters for registration is

developed to visualize design projects. Design stakeholders

could study the size and facade of the office building by using

the full-scale displayed AR at the planning site. It is possible to

obtain a more constructive communication among

stakeholders.

A future study should be examined further integration into VR

and AR based on the technology to restore a 3D model in real

time from camera images (e.g., Simultaneous Localization and Mapping).

42

Acknowledgements The research was partly supported by JSPS KAKENHI Grant

Number JP25350010 and JP26-04368, and joint research

funding between Sakaiminato city office and Osaka University.

We would also like to take the opportunity to thank Kyokuyo

Electric Co., Ltd. and atelier DoN to apply our proposed AR

system to the building design project.

We thank all the participants for their generous assistance in

conducting projects and experiments.

4343Mizuki Shigeru Road (Jul 2017) Kyokuyo Electric Co. Ltd. New head office (Jul 2017)