Синтез изображений по изображениям.

Рельефные текстуры

Сегодня на лекции

• Введение в Введение в Синтез Изображений по Синтез Изображений по ИзображениямИзображениям (Image-Based Rendering)(Image-Based Rendering)

• Простейшие методы Простейшие методы IBRIBR

• Рельефные текстурыРельефные текстуры (relief textures)(relief textures)

• Введение в Введение в Синтез Изображений по Синтез Изображений по ИзображениямИзображениям (Image-Based Rendering)(Image-Based Rendering)

• Простейшие методы Простейшие методы IBRIBR

• Рельефные текстурыРельефные текстуры (relief textures)(relief textures)

Traditional Rendering

User inputtexture maps,survey data

User inputtexture maps,survey data

GeometryTextures

Light sources

GeometryTextures

Light sources

ImagesImages

Modeling

Rendering

For photorealismFor photorealism

• Modeling is hardModeling is hard

• Rendering is slowRendering is slow

For photorealismFor photorealism

• Modeling is hardModeling is hard

• Rendering is slowRendering is slow

Image-Based Rendering

Основные идеи:Основные идеи:

• Использование изображений (фотографий) в Использование изображений (фотографий) в качестве исходных данныхкачестве исходных данных

• Использование методов обработки изображений Использование методов обработки изображений для визуализации.для визуализации.

Основные идеи:Основные идеи:

• Использование изображений (фотографий) в Использование изображений (фотографий) в качестве исходных данныхкачестве исходных данных

• Использование методов обработки изображений Использование методов обработки изображений для визуализации.для визуализации.

Image-Based Rendering

Imagesuser input

range scanners

Imagesuser input

range scanners

Images & depth mapsImages &

depth maps

ImagesImages

Modeling

Rendering

For photorealismFor photorealism

• Fast modeling Fast modeling

• Complexity independent Complexity independent renderingrendering

For photorealismFor photorealism

• Fast modeling Fast modeling

• Complexity independent Complexity independent renderingrendering

Traditional vs. Image-based

ImageImage based computer graphics has three main based computer graphics has three main advantagesadvantages

• Photorealism of produced imagesPhotorealism of produced images

• The speedThe speed

• Simple modelingSimple modeling

ImageImage based computer graphics has three main based computer graphics has three main advantagesadvantages

• Photorealism of produced imagesPhotorealism of produced images

• The speedThe speed

• Simple modelingSimple modeling

Simplest IBR methods. Texture

Texture - is the simplest Texture - is the simplest of IBR methods.of IBR methods.

Texture - is the simplest Texture - is the simplest of IBR methods.of IBR methods.

Simplest IBR methods. Sprites

Texture + simple planar Texture + simple planar geometry = Spritegeometry = Sprite

Sprites are taken from Sprites are taken from certain camera position certain camera position

(sprite camera)(sprite camera)

Texture + simple planar Texture + simple planar geometry = Spritegeometry = Sprite

Sprites are taken from Sprites are taken from certain camera position certain camera position

(sprite camera)(sprite camera)

Simplest IBR methods. Sprites

No geometry No geometry information and...information and...

Sprites are looking good Sprites are looking good from view-points close from view-points close

to the sprite camera, but to the sprite camera, but awful from othersawful from others

No geometry No geometry information and...information and...

Sprites are looking good Sprites are looking good from view-points close from view-points close

to the sprite camera, but to the sprite camera, but awful from othersawful from others

3D model rendering distortions

What are we missing?What are we missing?

The effect of 3-dimensionality on the screen is a combination of two effects: perspective distortion and parallax

Sprites are capable of producing perspective Sprites are capable of producing perspective distortions but they are unable to produce distortions but they are unable to produce correct parallax effectcorrect parallax effect

What are we missing?What are we missing?

The effect of 3-dimensionality on the screen is a combination of two effects: perspective distortion and parallax

Sprites are capable of producing perspective Sprites are capable of producing perspective distortions but they are unable to produce distortions but they are unable to produce correct parallax effectcorrect parallax effect

Sprites with Depth

Combine depth and Combine depth and color:color:

Color texel now is a 3D Color texel now is a 3D sample.sample.

Combine depth and Combine depth and color:color:

Color texel now is a 3D Color texel now is a 3D sample.sample.

Warping

Using the samples z-Using the samples z-values, image can be values, image can be transformed (or transformed (or warpedwarped), to enchance ), to enchance the image descriptive the image descriptive power (realism)power (realism)

Using the samples z-Using the samples z-values, image can be values, image can be transformed (or transformed (or warpedwarped), to enchance ), to enchance the image descriptive the image descriptive power (realism)power (realism)

Planar pinhole camera

Orthographic camera

3D point position from a pinhole camera

xtCenterv

u

tPCenterX

1

xtCenterv

u

tPCenterX

1

One image is not One image is not enough to enough to

determine location determine location of a point in 3D. of a point in 3D.

One image is not One image is not enough to enough to

determine location determine location of a point in 3D. of a point in 3D.

From two cameras

If we have two cameras, CameraIf we have two cameras, Camera11 and and

CameraCamera22 with different parameters, capturing with different parameters, capturing

the same scene from different locations, a the same scene from different locations, a point can be expressed aspoint can be expressed as

If we have two cameras, CameraIf we have two cameras, Camera11 and and

CameraCamera22 with different parameters, capturing with different parameters, capturing

the same scene from different locations, a the same scene from different locations, a point can be expressed aspoint can be expressed as

222111 xtCenterxtCenterX

222111 xtCenterxtCenterX

Warping example

Schematic view

Depth map

Final view

Warping

x’ = warp(x)

Parallax,

perspective projection,

translation

Relief textures

Most of modern 3D accelerators can Most of modern 3D accelerators can ultimately fast render textured trianglesultimately fast render textured triangles

=> => We can use this capability to speed upWe can use this capability to speed up and simplifyand simplify Image-Based Rendering Image-Based Rendering

SSuch an algorithmuch an algorithm is is called called Relief TexturesRelief Textures

Most of modern 3D accelerators can Most of modern 3D accelerators can ultimately fast render textured trianglesultimately fast render textured triangles

=> => We can use this capability to speed upWe can use this capability to speed up and simplifyand simplify Image-Based Rendering Image-Based Rendering

SSuch an algorithmuch an algorithm is is called called Relief TexturesRelief Textures

Schematic view

Relief textures

Final view

Pre-warping

Parallax

Warped textures

Texture mapping

Perspective projection,

translation

Software Hardware

New warping function

x’ = warp(x) = g(h(x), Poly),x’ = warp(x) = g(h(x), Poly),

wherewhere

– g(y, Poly) is texture-mapping function and usually g(y, Poly) is texture-mapping function and usually done in hardwaredone in hardware

– h(x) is pre-warpring functionh(x) is pre-warpring function

x’ = warp(x) = g(h(x), Poly),x’ = warp(x) = g(h(x), Poly),

wherewhere

– g(y, Poly) is texture-mapping function and usually g(y, Poly) is texture-mapping function and usually done in hardwaredone in hardware

– h(x) is pre-warpring functionh(x) is pre-warpring function

New warping function (2)

Prewarping function h(x) after some Prewarping function h(x) after some optimizations looks like following:optimizations looks like following:

u2 = (u1+mu[d])*nu[d];

v2 = (v1+mv[d])*nv[d];

Extremely simple, isn’t it?Extremely simple, isn’t it?

Prewarping function h(x) after some Prewarping function h(x) after some optimizations looks like following:optimizations looks like following:

u2 = (u1+mu[d])*nu[d];

v2 = (v1+mv[d])*nv[d];

Extremely simple, isn’t it?Extremely simple, isn’t it?

Two stage warping example

Two stage warping example (2)

Filling holes

This sprite with depth This sprite with depth was warped to the new was warped to the new viewpointviewpoint

Look how many empty Look how many empty spaces on the women spaces on the women face and hair at the face and hair at the picturepicture

Let’s call them Let’s call them holesholes

This sprite with depth This sprite with depth was warped to the new was warped to the new viewpointviewpoint

Look how many empty Look how many empty spaces on the women spaces on the women face and hair at the face and hair at the picturepicture

Let’s call them Let’s call them holesholes

Two classes of holes

All the holes fall into All the holes fall into two classes by its two classes by its naturenature

• Resampling problemResampling problem

• Missing informationMissing information

All the holes fall into All the holes fall into two classes by its two classes by its naturenature

• Resampling problemResampling problem

• Missing informationMissing information

Resampling

Before warping After warping

Resampling methods

Methods to fill the holesMethods to fill the holes

• Inverse warpingInverse warping

• MeshingMeshing

• SplattingSplatting

• InterpolationInterpolation

Methods to fill the holesMethods to fill the holes

• Inverse warpingInverse warping

• MeshingMeshing

• SplattingSplatting

• InterpolationInterpolation

Splatting

Draw a little cloud (Draw a little cloud (splatsplat) instead of a pixel ) instead of a pixel in desired image. This cloud has to be in desired image. This cloud has to be opaque in its center becoming more and opaque in its center becoming more and more transparent to its sidesmore transparent to its sides

FeaturesFeatures• Relatively small computation costRelatively small computation cost

• Not all the holes are filledNot all the holes are filled

Draw a little cloud (Draw a little cloud (splatsplat) instead of a pixel ) instead of a pixel in desired image. This cloud has to be in desired image. This cloud has to be opaque in its center becoming more and opaque in its center becoming more and more transparent to its sidesmore transparent to its sides

FeaturesFeatures• Relatively small computation costRelatively small computation cost

• Not all the holes are filledNot all the holes are filled

Interpolation

Use the fact that u and v are independent Use the fact that u and v are independent from each other after pre-warping (two-pass from each other after pre-warping (two-pass algorithm) and linearly interpolate depth and algorithm) and linearly interpolate depth and color in the intermediate and final imagescolor in the intermediate and final images

Use the fact that u and v are independent Use the fact that u and v are independent from each other after pre-warping (two-pass from each other after pre-warping (two-pass algorithm) and linearly interpolate depth and algorithm) and linearly interpolate depth and color in the intermediate and final imagescolor in the intermediate and final images

Interpolation (2)

Пример

Пример 1

Пример 2