Robust global motion estimation and novel updating strategy for sprite generation

IET Image Processing, Mar. 2007.H.K. Cheung and W.C. SiuThe Hong Kong Polytechnic Univ. ( 香港理工大學 )

Outlines

Overview / Introduction Proposed system

New global motion estimation Combing short- and long-term estimation Dynamic reference frame

2-pass sprite blending Preserving frame resolution loss

Sprite updating Overcoming illumination variations & object

changing Experimental results Conclusions

Overview

Overview

Sprite High resolution image Composed of information belonging to an object

visible throughout a video sequence Background of a scene

Overview

Sprite

background of frame 1(Dimension: 352x288)

background of frame 20

Sprite(Dimension: 2670x1072)

Overview

Core of sprite generation Global motion estimation (GME)

Finding a set of parameters representing camera motion between frames Image registration Iterative minimization

Blending Temporal (weighted) averaging, median, updating

Introduction

Introduction

Global motion estimation Image registration Short-term motion estimation

Estimation between consecutive frames Easy and accurate

Long-term motion estimation Estimation between frames with temporal

distance Harder Required to perform sprite coding

Single sprite for all frames in sequence

Introduction

Global motion estimation (cont.) Short- to long-term estimation

Converting short-term motion parameters to long-term parameters

Error propagation Directly long-term estimation

Estimation every frames directly to a specified base frame (reference frame) No error propagation

Search range may be huge Hard to find overlapping area

Introduction

Global motion estimation (cont.) Hierarchical estimation

Rough estimation to find coarse parameters Refining parameters

Using coarse parameters as initials Iterative minimization

Some existing methods Dufaux and Konrad Szeliski Smolic et. al. Lu et. al.

Introduction

Restrictions Background must be really static

Background objects must be still No illumination variations

Dynamic sprite

Introduction

Classification Static sprite

Build offline before coding individual frames Quality degradation as frame increases

Motion estimation errors Illumination variations Background object changes

Dynamic sprite Built dynamically online in both encoder and

decoder while coding individual frames Sprite is updated using reconstructed frame

Short-term estimation is employed Error accumulated

Introduction

Proposed system New global motion estimation

Directly estimating the relative motion between current image and a chosen reference frame Give accurate, stable and robust estimation Alleviate error accumulation

Hierarchical 3-levels approach Coarse-to-fine approach

Sprite updating Updating sprite only if necessary

Sprite update frames are generated and sent

Proposed system

Proposed system

Short-term GME to long-term GME

Frame 1A11

Frame mAm1

Frame m+1

GME

reference frame

……

A(m+1)1

Am1+

A(m+1)m Registration Error

= A(m+1)m Am(m-1) … A21

A(m+1)k = A(m+1)m Am1

Registration Error

Registration errors are ACCUMULATED

More Error

Proposed system

Directly measure to reference frame

GME

Frame 1A11

Frame mAm1

Frame m+1

reference frame

……

A(m+1)1

Am1

initial guessRegistration

Error

Registration Error

Registration errors are COMPENSATED

Proposed system

Weakness Reference frame is temporally far from current frame

Frame contents may change largely Background objects activities Lighting conditions changes

Overlapping area could be smaller Unfavorable to GME

Proposed system

Combining the advantages Dividing video into groups of consecutive frames 1st frame of each group is selected as reference

Frames in a group Each frame is directly measured to the 1st frame

Smaller registration error Merging groups

GMEs of reference frames of all groups are merged Registration error is slightly increased

R1 R2 R3…… ……

+ +A(R1)(R1)

A(R2)(R1) A(R3)(R2)

A(R2)(R1) A(R3)(R1)

Proposed system

Proposed GME structure

MotionEstimation

Frame kAk1

Frame mAmk Am1

Frame m+1

Frame z

Chosen to bereference frame

……

A(m+1)k

Amk

Proposed system

Dynamic reference frame 1st frame is the initial reference frame Assigning current frame as new reference frame if

Displaced frame difference between registered current frame and the reference frame it large Reference frame is not like current frame

Relative displacement between current frame and the reference frame is large Overlapping area is too small

21area goverlappin ThwT

NrhwT 2area goverlappin-nonor

where Nr is a parameter between 0 and 1 (Nr=0.1 in practical)

Proposed system

Advantages Accuracy

Accurate than short-term and directly long-term estimation

Very few memory usage Estimations are performed frame-to-frame Sprite building is not necessary

Proposed system

GME

Reference frame(frame k)

Frame z

Three step search

Block-based partialdistortion search

Fast gradient method

A(m+1)k

Amk

+

Proposed system

Motion model Perspective motion model

8 motion parameters to be determined

Three-step matching 3-level pyramids for frame k and z are built using

Gaussian down-sampling filter [¼, ½, ¼]

k

y

x

hg

fed

cba

k

y

x

1'

'

'

frame k: reference frameframe z: transformed current frame m+1

Proposed system

Block-matching Affine parameters are estimated by solving over-fitting eq

uations

Results of block-based motion estimation are used to construct the equations

Parameter estimation Fast gradient descent method by Keller and Averbuch

'),,(

'),,(

yWkyx

xVkyx

1'

),,(

),,(

kk

fedW

cbaVT

T

where

Proposed system

Two-passed blending to avoid resolution loss First pass: 1st frame as base frame

All frames are projected into 1st frame Frame with minimal area of projected frame is

selected as new base frame Avoiding resolution loss

No real pixel blending applied Second pass: new base frame

All frames are projected into new base frame Simple temporal average blending

With bilinear interpolation

Proposed system

Dynamic sprite updating Overcoming illumination variations

Single value in sprite can not represent intensity variations over the time

Accumulation of GME error blurring the frame GME error in a reference frame will inherit into all

of frames in the group

Proposed system

Studying the generated intensity error

an edge pixela pixel from

homogeneous areaa pixel

from texture area

translation in x-direction# of pixel withsignificant error

Proposed system

Distribution of intensity error correlates roughly to the panning motion Errors tends to be clustered in the temporal domain

Errors of homogeneous and texture regions are tend to randomly around zero

Proposed system

Sprite updating Selecting frames with significant change in panning

direction/speed

0 51 108 174 206

Proposed system

Sprite updating (cont.)

Reconstruct next N frame from the sprite

Blend the N error frames into a sprite-sized buffer(the sprite update frame)

Compute the N error frames

Encode and send the sprite update frameto the decoder

MPEG4 I-VOP frame

Experimental results

Experimental results

Testing Constructing sprite Reconstructing frames from sprite Compute PSNR

Comparison Short-term motion estimation

Estimating between current and previous frame Long-term motion estimation

Estimating between current frame and sprite No parameters predicting

Long-term motion estimation by MPEG-4 VM Long-term motion estimation by Smolic et. al.

Experimental results

Short-term

Long-term

Experimental results

MPEG-4 VM

Proposed method

Experimental results

PSNR

Proposed

MPEG-4

Short-term

Long-term

Smolic et. al.

Experimental results

Average PSNR (dB)

Short-term

Long-termMPEG-4

VMSmolicet. al.

Proposed(affine)

Proposed(per-specti

ve)

Stefan (150)

18.955 19.058 20.889 20.347 22.046 22.645

Foreman (150)

27.941 28.432 28.057 26.973 28.458 28.305

Coast Guard (150)

22.294 22.543 23.586 20.213 23.538 23.450

Stefan (300)

19.152 20.442 18.871 Failure 21.364 21.311

Experimental results

Selecting threshold Nr Proposed method is better than simple short-term

and long-term estimation

Short-term

0.1 Long-term

Experimental results

Performance of sprite updating

Sequence Update frames Average PSNR (dB) Size of updates (kB)stefan - 21.133 -

stefan 0,51,108,174,206* 22.319 84.5

stefan 0,60,120,180,240 22.265 79.1

stefan 0,51,108,106* 22.215 74.0

stefan 0,80,160,240 21.994 59.5

coast guard - 23.538 -

coast guard 0,76* 24.085 7.42

foreman - 28.758 -

foreman 0,10,25,64,110* 30.590 11.7

foreman 0,30,60,90,120 30.714 12.1

* Update frames is figured out from the major camera operations of the sequences

Conclusions

Conclusions

New global motion estimation method Directly estimation from current frame to a chosen

reference frame Combing advantages of short-term and long-term

estimation Error accumulation prevented Keeping reference frame close to current frame

Sprite updating Encoding & sending sprite update frames

Errors of a group of reconstructed frames Reducing sprite blurring