Seminar

Seminar

A Digital Image Stabilization Method Based

on the Hilbert–Huang Transform

Konstantinos Ioannidis and Ioannis AndreadisIEEE TRANSACTIONS on Instrumentation And Measurement,

VOL. 61, NO. 9, SEPTEMBER 2012

Presented by

Shibudas K Subhashdas

1/22

Introduction

An innovative technique for digital image stabilization

(DIS) based on the Hilbert Huang transform (HHT)

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

2/22

Proposed method in a Nutshell

Local Motion Vector (LMV) Estimation

Hilbert Huang Transform ( HHT)

Empirical Mode Decomposition (EMD) process

Hilbert Transform

Detect the unwanted Camera Motion

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

3/22

Local Motion Vector (LMV) Estimation

LMV represents the offset of specific image regions be-

tween two consecutive images (displacement vector)

LMV includes intentional and unwanted motion of the

camera

Block based motion estimation used in LMV

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

4/22

Local Motion Vector (LMV) Estimation

The criteria for selecting the best match is sum of absolute

difference (SAD)

1 1

21 0 2 0

1( , ) [ ( 1, 2, ) ( 1 , 2 , 1)]

N N

n n

SAD i j s n n k s n i n j kN

S(0,0)S(0,1)S(1,0)S(1,1)

Current Frame

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

5/22

Previous Frame

Local Motion Vector (LMV) Estimation

Minimum SAD value defines the displacement vector

Full search (FS) algorithm is used in this method

[ 1, 2] arg min[ ( , )]d d SAD i j

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

6/22

Local Motion Vector (LMV) Estimation

Comparison of Full search (FS) algorithm with other

search method

three step search algorithm

Four step search algorithm

Diamond search algorithm

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

7/22

Local Motion Vector (LMV) Estimation

Using LMV estimation - x(t) (displacement vector)

[ 1, 2] arg min[ ( , )]d d SAD i j

1 1

21 0 2 0

1( , ) [ ( 1, 2, ) ( 1 , 2 , 1)]

N N

n n

SAD i j s n n k s n i n j kN

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

8/22

Hilbert Huang Transform ( HHT)

Empirical Mode Decomposition (EMD) process • Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

9/22

Hilbert Huang Transform ( HHT)

Empirical Mode Decomposition (EMD) process

11

( ) ( )( )

2

U t L tm t

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

10/22

Hilbert Huang Transform ( HHT)

Empirical Mode Decomposition (EMD) process Shifting Process

This will continue until value sum of difference (SD)

reaches between 0.2 and 0.3

1 1( ) ( ) ( )h t x t m t

11 1 11( ) ( ) ( )h t h t m t

12 11 12( ) ( ) ( )h t h t m t

1 1( 1) 1( ) ( ) ( )k k kh t h t m t

21

0

21

0

| ( ) ( ) |

( )

T

k kt

T

kt

h t h tSD

h t

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

11/22

Hilbert Huang Transform ( HHT)

Empirical Mode Decomposition (EMD) process If 0.2<= SD <=0.3

c1= h1k be the first IMF (Intrinsic Mode function)

c1 (t) is removed from the image to get residue

r1 is the residue

Residue is treated as the new data and subjected to same shifting process to give c2(t)

1 1( ) ( ) ( )r t x t c t

2 1 2( ) ( ) ( )r t r t c t

1( ) ( ) ( )w w wr t r t c t Cw(t) is the wth

IMF

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

12/22

Hilbert Huang Transform ( HHT)

Empirical Mode Decomposition (EMD) process

When the residue rw becomes a monotonic function

from which no IMF can be extracted.

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

13/22

Hilbert Huang Transform ( HHT)

Empirical Mode Decomposition (EMD) process

Sum of the IMFs and the residue recovers the original

signal.

1

( )w

j wj

x t c r

1 1

11 1

( )w w

j w w j wj j

x t c c r c r

2 1

1 1 21 1

w w

j w w j wj j

c c r c r

1

2 2 1 11

( )jj

c c r c r x t

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

14/22

Hilbert Huang Transform ( HHT)

Hilbert Transform

It is used to compute instantaneous frequencies and

amplitudes

Where P denotes Cauchy principal value

z(t) analytical signal is given by

Where α(t) – Instantaneous Amplitude

θ(t) – Instantaneous Phase

1 ( )( ( )) ( )

xH x t y t P d

t

( )( ) ( ) ( ) ( ) i tz t x t iy t t e

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

15/22

Hilbert Huang Transform ( HHT)

Hilbert Transform

Instantaneous Amplitude

Instantaneous Phase

Instantaneous Frequency

2 2( ) ( ) ( )t x t y t

1 ( )( ) tan

( )

y tt

x t

1 ( )( )

2

d tf t

dt

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

16/22

Hilbert Huang Transform ( HHT)

Detect the unwanted Camera Motion

EMD process divides the initial signal into finite

number of sub signals based on their frequencies

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

17/22

Hilbert Huang Transform ( HHT)

Detect the unwanted Camera Motion

The power of each IMF is proportional to the ampli-

tude of its sample.

Where αi – amplitude of a IMF’s

i = 1,2 ….. w+1

t – frame number

2

0

( )K

it

Pi t

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

18/22

Hilbert Huang Transform ( HHT)

Detect the unwanted Camera Motion

IMF with lower indices include high frequency com-

ponent (jitter)

From lower to higher IMFs , the energy content is re-

duced

After a certain IMF , a sharp increase of the energy

occur due to intentional camera motion

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

19/22

Hilbert Huang Transform ( HHT)

Detect the unwanted Camera Motion

IMF with higher index and lower energy content to

the last IMF which includes jitter components

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

20/22

Hilbert Huang Transform ( HHT)

Detect the unwanted Camera Motion

Jitter and intentional camera motion

where

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

21/22

1

( ) ( )d

J ii

X t c t

( ) ( )w

G i wi d

X t c t r

arg min[ ]id P

Experimental Results

2 2

1

1( ) ( )

N

rms n n n nn

e x x y yN

Where N is the number of frames

Optimal camera motion

Resulting camera motion,( )n nx y

( , )n nx y

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

22/22

Conclusion

DIS method based on HHT has been presented

Jitter signal is defined based on its two principle feature

High frequency

Low energy

Experimental results have proven that the proposed

method can successfully decompose two camera motions

• Introduction

• Proposed method in a Nutshell

• Local Motion Vector (LMV) Es-

timation

• Hilbert Huang Transform( HHT)

• Empirical Mode Decomposi-

tion (EMD) process

• Hilbert Transform

• Detect the unwanted Camera

Motion

• Experimental Results

• Conclusion

23/22