Advance in Scalable Video Advance in Scalable Video CodingCoding

Proc. IEEE 2005,Proc. IEEE 2005,Invited paperInvited paper

Jens-Rainer Ohm, Member, IEEEJens-Rainer Ohm, Member, IEEE

NTU CSIE, CMLab謝俊瑋 2

OutlineOutline

IntroductionIntroduction Scalability in Existing StandardScalability in Existing Standard Principles of Scalable Predictive CodingPrinciples of Scalable Predictive Coding Drift controlDrift control Interframe Wavelet CodingInterframe Wavelet Coding ConclusionConclusion

NTU CSIE, CMLab謝俊瑋 3

IntroductionIntroduction

Scalable video coding is attractive due to the Scalable video coding is attractive due to the capability of reconstructing lower resolution or capability of reconstructing lower resolution or lower quality signals from partial bit streams.lower quality signals from partial bit streams.

A simple and flexible solution for transmission A simple and flexible solution for transmission over heterogeneous network.over heterogeneous network.

Allow simple adaptation for a variety of Allow simple adaptation for a variety of storage devices and terminals.storage devices and terminals.

NTU CSIE, CMLab謝俊瑋 4

Scalability in Existing StandardScalability in Existing Standard

MPEG-2MPEG-2 First general-purpose video compression standard First general-purpose video compression standard

which also include tools providing scalability.which also include tools providing scalability. Layered coding, support spatial, temporal, SNR Layered coding, support spatial, temporal, SNR

scalability.scalability. Number of layers is restricted to maximum of 3.Number of layers is restricted to maximum of 3.

NTU CSIE, CMLab謝俊瑋 5

Scalability in Existing Standard(2)Scalability in Existing Standard(2)

MPEG-4MPEG-4 More flexible scalability tools, including spatial More flexible scalability tools, including spatial

and temporal scalability within a more generic and temporal scalability within a more generic framework.framework.

SNR scalability with fine granularity and SNR scalability with fine granularity and scalability at level of scalability at level of video objects.video objects.

AVC can in principle be run in different temporal AVC can in principle be run in different temporal scalability modes.scalability modes.

NTU CSIE, CMLab謝俊瑋 6

Principles of Scalable Principles of Scalable Predictive CodingPredictive Coding

Prediction should not use any decoded Prediction should not use any decoded information from higher layers, otherwise drift information from higher layers, otherwise drift effect would occur.effect would occur.

However the rate-distortion performance However the rate-distortion performance toward higher rates will be worse than single-toward higher rates will be worse than single-layer coder.layer coder.

NTU CSIE, CMLab謝俊瑋 7

NTU CSIE, CMLab謝俊瑋 8

Drift controlDrift control

It is possible to track the drift within the local It is possible to track the drift within the local loop of the encoder that would occur in a loop of the encoder that would occur in a decoder only receiving the base-layer decoder only receiving the base-layer information.information.

Drift compensationDrift compensation Drift clippingDrift clipping Drift leakingDrift leaking

NTU CSIE, CMLab謝俊瑋 9

Drift compensationDrift compensation

It assume that the decoder side is not aware of It assume that the decoder side is not aware of the drift compensation made.the drift compensation made.

Usual MC decoder loop could be used without Usual MC decoder loop could be used without any modifications.any modifications.

The encoder have to find the balance between The encoder have to find the balance between the penalties for the base and enhancement the penalties for the base and enhancement decoding.decoding.

NTU CSIE, CMLab謝俊瑋 10

Drift clippingDrift clipping

The drift is dynamically limited if a maximum The drift is dynamically limited if a maximum value Dvalue Dmaxmax is reached. is reached.

A good choice for DA good choice for Dmaxmax is approximately by the is approximately by the

base-layer quantizer step size.base-layer quantizer step size.

NTU CSIE, CMLab謝俊瑋 11

Drift leakingDrift leaking

The accumulation of drift is limited by The accumulation of drift is limited by multiplying D’ = a * D (a < 1)multiplying D’ = a * D (a < 1)

The best selection of drift coefficient is The best selection of drift coefficient is dependent on the operational target and dependent on the operational target and sequence characteristic.sequence characteristic.

NTU CSIE, CMLab謝俊瑋 12

NTU CSIE, CMLab謝俊瑋 13

Interframe Wavelet CodingInterframe Wavelet Coding

To overcome the limitation which are caused To overcome the limitation which are caused by drift problem, it would be desirable to by drift problem, it would be desirable to discard the temporal recursion.discard the temporal recursion.

Motion-Compensated Temporal FilteringMotion-Compensated Temporal Filtering

NTU CSIE, CMLab謝俊瑋 14

Spatiotemporal wavelet Spatiotemporal wavelet decompositiondecomposition

NTU CSIE, CMLab謝俊瑋 15

Spatiotemporal wavelet decompositiSpatiotemporal wavelet decomposition (2)on (2)

NTU CSIE, CMLab謝俊瑋 16

Lift structureLift structure

NTU CSIE, CMLab謝俊瑋 17

Operation of Motion CompensationOperation of Motion Compensation

NTU CSIE, CMLab謝俊瑋 18

NTU CSIE, CMLab謝俊瑋 19

ConclusionConclusion The fully open loop property of MCTF The fully open loop property of MCTF

provides high flexibility in bit stream provides high flexibility in bit stream scalability.scalability.

Combination of MCTF with the new drift-Combination of MCTF with the new drift-controlled prediction strategies is also a controlled prediction strategies is also a promising path.promising path.

Decoder could integrate additional signal Decoder could integrate additional signal synthesis element whenever the received synthesis element whenever the received information is incomplete.information is incomplete.

NTU CSIE, CMLab謝俊瑋 20

Conclusion (2)Conclusion (2)

When low delay is required, the update step When low delay is required, the update step must be omitted, or the number of temporal must be omitted, or the number of temporal wavelet decomposition levels must be low.wavelet decomposition levels must be low.

Seamless transition between intraframe and Seamless transition between intraframe and interframe coding methods.interframe coding methods.