8/8/2019 2-3_MPEG 3D (TTA)

1/28

MPEG 3D

2010.5.11.

8/8/2019 2-3_MPEG 3D (TTA)

2/28

Contents

Depth cue in human visual system

MPEG standard overview MPEG-2 MVP

MPEG-4 MVC

MPEG-C Part 3 3DV

Overall Summary

1

8/8/2019 2-3_MPEG 3D (TTA)

3/28

3 - Depth Cue in the human visual system (1/2) (psychological cues)

(Linear Perspective) (Overlapping)

(Shades and Shadows) (Texture Gradient)

2

8/8/2019 2-3_MPEG 3D (TTA)

4/28

3(2/2) (physiological cue)

(Accommodation) (Binocular Parallax)

(Convergence) (Motion Parallax)

3

8/8/2019 2-3_MPEG 3D (TTA)

5/28

Stereoscopic Viewing Display

4

(Binocular Parallax) /

(Disparity) . 3.

Stereoscopic Viewing Display, /.

,

.

!

8/8/2019 2-3_MPEG 3D (TTA)

6/28

Consideration of redundancy in video compression

2D Video Compression

Spectral redundancy: Color Sub-samplingSpatial redundancy: Frequency domain(DCT+Quantization)

Temporal redundancy: Inter-frame

Statistical redundancy: Entropy coding

3D Video Compression??Spectral redundancy: Color Sub-sampling

Spatial redundancy: Frequency domain

Temporal redundancy: Inter-frame, Inter-view

Statistical redundancy: Entropy coding

5

8/8/2019 2-3_MPEG 3D (TTA)

7/28

MPEG Standard Overview

6

ISO/IEC JTC1 SC29/WG11 = MPEG

MPEG-1: the standard for storage and retrieval of moving pictures and audio on storage media (approved Nov.1992)MPEG-2: the standard for digital television (approved Nov. 1994)MPEG-4: the standard for multimedia applicationsMPEG-7 :the content representation standard for multimedia information search and filteringMPEG-21: the multimedia frameworkMPEG-A: the collection of standards for Application FormatsMPEG-B: the collection of Systems-related standardsMPEG-C: the collection of Video-related standardsMPEG-D: the collection of Audio-related standardsMPEG-E: the Multimedia Terminal standardMPEG-M: the standard for packaging and reusability of MPEG technologiesMPEG-U: the standard for rich-media user interfaces

MPEG-V: the standard for real and virtual worlds, and for their interactions3D Video Coding: the standard for coding 3D Visual informationAdvanced IPTV Terminal: the standard for digital media ecosystemsHigh Efficiency Video Coding: the standard for a new frontier in video coding.

MPEG-4: in its different components (Systems, Video, Audio, 3D Graphics, Composition, Fonts etc.)MPEG-7: particularly MPEG Query FormatMPEG-C: particularly Reconfigurable Video CodingMPEG-D: particularly Universal Speech and Audio Coding

System(Transport, Storage, Representation)

Media(Video, Audio)

: http://mpeg.chiariglione.org/who_we_are.htm

8/8/2019 2-3_MPEG 3D (TTA)

8/28

Video Coding Standard of MPEG

7

Video CD,Internet,

VHS quality:< 1.5Mbps

H.261v1

MPEG-1 MPEG-2 MPEG-4 MPEG-4MPEG-4

AVC

H.262

H.263 H.263 H.263 H.26L H.264

ITU-TVCEG

ISO/IECMPEG

Digital Broadcasting,DVD, Digital CamcorderHigh quality:1.5~80 Mbps

Video phone: PSTN, B-ISDN

Low quality: 64kbps~1.5Mbps

Multimedia AuthoringMobile communicationInternet streamingVarious quality: 64kbps~2Gbps

JVTH.264/AVC, SVC, MVC

1992 1996 2000 2004

?

?

JCT-VCHVEC

2010

8/8/2019 2-3_MPEG 3D (TTA)

9/28

3D Video Coding Standard of MPEG

8

MPEG-2MPEG-4

AVC

H.262 H.264ITU-TVCEG

ISO/IECMPEG

JVT

MVC(Multiview Video Coding)

1992 1996 2000 2004 2010

MPEG-2MVP(Multi-ViewProfile

MPEG-CPart 3

2007

Representation ofauxiliary video andsupplementalinformation

3DV(3D Video

Coding) cont.

8/8/2019 2-3_MPEG 3D (TTA)

10/28

Summary: Stereo Video

L/R simulcast possible with any MPEG standard

MPEG-2 Multi-view profile is essentially stereo withtemporal L/R interleaving

Stereoscopic MAF ISO/IEC 23000-11 based on MPEG-4part 2 video (L/R packing, for handhelds)

MPEG-4 part 10 AVC Stereo SEI message and FramePacking Arrangement SEI message (the latter in 14496-10/5e Amd.1, to be finalized by July 2009) allow variousmethods of L/R packing

Temporal, spatial row/column, spatial side-by-side/up-and-bottom,checkerboard (quincunx)

MPEG-4 AVC Stereo High Profile (new in Study 14496-10/5e Amd.1, to be finalized by July 2009)

Subset of MVC, restricted to 2 views, allows progressive and interlaced

stereo 9

8/8/2019 2-3_MPEG 3D (TTA)

11/28

MPEG-2 Multi-view Profile(MVP)

Two-Layer Video Coding Scheme

Base layerAssigned to Left View Video

MPEG-2 Main Profile(MP) Video Coding

Enhancement Layer

Temporal Scalability Video Coding(Disparity estimation+Motion estimation)

Assigned to Right View VideoSame spatial resolution in both layers

Forward compatibility and Backward compatibility

Decoder for MVP can process MP(Forward compatibility)

Decoder for MP can process Base layer of MVP(Backward compatibility)

Support a bit stream syntaxincluding camera position for generating new scene from any other angle

10

8/8/2019 2-3_MPEG 3D (TTA)

12/28

Enhancement layer predict mode

11

For P picture,

For B picture,

Disparityestimation

8/8/2019 2-3_MPEG 3D (TTA)

13/28

MPEG-2 MVP Performance

12

: PSNR(dB)

Up to 1.6dB over the simulcast approach

Consideration of enhancement techniques(1) Brightness balancing of two views for disparityestimation and compensation;

(2) Horizontal view offset for disparity estimation andcompensation;(3) Rate control for stereoscopic video encoding

8/8/2019 2-3_MPEG 3D (TTA)

14/28

Multi-view Video Coding(MVC)

13

Standard was approve in July 20081. Specified as an amendment of H.264/AVC2. Integrated into 5th Edition of ISO/IEC 14496-10 (Annex H)

8/8/2019 2-3_MPEG 3D (TTA)

15/28

MVC scheme

Extension of the H.264/AVC for multiple view video

Temporal/Inter-view prediction video coding scheme

Temporal prediction video coding schemeHierarchical B picture structure

Inter-view prediction video coding scheme

Key-picture/Nonkey-picture prediction structure

14

Example of multiview video data with linear camera arrangement

8/8/2019 2-3_MPEG 3D (TTA)

16/28

Prediction mode evaluation result

Temporal/Inter-view prediction mode

15

Temporal prediction axis

Inter-view prediction axis

Probability of chosen predictor when minimizing a Lagrangian cost function in

motion estimation for sequences "Uli" and "Breakdancers".

Probability of temporal mode

Probability of Inter-view mode

8/8/2019 2-3_MPEG 3D (TTA)

17/28

The structure of MVC

16

Temporal prediction using Hierarchical B pictures

Inter-view prediction for key pictures

MVC basic structure: Inter-view prediction for key/nonkey picture

8/8/2019 2-3_MPEG 3D (TTA)

18/28

Coding order of MVC Structure

17

: http://ip.hhi.de/imagecom_G1/cod_pattern.htm

8/8/2019 2-3_MPEG 3D (TTA)

19/28

MVC performance & summary

18

Ballroom sequence Race-1 sequence

Up to 3.2 dB better than anchor coding(MVC) Half of coding gain when using Hierarchical B pictures(Simulcast) High Level Syntax

1. Improved random access2. Low delay3. Memory optimization

Limitation/Issues1. Acquision/production with large camera array is not common2. Although more efficient than simulcast, rate of MVC is still proportional to the

number of views: Varies with scene, camera arrangement etc.

~25%

8/8/2019 2-3_MPEG 3D (TTA)

20/28

Depth Based Rendering

19

3D warporiginal Depth map

Virtual view

8/8/2019 2-3_MPEG 3D (TTA)

21/28

MPEG-C Part 3:ISO/IEC 23002-3

20

Video+Depth as data representation for 3DTV Initiative driven by Philips, FhG-HHI and other partners as a result of

ATTEST project Define a simple container format

Not specify transport and compression techniques Finalize at January 2007 FDIS ISO/IEC 23002-3 Representation of Auxiliary Video and Supplemental

Information ISO/IEC 13818-1: 2003 Carriage of Auxiliary Data

8/8/2019 2-3_MPEG 3D (TTA)

22/28

Extension to 3DV: Current

MVC

Usage of N views

No continuum

For large N very inefficient

MPEG-C part 3Disocclusion artifacts increase with distance of virtualview from available original view

Does not support wide range multi-view 3D displays

Very limited free viewpoint navigation

21

8/8/2019 2-3_MPEG 3D (TTA)

23/28

3DV Data Format

22

8/8/2019 2-3_MPEG 3D (TTA)

24/28

Bit rate vs 3D Rendering Capabilities

23

8/8/2019 2-3_MPEG 3D (TTA)

25/28

3D Video Framework

24

8/8/2019 2-3_MPEG 3D (TTA)

26/28

3DV summary

Main Objectives

Support auto-stereoscopic displays from a limitednumber of input views and also variable baseline forstereo processing

Inclusion of depth: decouple number of transmitted

views with number of required views for display

MPEG exploration underwayIn the process of establishing suitable reference

Gathering available Test multi depth and video sequence

To subjective quality testing, Depth Estimation/View Synthesis

Anchor Coding Experiment

Expecting to issue Call for Proposals at Oct. 2010

25

8/8/2019 2-3_MPEG 3D (TTA)

27/28

Overall summary

MEPG has actively contributed compressiontechnology for stereo and multi-view video and isconsidering to take the next steps towards 3Dand free-viewpoint video

In 3D video part, we are always trying to definegeneric formats that support to high-fidelity andcompatibility with other standard(not easy!)

ETRI, Samsung, LG and GIST are very activelyparticipating in 3DV group

26

8/8/2019 2-3_MPEG 3D (TTA)

28/28

Reference

Chen, Xuemin; Luthra, Ajay, MPEG-2 Multi-View Profile and its application in 3DTV,Proc. SPIE Vol. 3021, p. 212-223, Multimedia Hardware Architectures 1997.

Philipp Merkle, Aljoscha Smolic, Karsten Mller, T.Wiegand, Efficient PredictionStructures for Multiview Video Coding, IEEE TRANSACTIONS ON CIRCUITS ANDSYSTEMS FOR VIDEO TECHNOLOGY, VOL. 17, NO. 11, NOVEMBER 2007

P. Merkle, K. Mller, A. Smolic, and T. Wiegand, EFFICIENT COMPRESSION OF MULTI-VIEW VIDEO EXPLOITING INTER-VIEW DEPENDENCIES BASED ON H.264/MPEG4-AVC,IEEE International Conference on Multimedia and Expo (ICME'06), Toronto, Ontario,

Canada, July 2006.Video and Requirements, Applications & Requirements on 3D video coding, ISO/IECJTC1/SC29/WG11 Doc. N11061, Xian, China, October 2009.

27