112/04/18 MC 2009 1

多媒體編碼技術導論Introduction to Multimedia

Coding

Text Book Iain E.G. Richardson, “H.264 and MPEG-4

Video Compression,” John Wiley & Sons, 2003.

References1. Yao Wang, Jorn Ostermann, and Ya-Qin Zhang,

“Video Processing and Communications,” Prentice Hall Signal Processing Series, 2002.

2. Khalid Sayood, “Data Compression,” Morgan Kaufmann, 2nd edition, 2000.

3. M.S. Drew and Z.-N. Li, "Fundamentals of multimedia," Prentice-Hall, 2003.

112/04/18 MC 2009 2

ContentIntroduction

Video Formats and Quality

Video Coding Concept H.261 Variable length coding, DCT, Motion

estimation

The MPEG-4 and H.264 Standards H.264/AVC, Main/High/SVC/Multi-ViewWavelets: Image and Video Coding

Image Coding Standard: JPEG-2000, HD Photo

Voice/Audio Coding Standards: iLBC (Skype) MP3, and AAC Applications and Directions

112/04/18 MC 2009 3

Bernd Girod, Stanford Universityhttp://www.stanford.edu/class/ee398

/

Entropy and lossless coding techniquesRun-length coding, fax standardsArithmetic codingRate-distortion limits and quantizationLossless and lossy predictive codingTransform coding, JPEG standard Subband coding, wavelets, JPEG-2000Motion compensated coding, MPEG standards

112/04/18 MC 2009 4

Learn and PurposeProvide students with a global overview of state-of-the-art in the area of VIDEO and VOICE COMPRESSION

Video: current standards, H.264 and MPEG-4

Voice/Audio: current standards, iLBC (for VoIP), MP3, G.729

Purpose: inspire the students to develop value-added supplementary on H.264 or iLBC

112/04/18 MC 2009 5

Schedule on Experiments

3th-4th weeks: Trace H.261 code5th-12th weeks: Trace H.264 code…

112/04/18 MC 2009 6

評分標準 Midterm Exam.: 35%

Homework: 15%

H.264 program trace, Report: 20% Presentation and Discussion: 30%

112/04/18 MC 2009 7

Applications on H.264/AVC standardsBroadcast over cable, satellite, cable modem, DSL, terrestrial, DVB-H, etc.Interactive or serial storage on optical and magnetic devices: Blu-ray DVD, HD-DVD, etc.Conversational services over ISDN, Ethernet, LAN, DSL, wireless and mobile networks, modems, etc. or mixtures of these.Video-on-demand or multimedia streaming services over ISDN, cable modem, DSL, LAN, wireless networks, etc.Multimedia messaging services (MMS) over ISDN, DSL, Ethernet, LAN, wireless and mobile networks, 3G, etc.

112/04/18 MC 2009 8

Chronological Table of Video Coding Standards

H.261

(1990)

MPEG-1

(1991)

H.263

(1995/96)

H.263+

(1997/98)

H.263++

(2000)

H.264

( MPEG-4

Part 10 )

(2002)MPEG-4 v1

(1998/99)MPEG-4 v2

(1999/00)MPEG-4 v3

(2001)

1990 1992 1994 1996 1998 2000 2002 2003

MPEG-2

(H.262)

(1993/94)ISO/IEC

MPEG

ITU-TVCEG

112/04/18 MC 2009 9

Position of H.264

112/04/18 MC 2009 10

H.264/AVC Profiles (1)

112/04/18 MC 2009 11

H.264/AVC Profiles (2)

112/04/18 MC 2009 12

H.264/AVC Profiles (3) -Scalable Video Coding (SVC)

Functionality of SVC Graceful degradation when “right” parts

of the bit-stream are lost Bit-rate adaptation to match the

channel throughput Format adaptation for backwards

compatible extension Power adaptation for trade-off between

runtime and quality

2007/8 Kai-Chao Yang, NTHU, Taiwan 12

112/04/18 MC 2009 13

Then …MPEG 3D Audio-visual encoding – built on

AVC ? Workshops on future directions in

video compression

Multi-view Video Coding ITU-T VCEG Reduced complexity H.264 ? H.265 with 50% better compression by

2010

112/04/18 MC 2009 14

Terminology, Chapter 1 Sampling: spatial and temporal Frame and field; progressive and raster scan Color space: RGB, YIQ, YCbCr, CMY, … Video format

Analog: NTSC, PAL, SECAM Digital: CCIR601, Sub-QCIF, QCIF, CIF, BT.601(720480/576),

SMPTE295M(19201080) 4:2:0, 4:2:2, 4:4:4

112/04/18 MC 2009 15

Terminology, Chapter 1

112/04/18 MC 2009 16

Terminology, Chapter 1

112/04/18 MC 2009 17

Terminology, Chapter 1Digital Video:BT.601

112/04/18 MC 2009 18

Terminology, Chapter 1

112/04/18 MC 2009 19

Terminology, Chapter 1

112/04/18 MC 2009 20

Subjective Quality Measurement

Visual scene: complex interaction between the Human Visual System, the eye, and the brain

ITU-R BT.500, Double Stimulus Continuous Double Stimulus Continuous Quality ScaleQuality Scale (DSCQS) Mean Opinion Score (MOS) The assessor is presented with a pair of images A

and B, one after the other, and is asked to give A and B a ‘quality score’ with five intervals ranging from ‘Excellent’ to ‘Bad’. The selection of which sequence is A (source) and which is B (impaired) is randomized.

112/04/18 MC 2009 21

Objective Quality MeasurementPeak Signal to Noise Ratio (PSNR)

Other objective quality metrics, ITU-T Video Quality Experts Group (VQEG) Currently, no objective measurement

system is able to replace subjective testing, no one objective model outperforms the others in all cases.

MSEPSNR

n

dB

2

10

)12(log10

112/04/18 MC 2009 22

Motion Vector

Frame i-1 Frame i

Current frame(to be encoded)

Reference frame(existed)

Occlusion

matched

Motion vector

Block matching

112/04/18 MC 2009 23

Motion Compensation & Residuals

motion Residualsocclusion

112/04/18 MC 2009 24

Encoder

MotionCompensation

DCT + Q

iDCT + iQ

MV = (dx, dy)①

② ③

④

Residuals

PreviousFrame Buffer

Encoder (DCT Quantization Binary coding)Residual only

112/04/18 MC 2009 25

Decoder

CodedBitstream VLD

1Q IDCT

MotionCompensation

PreviousFrame memory

Reconstructedframe

MV

Residuals

112/04/18 MC 2009 26

DCT (1)

8

8

139

148

150

149

155

164

165

168

98 115

130

135

143

146

142

147

89 110

125

128

129

121

104

106

96 116

128

132

134

132

113

109

111

125

127

131

137

137

120

110

122

126

126

131

133

131

126

112

133

134

136

138

140

144

141

139

138

139

139

139

140

146

148

147

112/04/18 MC 2009 27

DCT

IDCT

Pixel values in spatial domain

DCT coefficients in transform domain

DC term:The coefficient of the basis vector (0,0)

DCT (2)

112/04/18 MC 2009 28

Does DCT Transform Really Make Sense ?

Energy compaction De-correlation: dependency elimination

112/04/18 MC 2009 29

112/04/18 MC 2009 30

112/04/18 MC 2009 31

Natural Audio Coder

Quality

Cellular

Telephone

AM

FM

CD

2 4 8 16 32 64 kbit/s

Parametric speech(HVXC)

High quality speech(CELP)

General audio(AAC, TwinVQ)

Parametric audio(HILN)

From Olivier Dechazal