Transporting Real-Time Video Over the Internet: Challenges

Transporting Real-Time Transporting Real-Time Video Over the Internet: Video Over the Internet: Challenges and Challenges and ApproachesApproaches

Dapeng Wu, Yiwei Thomas Hou, and Ya-Qin ZhangPROCEEDINGS OF THE IEEE, VOL. 88, NO. 12,DECEMBER 2000

指導教授 : 林永松博士

台大資管所碩一 R89725048 宋之揆

2

Introduction

Internet multimedia applications: video conference, distance learning, digital libraries, and video-on-demand.There is no quality of service (Qos) guarantee for video transmission over the current Internet.

3

Challenging QoS issues

Bandwidth: Current Internet does not provide bandwidth reservation Delay: Real-time video requires bounded end-to-end delay. Internet does not offer delay guarantee.Loss: Packet loss ratio required to be kept below a threshold. Internet does not provie any loss guarantee.

4

Unicast and Multicast video distribution

5

Tradeoff between bandwidth and flexibility

6

Another challenge: Heterogeneity

Network heterogeneity: Different processing, bandwidth, storage, and congestion control policies.Receiver heterogeneity: Receivers have different latency requirement, visual quality requirement, and processing capability.

7

Two general approaches to the above challenges

Network-centric: Routers/switches in the network are required to provide QoS support.*End system based: Guarantee QoS without imposing any requirements on the network.

8

Congestion control

Rate controlRate-adaptiveVideo encodingRate shaping

9

Rate control

Windows-based: Like TCP, using congestion window. Increase the window slowly, but decrease the window greatly. (may introduce intolerable delay) *Rate-based: Source-based, receiver-based, and hybrid rate control

10

Source-based rate control

The sender is responsible for adapting the transmission rate of the video stream. It can minimize the amount of packet loss by matching the rate of the video stream to the available network bandwidth.Feedback is employed to convey the changing status of the Internet.hhg

11

Source-based approach for unicast (cont’d)

<1> Probed-based approach: additive increase and multiplicative decrease (AIMD) MIMD

}),max{(:

}),min{(:

)(

MinRrr

else

MaxRAIRrr

Ppif th

p: packet loss ratio

Pth: threshold for the packet loss ratio

AIR: additive increase rate

r: sending rate at the source

α: multiplicative decrease factor

12

Source behavior under the AIMD rate control

13

<2> Model-based approach: also called TCP friendly rate control

Source-based approach for unicast (cont’d)

pRTT

MTU

22.1

λ: throughput of a TCP connection

MTU: maximum transit unit

RTT: round trip time

p: packet loss ratio

14

Single channel Multicast

Only the probe-based rate control can be employed

}),min{(:

%)100(

),2/max(:

)(

MaxRAIRrr

Felseif

MinRrr

TFif

un

concon

Fcon: fraction of congested

Fun: fraction of unloaded

Tcon: threshold

AIR: additive increase rate

r: sending rate at the source

15

Receiver-based control

Target at solving heterogeneity problemLayered multicast videoProbe-based approach and model-based approach

16

Layered multicast video

Raw video sequence is compressed into multi layers: a base layer and one or more enhancement layers

17

Probe-Based approach

When no congestion, a receiver probes for the available bandwidth by joining a layer, which leads to an increase of its receiving rate.When congestion id detected, the receiver drops a layer, resulting in reduction of its receiving rate.

18

Model-based approachγi: transmission rate of layer i

19

Hybrid rate control

Receiver regulate the receiving rate of video streams by adding/dropping channels while sender also adjusts the transmission rate of each channel based on feedback information from the receiver.

20

Rate-adaptive video encoding: A compression approach

Video conference with H.261 and H.263MPEG-1, MPEG-2, and MPEG-4Maximize the perceptual quality under a given encoding rate.Encoder’s quantization parameter (QP)

21

Rate-distortion (RD) theory

22

RD theory approachesModel-based approach: Assumes various input distribution and quantizer characteristics. Close-form solutions can be obtained by using continuous optimization theory.Operational R-D based approach: The admissible quantizers are used by the rate control algorithm to determine the optimal strategy to minimize the distortion under the constraint of a given bit budget.

23

Rate shaping

A rate shaper is a filter between the encoder and the network, with which the encoder’s output rate can be match to the available network bandwidth.It is applicable to any video coding scheme and both live and stored video.

24

Rate shaping approaches (cont’d)

Transport perspective: Server selective frame discard. Two advantage: Taking the network bandwidth and client buffer constraint into account; take advantage of application-specific information such as regions of interest and group of pictures structure, in its decision in discarding frames.

25

Rate shaping approaches (cont’d)

Compression perspective: Based on the R-D theory, the dynamic rate shaper selectively discards the discrete cosine transform (DCT) coefficients of the high frequency so that the target rate can be achieved.

26

Error control

Forward error correction (FEC)RetransmissionError resilience

both the source and receiver sideError concealment

only receiver side

27

FEC

Add extra information to a compressed video bit stream.Channel codingSource coding-based FECJoint source/channel coding

28

Channel coding FECDisadvantages: It increases the transmission rate and delay. It is not adaptive to varying loss characteristics and works best only when the packet loss rate is stable.Unequal error protection: In MPEG I -frame >P -frame >B -frameHierarchical FEC: The FEC stream is used for recovery of a different video layer. More flexibility and bandwidth efficiency.

29

Source coding-based FEC

The redundant information added by SFEC is more compressed versions of the raw data.SFEC recovers the video with reduced quality.Advantage: Lower delayDisadvantage: It increases the transmission rate and is inflexible to varying loss character.

30

Joint source/channel coding

31

Retransmission (cont’d)

32


33


Hybrid control: It could achieve better performance at the cost of higher complexity.Multicast: Retransmission has to be restricted within closely located multicast member.

34

Optimal mode selection

Intermode vs Intramode

35

Multiple description coding

A raw video sequence is compressed into multiple streams.Advantages:

<1>robustness to loss: even if a receiver gets only one description, it can still reconstruct video with acceptable quality.

<2>enhance quality

36

Error concealment: A compression approach

Human eyes can tolerate a certain degree of distortion in video signals, error concealment is a viable technique to handle packet loss.Spatial and temporal interpolation

37

Error resilience: A compression approach

Prevent error propagation or limit the scope if the damage.Optimal mode selectionMultiple description coding

38

Simple error-concealment shemes

The receiver replaces the whole frame with the previous reconstructed frame.The receiver replaces a corrupted block with the block at the same location from the previous frame.The receiver replaces the corrupted block with the block from the previous frame pointed by a motion vector.

39

40

MPEG-4

Transport MPEG-4 video over Internet is expected to be available !

42

Introduction to MPEG-4 (cont’d)

以物件導向為基礎的壓縮方式針對不同的視訊物件 , 予以個別獨立之壓縮

43

Introduction to MPEG-4 (cont’d)

每一個視訊物件還可以分層 , Base layer提供較低解析度 , Enhancement layer 提供較高解析度 .每一層由 Video object plane 組成 , 其中包含了物件的 shape 和 texture data.Spatial scalability: 透過分層調整物件的解析度Temporal scalability: 改變物件的播放速度

44

MPEG 家族比較表Compression MPEG-4 MPEG-2 MPEG-1

媒體網際網路類電話 DVD與 DTV VCD

解析度320X240160X120 720X480 352X240

訊框比可變式

固定 60 個畫面 / 秒(NTSC/ 交錯 )

30 個畫面 / 秒(NTSC/ 漸進 )

固定 30 個畫面 / 秒(NTSC/ 漸進 )

傳輸速度網際網路攝影機約 28~384kbp

DTV： 3~10Mbps VCD ：約 1.5Mbps

45

MPEG-7

MPEG-7標準著重在影音內容的描述和定義，以明確的資料結構和語法來定義影音資料的內容。它的正式名稱是「 Multimedia Content Description Interface 」。藉由MPEG-7格式所定義的資訊，使用者可以有效率的搜尋，過濾和定義所要的影音資料。

46

MPEG-21

MPEG-21是一個支援通過異質性網路和設備 , 使用戶透明而廣泛地使用多媒體資源的標準 , 其目標是建立一個易於交易的多媒體框架。主要功能 : 通過網路設備存取、使用並交互操作多媒體物件；實現多種版權和交易機制的自動管理；對內容使用者隱私的尊重。

On End-to-End On End-to-End Architecture for Architecture for Transporting MPEG-4 Over Transporting MPEG-4 Over the Internetthe Internet

Dapeng Wu, Yiwei Thomas Hou, Wenwu Zhu, Hung-Ju Lee, Tihao Chiang, Ya-Qin Zhang, and H.Jonathan ChaoIEEE TRASACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 10, NO. 6, SEPTEMBER 2000

48

An end-to-end architecture for transporting MPEG-4 video

49

50

MPEG-4 layersCompression layer compresses the visual information and generates elementary streams, which contain the coded representation of the VOs.SyncLayer packetized streams, and provide time and synchronization information. The SL-packetized streams are multiplexed into a FlexMux Stream at the TransMux layer.

51

RTP/RTCP protocol

RTP support the packet sequence number, timestamps, and some application-specific profiles.RTCP provides QoS feedback through use of receiver report (RR), sender report (SR), source description items (SDES) , BYE, and application specific functions (APP) .

52

53

End-to-end feedback control protocol

54

Adaptive encoding rate control (ARC) for MPEG-4 video

55

Initialization stage

rtβt: remaining available bit count for encoding the subsequent P -frames at the coding time instant t

T: duration of the video sequence (in second)

r: bit rate for the sequence (in bps)

I: number of bits actually used for the first I frame Channel output rate: β0 /N0 where N0 is the number of P -frames in the sequence

Buffer size: 0.5 × r

56

Pre-encoding stage (cont’d)

SASN

R tt

tt )1(1

Rt+1: the target bit count for the P -frame at time t+1

Nt: the remaining number of P -frames at time t

S: the weight factor in target bit estimation, default value is 0.05

At: the actual bits used for the P -frames at time t

57


ttt

ttt R

FF

FFR

)(2

)(2

tt RR

1

2

Ft: the current buffer fullness at time t

Θ: buffer size

μ= Ft /Θ

58


},max{ tf

t RR

ρ : application bit rate

ρf : frame rate of the source video

59


Avoid buffer fullness )1()( mFR tt

tt FmR )1(m: safety margin

Avoid buffer underflow

mFCR

mCFR

tt

tt )(C: β0 /N0 channel output rate

60

Dynamic target Bit-rate distribution among VOs

Target bit budget for the i th VO at time t

MAD: mean absolute difference, which is computed after the motion compensation for the luminance component

61

Encoding stage

Encoding the video frame and recording all actual bit-rate.Activating macroblock-layer rate control.

62

Post-encoding stage (cont’d)

R-D Model update

63

Post-encoding stage (cont’d)

64

Shape-threshold control

65

Frame-skipping control

66

Packetization algorithm

67

Simulation result <1>

68

69

Simulation result <2>

70

71

72

Future work

MPEG-4 video over ADSLMPEG-4 video over Cable-modemMPEG-4 video over GPRS mobile phoneMPEG-4 video over 3G wireless mobile devices

Thank You!

Technology

Transporting Real-Time Video Over the Internet: Challenges