口試投影片(詹智傑) Final

自適性串流架構下之視訊影像品質最佳化設計與實作

指導教授：楊士萱學生：詹智傑

Design and Implementation of Quality Optimization Strategy in Adaptive Streaming

2

Outline

• Introduction to MPEG-DASH • Related Works• Proposed State-based Strategy• Results and Future Works

3

Outline

• Introduction to MPEG-DASH • Related Works• Proposed State-based Strategy• Results and Future Works

4

Introduction to MPEG-DASH

• Dynamic Adaptive Streaming over HTTP– 由 3GPP 與 MPEG 一起訂定的自適性串流標準– 屬於 HTTP/TCP 的串流協定– 唯一的自適性串流標準

5


• Dynamic Adaptive Streaming over HTTP– 何謂自適性串流 (Adaptive bit-rate streaming)

• 依據環境與應用需求即時性的調整視訊串流之碼率– 視訊之解析度 (Resolution) 與品質 (Quality) 需求– 網路狀態與撥放延遲 (playback delay)– 客戶端之暫存器使用效率 (Video buffer)– 使用終端之計算能力與複雜度

• 目的是獲得更佳之使用經驗品質 (Quality of Experience, QoE) 與頻寬使用效率 (Bandwidth efficiency)

6


• Dynamic Adaptive Streaming over HTTP– 使用 HTTP/TCP (not RTSP/UDP) 的優勢

• 多數使用終端支援 HTTP 協定• TCP 能提供高可靠性 (reliable) 且有秩序的傳送

(ordered transmission) 並且能避免 NAT 與防火牆之阻擋問題

• HTTP-based streaming 能夠重複使用已建置之 HTTP servers 與 caches 做資料內容 (content delivery) 的傳遞

• HTTP-based streaming 將串流連線之控制權由 Server端移轉至 Client 端

7


• Dynamic Adaptive Streaming over HTTP– 世界上第一個也是唯一的 International standard

• Apple HTTP Live Streaming (Apple-HLS)• Microsoft Live Smooth Streaming (Microsoft-LSS)• Adobe Dynamic Streaming for Flash (ADOBE-DSF)• OPEN IPTV Forum: HTTP Adaptive Streaming• Dynamic Adaptive Streaming over HTTP (MPEG-DASH)• …

Draft International Standard : 2011 Jan

International Standard : 2011 Nov

ISO/IEC International Standard : 2012 April

ISO/IEC Technical Corrigendum 1 : 2013 May

8


• 系統架構 (System Architecture)

HTTP server DASH clientSegment

Segment

Segment

Segment

Segment

Segment

Segment

Segment

Segment

Segment

Segment

Segment

Segment

Segment

Segment

Segment

MediaPresentationDescription

(MPD)

Control heuristics

HTTP client

MPD parser

Segment parser

Mediaplayer

HTTP 1.1

視訊串流服務之”目錄”資料模

型

視訊串流服務之”媒體”資料模

型

9


• 資料模型 (Data Model)– Media Presentation Description (MPD)

• Descript the video segment’s information in xmlex: timing, codec, resolutions, bit-rate … etc.

– Segments• The media data.• ex: mp4, m4s, mp3 … etc.

10


• 資料模型 (Data Model)– Media Presentation Description (MPD)Media presentation

DescriptionPeriod ID = 1, start=0 s

Period ID = 2, start=100 s

Period ID = 3, start=200 s

Period ID = 2start=100s

AdaptationSet 0

AdaptationSet 1

AdaptationSet 2

Adaptation set 1

Representation 1 5Mbps

Representation 2 2Mbps

Representation 3 500kbps

Representation 22 Mbytes

Segments

11


• 系統流程圖 (System Flowchart)

Process MPD

HTTP Get MPD

Measure Download Rate

Select Representation

Measure Download Rate

HTTP Get Segment #1

HTTP Get Segment #2

#1

#2

12


• 客戶端系統流程圖 (Client side system flowchart)

Main part of MPEG-DASH

Segment buffer

13

Major contribution of this thesis

• 實際建置一個包含伺服器與客戶端之視訊串流系統，並加入 MPEG-DASH 框架

• 我們採用 emulation 的形式，相較於simulation 能提供更真實、更有修改與加入環境參數彈性的系統

• 我們探索視覺品質 (visual quality) 、網路頻寬(network bandwidth) 以及暫存器使用量(buffer usage) 的關聯性，並提出一套自適性品質控制以提升感知上的品質 (perceived quality)

14

Outline

• Introduction of MPEG-DASH • Related Works

– VLC Player MPEG-DASH rate-based Quality Strategy

– Related Researches in MPEG-DASH• Proposed State-based Strategy• Results and Future Works

15

VLC Player Rate-based Quality Strategy

• 實作於 VLC Player DASH plug-in 中

• Main consider principle– 當暫存器容量低於預設

threshold ，接收最低碼率串流以加速暫存器容量補充

< minimum buffer threshold

> minimum buffer threshold

NoneAt least one

Algorithm start

InitialMPD ManagerCurrent Period

Get current Bandwidth

Check current Buffer capacity

Set current bandwidth as zero

Gather Representations withsame resolution of the player

Check the quantity of Representations

in candidate set

Find target Representationfrom every AdaptationSets

Find target Representationfrom candidate set

Algorithm end

Stage one

Stage two

16


• Advantage– 視訊片段緩衝器補充速度快

• Disadvantage– 串流初始狀態的品質等級低– 易受到網路瓶頸而品質崩潰

17


• Advantage• Disadvantage

– 串流初始狀態的品質等級低– 易受到網路瓶頸而品質崩潰

Bottleneck

18

Outline

• Introduction of MPEG-DASH • Related Works

– VLC Player MPEG-DASH rate-based Quality Strategy

– Related Researches in MPEG-DASH• Proposed State-based Strategy• Results and Future Works

19

Related Researches in MPEG-DASH

• [23] 以馬可夫鍊的方式對無線網路的頻寬變化特性做分析與歸類，並提出找尋最佳視訊品質的演算法。

• [25] 探討如何使用可調式編碼 (Scalable video coding) 提升網路儲存空間的使用效率。

• [26] 以代理伺服器 (Proxy) 的角度出發，以網路管理的 QoE 機制來替客戶選擇最佳的視訊品質。

20

Related Researches in MPEG-DASH

• 雖然過去兩年 MPEG-DASH 相關的投稿文獻絡繹不絕，但多數的共通點都如 [23][25] 採用純模擬 (Simulation) 或數學推導，無法於實際網路中驗證。

• 故本論文除了提出一套 MPEG-DASH 的品質調控機制，也將實作一套系統進行驗證。

21

Outline

• Introduction of MPEG-DASH • Related Works• Proposed State-based Strategy• Results and Future Works

22

Proposed State-based Quality Strategy

• 本論文提出之 DASH 品質調控機制分 two stage– First Stage ：以下載 MPD 的網路資訊預測前幾

個 Segment 的最佳品質– Second Stage ：依照網路與視訊片段暫存器使

用率來預測下一個 Segment 的最佳品質

23


• Flowchart of first stage– Case 1:

• MPD 檔中的第一個Segment 。

– Case 2:• 新 Adaptation set 的第一

個 Segment 的決策流程。– Case 3:

• 一般之 Segment 的決策流程

Choise segmentStart

!First segment

!First segment in MPD

No

1 2 3

NoYes

Yes

InitialGet target RepresentationList

Sort RepresentationsRepresentationStream

24


1

Download InitializationSource URL

Download first five segmentby using MPD’s bitrates as ref.

Compute and save five segment’s downloaded bitrates

End

2

Download InitializationSource URL

Choice target level according to pre-algorithm’s last five segments

Compute and save segment’s downloaded bitrates

End

Download target level segments

3

Choice target level according tofive pre-segments

Download target level segments

Compute and save segment’s downloaded bitrates

End

Case 1: Case 2: Case 3:

25


• Finite state machine of second stage

State 1-1

State 2-3

State 2-2

State 2-1

Network Throughputs > Highest Representation

Network Throughputs < Highest Representation

Network Throughputs < 80% Representations

Network Throughputs < 80% Representations

NetworkThroughputs

(Bps)

SegmentBuffer

(%)0

Adaptation Set’sRepresentation

(Level)

Level n

Level 1

Level 2

State 3-2 State 3-1

Level k

Network Throughputs < Representations Level 2

Network Throughputs > Representations Level 2

Buffer Fill persentage > 30%


30% 50%Buffer Fill persentage > 50%


Case One

Case Two

Case Three

26

Outline


– Emulation Settings– Objective Evaluation– Subjective Evaluation

27

Emulation Settings

• 實驗前提– 本論文探討“頻寬高於 MPD 所能提供的最低碼

率視訊片段”的情境– 本論文實作的是 On demand 的視訊串流服務– 本論文探討“ Client有足夠計算能力解析 MPD

中任一等級的串流”– 本論文假設碼率與 PSNR 的關係為已知，故不須於解碼的同時檢測畫面品質。

28

Emulation Settings

• 測試平台 Client ServerName Asus PC Asus PCCPU Intel Core 2 Quad

2.67GHzIntel Core 2 Quad 2.67GHz

Memory 4G DDRII 2G DDRIIOS Windows 7 Ubuntu 12.04Network Ethernet Ethernet

29

Emulation Settings

• 測試平台

HTTP Server

Apache Server(on demand)

NetworkEmulator

Client

PC

Notebook

HTTP

Internet

DASH-AVC/264Encoder

30

Emulation Settings

• 所使用之開源式軟體Open Source Software

Operating System Linux Ubuntu 12.04 LTE

Web HTTP Server Apache 2

Server Database MySQL

Language Perl, Bash script

Network Emulator Linux tc commands

Network Monitor libpcap & winpcap

Server Monitor Monitorix

Client side libraries libdash, libcurl, libxml2, libav

31

Emulation Settings

• 測試序列Parameter Value

Length (min:sec) 09:46

Segment size (second/segment) 2

Numbers of Frames 14315

Video codec H.264/AVC

Container MP4 & M4S

Frame rate (fps) 25 fps

GOP size (frame) 25

Resolution (pixel) 480 x 360

32

Emulation Settings

• 測試序列Level Video bitrate Level Video bitrate

0 100 kbits/s 7 1300 kbits/s







33

Emulation Settings

• 序列特性：品質等級與碼率關係

The lower the bit-rate is, the smaller the gap is.

34

Emulation Settings

• 網路模型– 高品質穩態網路模型

• 網路頻寬維持於 MPD 最高品質等級之上

– 遞增型網路模型• 網路頻寬由低頻寬以穩定之速率上升

– 遞減型網路模型• 網路頻寬由高頻寬以穩定之速率下降

35

Emulation Settings

• 網路模型

0 100 200 300 4000

1000000

2000000

3000000

4000000

5000000

6000000

7000000

最高品質穩態網路模型遞增網路模型遞減網路模型

Time(s)

Bitr

ate(

kbits

/sec

)

36

Outline



Objective Evaluation

• 頻寬使用效率分析– 最高品質穩態網路模型

0 10 20 30 40 50 600

1000000

2000000

3000000

4000000

5000000

6000000

7000000

Bandwidth Throughput(VLC[31]) Throughput(Proposed)ChosenBitrate(VLC[31]) ChosenBitrate(Proposed)

Time(sec)

Bitr

ate(

kbit

/sec

)

37


• 頻寬使用效率分析– 最高品質穩態網路模型

0 10 20 30 40 50 600

1000000

2000000

3000000

4000000

5000000

6000000

7000000

Bandwidth Throughput(VLC[31]) Throughput(Proposed)ChosenBitrate(VLC[31]) ChosenBitrate(Proposed)

Time(sec)

Bitr

ate(

kbit

/sec

)

38

0 10 20 30 40 50 600

1000000

2000000

3000000

4000000

5000000

6000000

7000000

012345678910111213

Bandwidth Throughput(VLC[31]) Throughput(Proposed)ChosenLevel(VLC[31]) ChosenLevel(Proposed)

Time(sec)

Bitr

ate(

kbit

/sec

)

Leve

l(#)


• 頻寬使用效率分析– 遞增網路模型

39


• 頻寬使用效率分析– 遞減網路模型

40


• 頻寬使用效率分析– 遞減網路模型

41


• 使用者觀賞品質分析– 最高品質穩態網路模型

42



43

+4.02845 dB



• Average PSNR

44

PSNRY PSNRU PSNRV

Proposed 41.9227dB 47.8613 dB 49.9766 dBVLC[31] 41.4475 dB 47.4507 dB 49.5599 dB△PSNR +0.4752 dB +0.4106 dB +0.4167 dB


• 使用者觀賞品質分析– 遞增網路模型

45


• 使用者觀賞品質分析– 遞減網路模型

46



47



• Average PSNR

48

PSNRY PSNRU PSNRV

Proposed 39.311 dB 47.814 dB 48.930 dBVLC[31] 38.728 dB 47.325 dB 48.449 dB△PSNR +0.583 dB +0.489 dB +0.481dB



• Average PSNR (from 8000~12000 frame)

49

PSNRY PSNRU PSNRV

Proposed 38.411 dB 44.065 dB 44.54 dBVLC[31] 36.93 dB 42.875 dB 43.325 dB△PSNR +1.481 dB +1.19 dB +1.215 dB

畫面數 : 8000~12000張

50

Outline



51

Subjective Evaluation

• 串流起始狀態– 最高品質穩態網路模型 (frame 829)

Proposed (36.160dB) VLC[31] (27.810dB)

52


• 串流起始狀態– 遞增網路模型 (frame 279)


53


• 串流起始狀態– 遞減網路模型 (frame 373)


54


• 串流起始狀態 (Demo)– 最高品質穩態網路模型

Proposed VLC[31]

55


• 串流起始狀態 (Demo)– 遞減網路模型

Proposed VLC[31]

56


• 遞減網路模型低視訊品質狀態– frame 9872


57


• 遞減網路模型低視訊品質狀態– Demo

Proposed VLC[31]

58

Results and Future Works

• 本論文實作一套包含伺服器與客戶端的自適性串流系統。

• 本論文在客戶端提出一套視訊品質評估機制。

• 整體 PSNR 相較於 [31] 提升約 0.5dB；在初始狀態上，也能達到約 1dB 到 4dB 的提升；而在網路瓶頸時也有 1dB 的提升。

59

Results and Future Works

• 提升品質調控機制的準確度• 擴充其他視訊編碼標準與其相關特性探討• 切割自適性串流機制與實際解碼的框架，

以做為智慧型家庭、微型視訊共享機制的系統雛形。

60

Reference• [1] Cisco, Cisco Visual Networking Index: Global Mobile Data Traffic

Forecast Update, 2012–2017, Feb 6, 2013.• [2] R. Pantos, and W. May, HTTP Live Streaming, version 10, IETF Internet-

Draft, Oct 15, 2012.• [3] Information technology — Dynamic Adaptive Streaming over HTTP

(DASH), ISO/IEC 23009-1:2012 ,Aprl 01, 2012.• [4] Jack Y. B. Lee , Scalable Continuous Media Streaming Systems-

Architecture, Desgn, Analysis and Implementation, John Wiley & Sons Ltd, June, 2005.

• [5] H.264: Advanced Video Coding for Generic Audiovisual Services, ITU-T H.264, Jan, 2012.

• [6] I. E. G. Richardson, H.264 and MPEG-4 Video Compression: Video Coding for Next Generation Multimedia, John Wiley & Sons, 2003

• [7] 經濟部 , 2003 數位內容產業年鑑趨勢篇第二章數位內容技術趨勢

61

Reference• [8] Microsoft Corporation, IIS Smooth Streaming Technical Overview, Mar

25, 2009.• [9] Apple Corporation, HTTP Live Streaming Overview, April 1, 2011.• [10] Information technology — Dynamic Adaptive Streaming over HTTP

(DASH) — Part 1: Media Presentation Description and Segment Formats, AMENDMENT 1: Support for Event Messages and extended Audio Channel Configuration, ISO/IEC 23009-1:2012/DAM 1, Dec 07, 2012.

• [11] Information technology — Dynamic Adaptive Streaming over HTTP (DASH) — Part 1: Media Presentation Description and Segment Formats, TECHNICAL CORRIGENDUM 1, ISO/IEC 23009-1:2013/Cor.1:2012(E), Feb 02, 2013.

• [12] 蔡德明 , 鳥哥的 Linux私房菜基礎學習篇第二版 , 上奇科技出版事務處 , March, 2006.

• [13] 蔡德明 , 鳥哥的 Linux私房菜伺服器架設篇第二版 , 上奇科技出版事務處 , March, 2007.

62

Reference• [14] S. Blake, D. Black, M. Carlson, E. Davies, Z. Wang, W. Weiss, An Archi-

tecture for Differentiated Services, IETF RFC 2475, Dec 1998, http://tools.ietf.org/html/rfc2475

• [15] M. Allman, V. Paxson, W. Stevens, TCP Congestion Control, IETF RFC 2581, Aprl 1999, http://tools.ietf.org/html/rfc2581

• [16] D. Robinson, K. Coar, The Common Gateway Interface (CGI) Version 1.1, IETF RFC 3875, Oct 2004, http://tools.ietf.org/html/rfc3875

• [17] Information technology — Dynamic adaptive streaming over HTTP (DASH) — Part 2: Conformance and reference software, ISO/IEC 23009-2:2012 ,May 28, 2012.

• [18] N. Cranley, P. Perry, and L. Murphy, “User perception of adapting video quality,” ACM International Journal of Human-Computer Studies, pp. 637-647,Aug. 2006

http://tools.ietf.org/html/rfc2475



63

Reference• [19] T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra, "Overview of

the H.264/AVC video coding standard,” IEEE Transactions on Circuits and Systems for Video Technology, vol.13, no.7, pp.560,576, July 2003

• [20] M. Wien, H. Schwarz, T. Oelbaum, "Performance Analysis of SVC,” IEEE Transactions on Circuits and Systems for Video Technology, vol.17, no.9, pp.1194,1203, Sept. 2007

• [21] A. Begen, T. Akgul, M. Baugher, " Watching Video over the Web: Part 1: Streaming Protocols,” IEEE Internet Computing, vol.15, no.3, pp.59-63, May-June 2011

• [22] I. Sodagar, "The MPEG-DASH Standard for Multimedia Streaming Over the Internet,” IEEE Multimedia, vol.18, no.4, pp.62-67, April 2011

• [23] S. Xiang, L. Cai, and J. Pan, “Adaptive scalable video streaming in wireless networks,” MMSys'12 Proceedings of the 3rd Multimedia Systems Conference, pp. 167-172, 2012

64

Reference• [24] V. Swaminathan,and Wei Sheng, "Low latency live video streaming

using HTTP chunked encoding,” 2011 IEEE 13th International Workshop on Multimedia Signal Processing (MMSP), pp.1,6, 17-19, Oct. 2011

• [25] Y. Sánchez, T. Schierl, C. Hellge, and T. Wiegand, “iDASH: improved dynamic adaptive streaming over HTTP using scalable video coding,” MMSys'11 Proceedings of the second annual ACM conference on Multimedia systems, pp.257-264, Feb 2011

• [26] R.K.P. Mok, X. Luo, E. W. W. Chan, and R. K. C. Chang, "QDASH: A QoE-aware DASH system,” MMSys'12 Proceedings of the 3rd Multimedia Systems Conference, pp.11-22, Feb. 2012

• [27] O. Nemcic, M. Vranjes, and S. Rimac-Drlje, “Comparison of H.264/AVC and MPEG-4 part 2 coded video,” Proc. Electronics in Marine (ELMAR), pp.41,44, 12-14, Sept. 2007

65

Reference• [28] T. Lohmar, T. Einarsson, P. Fröjdh,F. Gabin, M. Kampmann, “Dynamic

Adaptive HTTP Streaming of Live Content,” IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks, June 20-24, 2011

• [29] T. Stockhammer, “Dynamic Adaptive Streaming over HTTP–Standards and Design Principles,” MMSys'11 Proceedings of the second annual ACM conference on Multimedia systems, Feb 2011,

• [30] S. Lederer, C. Müller, and C. Timmerer, “Dynamic Adaptive Streaming over HTTP Dataset,” In Proceedings of the ACM Multimedia Systems Conference 2012, Chapel Hill, North Carolina, February 22-24, 2012.

• [31] C. Müller and C. Timmerer. “A VLC media player plugin enabling dynamic adaptive streaming over HTTP,” In Proceedings of the 19th ACM international conference on Multimedia, New York, NY, USA, 723-726, 2011.

66

Reference• [32] L. R., Romero, “A Dynamic Adaptive HTTP Streaming Video Service for

Google Android,” Oct 06, 2011, Available from: http://dat.etsit.upm.es/~luciano /en/master-thesis.pdf

• [33] Jan Ozer, “What is streaming media protocol?,” http://www.streaming media.com/Articles/Editorial/What-Is-.../What-Is-a-Streaming-Media-Protocol-84496.aspx

• [34] Liz Gannes, “The Next Big Thing in Video: Adaptive Bitrate Streaming,” http://pro. gigaom.com/blog/how-to-deliver-as-much-video-as-users-can-take/

• [35] wiki Adaptive streaming: http://en.wikipedia.org/wiki/Adaptive_bitrate_stre aming

• [36] Adaptive Streaming Comparison: http://learn.iis.net/page.aspx/792/adaptive -streaming-comparison/

• [37] Video LAN Official page: http://www.videolan.org/vlc/index.html

http://dat.etsit.upm.es/~luciano%20/en/master-thesis.pdf

http://en.wikipedia.org/wiki/Adaptive_bitrate_stre%20aming

http://en.wikipedia.org/wiki/Adaptive_bitrate_stre%20aming

http://learn.iis.net/page.aspx/792/adaptive%20-streaming-comparison/

http://www.videolan.org/vlc/index.html

67

Reference• [38] September 2012 Web Server Survey :

http://news.netcraft.com/archives/2012/09/10/september-2012-web-server-survey.html

• [39] Ubuntu documentation: Guide line for installing LAMP in Linux Ubuntu : ht tps://help.ubuntu.com/community/ApacheMySQLPHP

• [40] Monitorix Project by Jordi Sanfeliu : http://www.monitorix.org/• [41] RRDtool – About RRDtool :

http://oss.oetiker.ch/rrdtool/index.en.html• [42] libdash | bitmovin.net multimedia :

http://www.bitmovin.net/libdash/• [43] libav－ Open source audio and video processing tools :

http://libav.org/• [44] Hierarchical token bucket theory :

http://luxik.cdi.cz/~devik/qos/htb/man ual/theory.html• [45] Traffic control HOWTO :

http://linux-ip.net/articles/Traffic-Control- HOWTO/



https://help.ubuntu.com/community/ApacheMySQLPHP

https://help.ubuntu.com/community/ApacheMySQLPHP

http://www.monitorix.org/

http://oss.oetiker.ch/rrdtool/index.en.html

http://www.bitmovin.net/libdash/

http://libav.org/

http://luxik.cdi.cz/~devik/qos/htb/man%20ual/theory.html



http://linux-ip.net/articles/Traffic-Control-%20HOWTO/