1

Distributed Video System realized on mobile device with efficient Feedback channel分散式影像編碼在手機上的實現與有效率的回饋通道Chen, chun-yuan 陳群元Advisor: Prof. Wu, Ja-Ling 吳家麟 教授2012/6/28

2

Outline

Motivation and introduction DVC architecture overview DVC architecture with mobile device Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work

3

Motivation

Video communication is essential in Mobile device

Decrease the coding time and battery consuming on mobile.

4

Introduction

For Mobile device video codec Conventional video coding (ex:MPEG-4 H.264)

Heavy weight encoder, light weight decoder. Distributed Video Coding

Light weight encoder, heavy weight decoder.

5

Conventional video codec

Encoding Decoding

6

DVC to H.264 Transcoder

Clouding server:DVC to H.264 transcoder

DVC encoderH.264 decoder

Feedback channel

Parity bits

Encoded sequence

7

DVC codec baseline: DISPAC+DVC transcoder

1.Encoder2.Decoderno feedback channel !

Tse-Chung Su. Yun-Chung Shen. and Ja-Ling Wu. 2011. Real-time Decoding for LDPC Based Distributed Video Coding. National Taiwan University

8

Time complexity analysis without feedback channel

DVC Decoding without Feedback channel Side-Information generation LDPCA

time ratio

SI generationLDPCA Decetc.

31%

Total decoding time: 17.10 sec

9

Time complexity analysis with feedback channel

DVC Decoding Side-Information generation Send-Request at Feedback channel LDPCA

time ratio

SI generationLDPCA DecFeedback channeletc.

1%

95%Bottle neck!

Total decoding time: 330.51 sec

10

Outline

Motivation and introduction DVC architecture Overview DVC architecture with mobile device Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work

11

Distributed Video Codec

DCT

levelQuantizer

WZ Frames

𝑋𝑘

LDPCA Encoder

Buffer LDPCA Decoder

Key FramesConventional

IntraframeEncoder

Conventional Intraframe

Encoder

SI generation&refinement

DCT

SI ReconstructionBitplanes

Mode selection

CRCCRC

check

CRC-8

WZ bitstream

Feedback channel

Correlation Noise

Modeling

IDCT&Deblocking filter

Decoded WZ Frames

Decoded Key Frames

WZFrame

WZFrame

WZFrame

GOP size 4

KeyFrame

KeyFrame

12

Outline

Motivation and introduction DVC architecture overview DVC architecture with mobile device Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work

Distributed Video Codec with mobile

13

DCT

levelQuantizer

WZ Frames

𝑋𝑘

LDPCA Encoder

Buffer LDPCA Decoder

Key FramesConventional

IntraframeEncoder

Conventional Intraframe

Encoder

SI generation&refinement

DCT

SI Reconstruction

Bitplanes

Mode selection

CRCCRC

check

CRC-8

WZ bitstream

Feedback channel

Correlation Noise Modeling

IDCT&Deblocking filter

Decoded WZ Frames

Decoded Key Frames

Transcode Decoded Video sequenceWith specific form

Video deliver

14

DVC Encoder Realization

Implement DVC encoder on Mobile Key frame encoding WZ frame encoding Efficient intra mode realization

Feedback channel transmission Network connection between mobile and server Determine payload of a packet with LDPCA

protocal

15

Key frame encoding on Mobile

Previous ver. use JM9.5 Transcoder ref x.264 on PC

x.264 lib

Cross compilerKeyframe encoding

16

DVC Encoder Realization

Implement DVC encoder on Mobile Key frame encoding WZ frame encoding Efficient intra mode realization

Feedback channel transmission Network connection between mobile and server Determine payload of a packet with LDPCA

protocal

17

WZ coding with NDK

Native code

DVC encoder

18

DVC Encoder Realization

Implement DVC encoder on Mobile Key frame encoding WZ frame encoding Efficient intra mode realization

Feedback channel transmission Network connection between mobile and server Determine payload of a packet with LDPCA

protocal

19

Efficient intra block coding

Previous ver. Transcoder Use JM code. Intra block encoding consumes lots of time

in encoding.

Clouding server:DVC to H.264

transcoder

20

DVC Encoder Realization

Implement DVC encoder on Mobile Key frame encoding WZ frame encoding Efficient intra mode realization

Feedback channel transmission Network connection between mobile and server Determine payload of a packet with LDPCA

protocal

21

Network connection

Create network connection to connect remote server. TCP socket in C

Clouding server:DVC to H.264

transcoder

22

DVC Encoder Realization

Implement DVC encoder on Mobile Key frame encoding WZ frame encoding Efficient intra mode realization

Feedback channel transmission Network connection between mobile and server Determine payload of a packet with LDPCA

protocal

23

ACK at Feedback channel

TCP socket

…

syndromes

Send syndrome

Request

syndromessyndromes

syndromes

24

Consuming time atFeedback channel

Packet header occupy large part of network bandwidth.

Network latency in communication at Feedback channel

25

Outline

Motivation and introduction DVC architecture with mobile device DVC architecture overview Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work

26

Efficient feedback channel

we propose two methods to decrease time consuming at feedback channel. Estimate the syndromes size per WZ frame Estimate the syndromes size per bitplane

Average number of requests per bitplane

1234567123456123456123451234512345123412341234123412312312312120

200

400

600

800

1000 bit rate

bit rate

DC AC1 AC2 AC3 AC4 AC5 AC6 AC7 AC8 AC9AC10AC11

AC12AC13

AC14

Bitplane Number

28

Estimate the syndromes size per WZ frame

Estimate syndromes size by correspond WZ frame in previous GOP

Ex. GOP 4 Key

frameKey

frameWZ

frameWZ

frameWZ

frame

First GOP

Key frame

Key frame

WZ frame

WZ frame

WZ frame

Second GOP

formula

(ESbt)WZn=(Sbt)WZ(n-GOPsize)

WZn means the WZ frame’s index n-GOPsize means the correspond WZ frame in

the previous GOP. bt is the bitplane index. (ESbt)WZn means the estimated syndromes bit for

the bt’th bitplane of WZn frame. (Sbt)WZ(n-GOPsize) means the syndromes bit for the

bt’th bitplane of WZ(n-GOPsize) frame

30

Efficient feedback channel

we propose two methods to decrease time consuming at feedback channel. Estimate the syndromes size per WZ frame Estimate the syndromes size per bitplane

Number of requests per bitplane per band

1234567123456123456123451234512345123412341234123412312312312120

200

400

600

800

1000 bit rate

bit rate

DC AC1 AC2 AC3 AC4 AC5 AC6 AC7 AC8 AC9AC10AC11

AC12AC13

AC14

Bitplane Number

Number of requests per bitplane per band

1234567123456123456123451234512345123412341234123412312312312120

200

400

600

800

1000 bit rate

bit rate

DC AC1 AC2 AC3 AC4 AC5 AC6 AC7 AC8 AC9AC10AC11

AC12AC13

AC14

Bitplane Number

Estimate by previous bands

1.formula 2.time ratio

Number of requests per bitplane per band

1234567123456123456123451234512345123412341234123412312312312120

200

400

600

800

1000

bit rateestimated bit rate

DC AC1 AC2 AC3 AC4 AC5 AC6 AC7 AC8 AC9AC10AC11

AC12AC13

AC14

Bitplane Number

formula

(ESacn bt)WZn= ( Sdc bt-1 )WZn

WZn means the n’th WZ frame. acn means the n’th AC band. bt means the bitplane index of this AC band. Which ESacn

bt mean the estimated syndrome size for the bt’th bitplane in n’th AC band.

dc means the DC band. Sdc bt-1 means the syndromes bits for bt-1’th

bitplane in DC band.

36

Outline

Motivation and introduction DVC architecture overview DVC architecture with mobile device Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work

Transcode to mobile codec

Because DVC only encode luminance part in YUV, so we got only Y for output.

Our mobile device can only read mp4 and 3gp.

Y

U V

R G BFrame 1Frame 2Frame 3Frame4Frame 5Frame 6Frame 7Frame n.mp4

38

Outline

Motivation and introduction DVC architecture overview DVC architecture with mobile device Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work

39

Experiment Result

規格 實驗數據 :

時間 (&decode) 、 packet 數量 ( 傳幾次 )+

Test condition

12 CPU, 24 processor Intel(R) Xeon(R) CPU X5650 @ 2.67GHz

GPU: Tesla M2050 Mobile device: HTC sensation

Test sequences :

QCIF, 15Hz, all frames GOP Size 2, 4 and 8 Only luminance component is used

Test materials

Soccer Foreman Coastguard Hall Monitor

Motion: High Low

Speed up

Bitrate

RD curve

Foreman with LRSS,GOP8

RD curve

Hall monitor with LRSS,GOP8

46

Outline

Motivation and introduction DVC architecture overview DVC architecture with mobile device Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work

47

Conclusion

We combine DISPAC and transcoder, then port it’s encoder on mobile.

Implement and propose Feedback channel. Record video with mobile device’s camera. Transcode the decoded sequence to mobile

codec.

48

Outline

Motivation and introduction DVC architecture overview DVC architecture with mobile device Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work

49

Future Work

Take a more precise estimation of syndrome size.

Use motion information to estimate the syndrome size.

Build a reliable DVC codec which is loss tolerant.

50

Thank You

52

To do

Real time decoder Efficient intra coding Print MSG on Mobile

53

DVC decoding time

Side Info Creation: 0.85 sec Overcomplete Transform: 2.35 sec Correlation Noise Modeling: 2.33 sec Conditional Bit Prob Compute: 0.60 sec Update Side Info: 7.14 sec Ldpca Decode: 123.34 sec Motion Learning: 1.79 sec Others: 1.80 sec

57

Distributed Video Codec

DCT

levelQuantizer

WZ Frames

𝑋𝑘

LDPCA Encoder

Buffer LDPCA Decoder

Key FramesConventional

IntraframeEncoder

Conventional Intraframe

Encoder

SI generation&refinement

DCT

SI ReconstructionBitplanes

Mode selection

CRCCRC

check

CRC-8

WZ bitstream

Feedback channel

Correlation Noise

Modeling

IDCT&Deblocking filter

Decoded WZ Frames

Decoded Key Frames

Distributed Video Codec with mobile

58

DCT

levelQuantizer

WZ Frames

𝑋𝑘

LDPCA Encoder

Buffer LDPCA Decoder

Key FramesConventional

IntraframeEncoder

Conventional Intraframe

Encoder

SI generation&refinement

DCT

SI Reconstruction

Bitplanes

Mode selection

CRCCRC

check

CRC-8

WZ bitstream

Feedback channel

Correlation Noise Modeling

IDCT&Deblocking filter

Decoded WZ Frames

Decoded Key Frames

Transcode Decoded Video sequenceWith specific form

Video deliver

59

Video recodeWith mobile camera

60

ACK at Feedback channel

…

Send syndrome

LDPCA Encoder

&CRC

LDPCA Decoder

&CRC

Syndromes for one bitplane

61

ACK at Feedback channel

…

Request

LDPCA Encoder

&CRC

LDPCA Decoder

&CRC

Syndromes for one bitplane Decoding one bitplane

with received syndromesAnd chack CRC

62

ACK at Feedback channel

…

Send syndrome

LDPCA Encoder

&CRC

LDPCA Decoder

&CRC

Syndromes for one bitplane

67

Contribution

Realize DVC encoding part on Mobile device Implement Feedback channel Implement encoder’s component with x264 Propose an efficient Feedback channel Transcode the decoded sequence to mobile

codec

68

Outline

Motivation and introduction DVC architecture with mobile device Propose an efficient feedback channel Transcoding Experiment Result Conclusion Future work