All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.

5G寬頻應用Future Video Codec

視訊規格標準化進程

工業技術研究院資訊與通訊研究所視訊多媒體通訊技術組林俊隆

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 2

Biography

• Ph.D. degree in Computer Science from

National Tsinghua University, 2010

• Information and Communications

Research Laboratories, ITRI − 2010/11~

• Leader of MPEG Standard team– Over 100+ MPEG standard contributions

– Over 80+ pending or granted patents

• Leader of tech. team on emerging

VR/AR/MR technology

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 3

Outline

• MPEG Roadmap

• JVET activities– Overview of Call for Proposal(CfP)

• Versatile Video Coding (VVC)/H.266

– Results of CfP responses

– WD and TM status

• MPEG activities– Point Cloud Compression(PCC)

– Coded Representation of

Neural Networks (NNR)

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.

� 行動多媒體影音傳輸與串流是行動通訊市場與寬頻網路市場的Killer

Application

� Mobile video (ex: video streaming,

video conferencing) 預計將占行動網路總流量的75%以上

� 4k8k UHD、3D Video、HDR/WCG

video and VR/AR等將大幅增加未來通訊頻寬的需求

� 高效能視訊編碼技術� 大幅降低多媒體影音資訊的資料傳輸量� 提升下世代行動通訊視訊應用的滲透率

� MPEG/ITU-T標準組織� 針對各種多媒體視訊應用需求制定編碼

及傳輸標準� MPEG-2, MPEG-4, MPEG-H

� H.261, H.263, H.264, H.265

Importance of Video Codec

Source :ERICSSON MOBILITY REPORT JUNE 2017

Source : Cisco Visual Networking Index: Global Mobile Data

Traffic Forecast Update, 2016–2021

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.

� Video codec and 3GPP

� ETSI 3GPP (H.263)

� 3GPP release 6 (H.264/AVC)

� 3GPP release 12 (H.265/HEVC)

� 5G (??? H.266/H.265/AV2/EQ-AVC)

� Versatile Video Coding (VVC)/H.266

� 為配合5G通訊標準制定，MPEG/ITU-T 2018年開始下一代 Video codec標準制定(暫名VVC/H.266)，並預計在2020年完成H.266 v1的標準制定

3GPP SA4 and Video Codec

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.

MPEG/ITU-T Standard Activities

6

MPEG

Moving Picture Experts Group

VCEG

Video Coding Experts Group

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 7

MPEG Roadmap

7

2018 20202017 2019 2021 2022 Jan 2023

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.

2018年開始制定年開始制定年開始制定年開始制定 H.266 第一第一第一第一版標準版標準版標準版標準，，，，預計預計預計預計2020完成完成完成完成

MPEG視訊標準的演進

AVC/H.264 HEVC/H.265VVC/H.266

Immersive Media

HDR

VR360

Light Field

(Sparse)

Point Cloud

Full-HD Mobile

UHD Broadcasting

Blu-ray

HDTV

Internet Video

第一版標準第一版標準第一版標準第一版標準於於於於2013完成完成完成完成第一版標準於第一版標準於第一版標準於第一版標準於2003完成完成完成完成

2003 2013 2020

FVC(Future Video Coding)

HDR(High Dynamic Range)

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 9

JVET Call for Proposals (CfP)

• San Diego, USA

• Date: 10 ~ 20 April, 2018

• Approximately 350+ participants

• 23 CfP proposals

• Approx. 80 input documents– Including 23 CfP proposals

• New project launched– Versatile Video Coding(VVC)

– Versatile Test Model(VTM)

9

*WD : Working Draft

*TM : Test Model

*CD : Committee Draft

*FDIS: Final Draft International Standard

Timeline

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 10

JVET Call for Proposal• Test categories:

– Standard Dynamic Range(SDR)

– High Dynamic Range(HDR)

– 360° Video

• 46 category-specific submissions to be tested

(not counting the anchors)– SDR:22 submissions (8 of which are registered only in this

category)

– HDR: 12 submissions (4 of which are registered only in this

category)

– 360:12 submissions (4 of which are registered only in this

category)

10

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 11

CfP Performance

• Measured by objective performance,

– >40% bit rate reduction compared to HEVC

– >10% compared to JEM (for SDR case)

– More elements show better performance

– Some proposals show similar performance as JEM with

significant run time reduction

– Similar ranges for HDR and 360°

• Results of subjective tests generally show similar (or

even better) tendency

– Benefit over HEVC very clear

– Benefit over JEM visible at various points

11

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 12

Performance of SDR

12

Y U V Enc Time Dec Time Y U V Enc Time Dec Time

Peking Univ. CN N

DJI CN N

Ericsson SE Y

Nokia FI Y

ETRI KR Y

Sejong Univ. KR Y

-7.55% -6.94% -5.96% 126% 102% -38.06% -46.88% -46.53% 1046% 780%

6.88% 6.10% 6.65% 126% 101% -28.64% -40.98% -41.19% 1047% 775%

InterDigital US N

Dolby US N

J0016 KDDI JP N JEM 7.0 -0.57% -0.52% -1.30% 108% 1886% -33.50% -43.57% -44.18% 858% 18614% Y

J0017 LG KR Y JEM -2.52% -5.29% -6.19% 191% 84% -34.75% -45.89% -46.73% 1523% 644%

-16.06% -6.75% -10.43% 152% 227% -43.81% -45.61% -47.41% 1190% 1302%

-14.40% -5.13% -8.82% 77% 232% -42.38% -44.64% -46.37% 606% 1330%

-2.28% -3.44% -3.88% 107% 56% -34.63% -45.06% -45.52% 817% 384%

-0.06% 0.91% 0.52% 60% 55% -33.18% -42.83% -43.15% 456% 372%

Qualcomm US Y -15.53% -3.66% -5.97% 148% 84% -43.08% -44.38% -46.05% 1180% 639%

Technicolor FR Y -10.26% 0.05% -1.65% 46% 85% -39.72% -42.80% -43.94% 370% 646%

Qualcomm US Y

Technicolor FR Y

J0023 RWTH Aachen Univ. DE Y JEM 7.0 -0.79% -1.52% -1.52% 440% 122% -33.68% -44.16% -44.37% 3507% 927%

Samsung KR Y

Huawei CN Y

GoPro US N

HiSilicon CN Y

Huawei CN Y

GoPro US N

HiSilicon CN Y

Samsung KR Y

Sharp JP Y

Foxconn TW N

NHK JP N -2.14% -5.55% -5.61% 237% 214% -34.57% -45.96% -46.32% 1890% 1630%

SHARP JP Y -3.26% -6.48% -6.57% 273% 257% -35.28% -46.42% -46.86% 2175% 1955%

J0028 Sony JP Y JEM -8.15% -8.66% -8.80% 644% 223% -38.41% -47.54% -48.07% 5133% 1830%

J0029 Tencent CN N NextSoftware -4.70% -8.34% -8.91% 242% 125% -36.17% -47.49% -48.15% 1928% 954%

J0031 Bristol Univ. UK N JEM 7.0 -4.54% 20.19% 18.68% 90% 262% -36.09% -19.30% -21.72% 767% 1678% Y

USTC CN Y

Peking Univ. CN N

HIT CN N

Wuhan Univ. CN N

Y

CNN

Y

CS1(Over HM16.16)Organizations Country code baseResponse HEVC

-1.57% -0.71%

Doc #CS1(Over JEM7.0)

-1.72% 100% 381%JEM7.0

J0012 JEM 7.0

-34.19% -43.75% -44.37% 765% 2911%

-33.73% -43.92% -44.10% 777% 777%

J0011

J0013 JEM 7.0

-0.90% -1.14% -1.15% 103% 98%

0.64% -0.39% -0.89% 105% 53% -32.74% -43.48% -43.89% 841% 404%

J0015 JEM

NextSoftwareJ0014 Fraunhofer HHI DE Y

-3.98% -3.28% -3.16% 205% 33% -35.72% -44.75% -44.95% 1710% 263%

J0021

J0018 Media Tek TW Y JEM

J0020

-13.60%

JEM7.0

JEM

Panasonic JP Y

-41.86% -44.77% -46.13% 728% 582%

-37.00% -35.96% -37.39% 902%

J0022

NEWJ0024

-3.80% -5.63% 90% 82%JEM

-6.01%

-4.24%

10.34% 8.53%

10.71% 9.23%

120%

68%

36%

39% -35.68% -36.17% -37.33% 513% 296%

274%

-38.76% -38.74% -40.30% 1058% 281%

-37.20%

J0025

-6.31%

NEW

-8.78% 5.89% 3.83% 141% 45%

-36.02% -37.55% 1043% 283%

-8.15% -8.66% -8.80% 644% 223%

10.11% 7.59% 139% 45%

J0027

JEM

J0026

JEM

-47.54% -48.07% 5133% 1830%JEM

J0032 281824%-10.11% -9.59% -9.97% 527% -39.63% -48.10% -48.69% 2184%79868%

-38.41%

Y

Y

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 13

Performance of SDR

• JVET-J0080: Report of subjective evaluation

contains 28 plots as below, one per sequence

HM

JEM

Rate 1...4

Proposals ranked by MOS (per rate)

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 14

WD1 / VTM1• SW code base: Next Software (HHI)

• Block structure– QTBTTT

– Unified tree (coding block unites prediction and transform)

– CTU size: 128x128, Maximum transform size 64x64

– Smallest luma block size 4x4

• Some removed elements of HEVC: – Mode dependent transform (DST-VII), mode dependent scan

– Strong intra smoothing

– Sign data hiding in transform coding

– High-level syntax (e.g. VPS)

– Tiles and wavefront

– Quantization weighting

14

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 15

Current Performance of VVC

• PSNR-based Common Test Conditions BD-Rate savings

relative to HEVC reference software (10 bit)

15

vs HM AI RA

gain Enc. Dec. gain Enc. Dec.

VTM 1.0 4% 9.6X 1.1X 8% 2.2X 0.8X

BMS 1.0 15% 98X 2.2X 23% 9.3X 2.3X

VTM 2.0 18% 18X 1.6X 23% 3.7X 1.3X

AI RA

gain Enc. Dec. gain Enc. Dec.

VTM 2.0 vs.

VTM 1.015% 1.9X 1.5X 16% 1.7X 1.5X

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 16

Point Cloud Compression

(PCC)

16

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.

Point Cloud

Static Objects and Scenes

(Category 1)

Dynamic Objects

(Category 2)

Dynamic Acquisition

(Category 3)

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.

Point Cloud

• A set of 3D points

– Not ordered,

– Without relations

between them

• Each point is

defined by

– (X, Y, Z)

– Attribute

• (R, G, B) or (Y, U, V)

• Reflectance,

transparency

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 19

PCC Timeline

19

PCC

Extension

2017

CfP

2018

Review CfP results

Develop PCC video standard

01 04 07 10 01 04 07 10

2019 2020

01 04 07 10 01 04 07 10

FDIS

CD

TM established

2014

WD establishedIssue CfP

Initial PCC

…

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 20

MPEG 123 meeting of PCC

• 4ndth F2F Meeting of PCC after CfP

• Date: 15 ~ 20 July, 2018

• Approximately 60+ participants

• 132 technical contributions

33%

67%

Cat.13 Cat2.

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 21

PCC Participants

21

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 22

Num. of contributions in PCC

22

40

15

0

15

57

67

3640

36

120 121 123

Percentage of each category

TMC1 TMC2 TMC3&13

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 23

Coded Representation of

Neural Networks (NNR)

23

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 24

Coded Representation of

Neural Networks (NNR)

• 3rd AHG meeting

• April 15 2018

• 20+ participants– ETRI, Fujitsu, Hanyang Univ., Huawei, Mitsubishi,

NEC, Nokia, Peking Univ.

• 9 contributions

24

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.

Overview of the evaluation process

25

• Image classification

• Feature extraction for compact video descriptors (CDVA)

• NN based components for video compression

• Classification of health care records

• (Re)training for machine reading comprehension (MRC)

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI.

Coded Representation of Neural

Networks• Call for test data

– Visual analysis, image coding, text

understanding

– Test data, training data, network, compressed

network

– Audio data

• Call for Evidence

– Submission 10/2018

All rights reserved. No part of this confidential report may be reproduced in any form or by any means without written permission from ITRI. 27

Thank You

27