Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRTМаксим Милаков, NVIDIA

v2

• You will learn:• How GPUs are used for DL now?

• Why do you want to use GPUs for inference?

• Why do you want to use TensorRT for inference on GPUs?

• This talk is NOT:• An intro to DL

• A set of code samples

NVIDIAThe AI Computing Company

NVIDIA Powering the Deep Learning Ecosystem

DEEP LEARNING FRAMEWORKS

COMPUTER VISION SPEECH AND AUDIO NATURAL LANGUAGE PROCESSING

Object Detection Voice Recognition Language TranslationRecommendation

EnginesSentiment Analysis

Mocha.jl

Image Classification

NVIDIA DEEP LEARNING SDK

NCCLcuDNN cuBLAS cuSPARSE TensorRT

ML development and deployment cycle

Training with SGD backpropagation

ImageNet: results for 2010-2014

15%

83%

95%28%26%

15%

11%

7%

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0%

5%

10%

15%

20%

25%

30%

2010 2011 2012 2013 2014

% Teams using GPUs

Top-5 error

Deployment scenarios - Hyperscale

• Input generated and output used at the client device

• Inference is running at data center

• High throughput

• On-the-fly batching

TensorRT for Hyperscale

Image Classification Object Detection

Image Segmentation

---

Deployment scenarios - Embedded

• On-device inference

• Small-batch inference

• Low latency

GPU Inference Engine for Automotive

PedestrianDetection

Lane Tracking

Traffic SignRecognition

---

NVIDIA DRIVE PX 2

TensorRT

High-performance deep learning inference for production deployment

0

1

2

3

4

5

6

7

8

1 8 128

CPU-Only Tesla M4 + TensorRT

Up to 16x More Inference Efficiency

Img/sec/watt

Batch Sizes

GoogLenet, CPU-only vs Tesla M4 + TensorRT on

Single-socket Haswell E5-2698 v3@2.3GHz with HT

EMBEDDED

Jetson TX1

AUTOMOTIVE

Drive PX

DATA CENTER

Tesla M4

Comparing to DL frameworks

• Particularly effective at small batch-sizes

• Improves perf for complex networks the most

Jetson TX1 HALF2 column uses fp16

GoogleNet Performance

TensorRT

• Fuse network layers

• Eliminate concatenation layers

• Kernel specialization

• Auto-tuning for target platform

• Select optimal tensor layout

• Batch size tuningTRAINEDNEURAL NETWORK

OPTIMIZEDINFERENCERUNTIME

Layers supported

• v1 designed for 2D images

• Layers supported• Convolution: 2D

• Activation: ReLU, tanh and sigmoid

• Pooling: max and average

• ElementWise: sum, product or max of two tensors

• LRN: cross-channel only

• Fully-connected: with or without bias

• SoftMax: cross-channel only

• Deconvolution

• Custom layers possible with sandwich approach now

Optimizations

• Eliminate unused layers

• Vertical layer fusion: Fuse convolution, bias, and ReLU layers to form a single layer

• Horizontal layer fusion: Combine layers with the same source tensor and the same parameters

Optimizations: Original network

concat

max pool

input

next input

3x3 conv.

relu

bias

1x1 conv.

relu

bias

1x1 conv.

relu

bias

1x1 conv.

relu

bias

concat

1x1 conv.

relu

bias5x5 conv.

relu

bias

Optimizations: Vertical layer fusion

concat

max pool

input

next input

concat

1x1 CBR 3x3 CBR 5x5 CBR 1x1 CBR

1x1 CBR 1x1 CBR

Optimizations: Horizontal layer fusion

concat

max pool

input

next input

concat

3x3 CBR 5x5 CBR 1x1 CBR

1x1 CBR

Optimizations: Concat elision

max pool

input

next input

3x3 CBR 5x5 CBR 1x1 CBR

1x1 CBR

TensorRT – 2 phases deployment

• Build• Apply optimizations on the network configuration

• Generate an optimized plan for computing the forward pass

• Deploy• Forward and output the inference result

Build

Deploy

File

Model

File

Deploy

Plan

Output

I/O

Layers

Max

Batchsize

Inputs

Batch

size

Pascal GPUs for inference

Maximum Efficiency for Scale-out Servers

TESLA P4 TESLA P40

Highest Throughput for Scale-up Servers

P40/P4 – New “Int8” for Inference

TensorRT v2: int8 accuracy

• Almost the same accuracy for major models

• Still working on the procedure to make quantization optimal

TensorRT v2: int8 performance

• Up to 3x performance• No FFT and Winograd yet

• Constant factors

• Beneficial on small batches even on bigger P40

TensorRT v2: more performance

178480

1,514

4,121

3,200

6,514

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

E5-2690v414 Core

M4(FP32)

M40(FP32)

P100(FP16)

P4(INT8)

P40(INT8)

All results are measured, based on GoogLenet with batch size 128Xeon uses MKL 2017 GOLD with FP32, GPU uses TensorRT internal development ver.

>35x

1.4

12.3 10.6

27.9

91.1

56.3

0

20

40

60

80

100

E5-2690v414 Core

M4(FP32)

M40(FP32)

P100(FP16)

P4(INT8)

P40(INT8)

>60x

P40 For Max Inference Throughput,

img/sec

P4 For Max Inference Efficiency,

img/sec/watt

Deep Learning Everywhere

• developer.nvidia.com/tensorrt

• developer.nvidia.com/deep-learning

• developer.nvidia.com/cuda-zone

• mmilakov@nvidia.com

Backup slides

Tesla Products Decoder