Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRTМаксим Милаков, NVIDIA
v2
• You will learn:• How GPUs are used for DL now?
• Why do you want to use GPUs for inference?
• Why do you want to use TensorRT for inference on GPUs?
• This talk is NOT:• An intro to DL
• A set of code samples
NVIDIAThe AI Computing Company
NVIDIA Powering the Deep Learning Ecosystem
DEEP LEARNING FRAMEWORKS
COMPUTER VISION SPEECH AND AUDIO NATURAL LANGUAGE PROCESSING
Object Detection Voice Recognition Language TranslationRecommendation
EnginesSentiment Analysis
Mocha.jl
Image Classification
NVIDIA DEEP LEARNING SDK
NCCLcuDNN cuBLAS cuSPARSE TensorRT
ML development and deployment cycle
Training with SGD backpropagation
ImageNet: results for 2010-2014
15%
83%
95%28%26%
15%
11%
7%
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0%
5%
10%
15%
20%
25%
30%
2010 2011 2012 2013 2014
% Teams using GPUs
Top-5 error
Deployment scenarios - Hyperscale
• Input generated and output used at the client device
• Inference is running at data center
• High throughput
• On-the-fly batching
TensorRT for Hyperscale
Image Classification Object Detection
Image Segmentation
---
Deployment scenarios - Embedded
• On-device inference
• Small-batch inference
• Low latency
GPU Inference Engine for Automotive
PedestrianDetection
Lane Tracking
Traffic SignRecognition
---
NVIDIA DRIVE PX 2
TensorRT
High-performance deep learning inference for production deployment
0
1
2
3
4
5
6
7
8
1 8 128
CPU-Only Tesla M4 + TensorRT
Up to 16x More Inference Efficiency
Img/sec/watt
Batch Sizes
GoogLenet, CPU-only vs Tesla M4 + TensorRT on
Single-socket Haswell E5-2698 [email protected] with HT
EMBEDDED
Jetson TX1
AUTOMOTIVE
Drive PX
DATA CENTER
Tesla M4
Comparing to DL frameworks
• Particularly effective at small batch-sizes
• Improves perf for complex networks the most
Jetson TX1 HALF2 column uses fp16
GoogleNet Performance
TensorRT
• Fuse network layers
• Eliminate concatenation layers
• Kernel specialization
• Auto-tuning for target platform
• Select optimal tensor layout
• Batch size tuningTRAINEDNEURAL NETWORK
OPTIMIZEDINFERENCERUNTIME
Layers supported
• v1 designed for 2D images
• Layers supported• Convolution: 2D
• Activation: ReLU, tanh and sigmoid
• Pooling: max and average
• ElementWise: sum, product or max of two tensors
• LRN: cross-channel only
• Fully-connected: with or without bias
• SoftMax: cross-channel only
• Deconvolution
• Custom layers possible with sandwich approach now
Optimizations
• Eliminate unused layers
• Vertical layer fusion: Fuse convolution, bias, and ReLU layers to form a single layer
• Horizontal layer fusion: Combine layers with the same source tensor and the same parameters
Optimizations: Original network
concat
max pool
input
next input
3x3 conv.
relu
bias
1x1 conv.
relu
bias
1x1 conv.
relu
bias
1x1 conv.
relu
bias
concat
1x1 conv.
relu
bias5x5 conv.
relu
bias
Optimizations: Vertical layer fusion
concat
max pool
input
next input
concat
1x1 CBR 3x3 CBR 5x5 CBR 1x1 CBR
1x1 CBR 1x1 CBR
Optimizations: Horizontal layer fusion
concat
max pool
input
next input
concat
3x3 CBR 5x5 CBR 1x1 CBR
1x1 CBR
Optimizations: Concat elision
max pool
input
next input
3x3 CBR 5x5 CBR 1x1 CBR
1x1 CBR
TensorRT – 2 phases deployment
• Build• Apply optimizations on the network configuration
• Generate an optimized plan for computing the forward pass
• Deploy• Forward and output the inference result
Build
Deploy
File
Model
File
Deploy
Plan
Output
I/O
Layers
Max
Batchsize
Inputs
Batch
size
Pascal GPUs for inference
Maximum Efficiency for Scale-out Servers
TESLA P4 TESLA P40
Highest Throughput for Scale-up Servers
P40/P4 – New “Int8” for Inference
TensorRT v2: int8 accuracy
• Almost the same accuracy for major models
• Still working on the procedure to make quantization optimal
TensorRT v2: int8 performance
• Up to 3x performance• No FFT and Winograd yet
• Constant factors
• Beneficial on small batches even on bigger P40
TensorRT v2: more performance
178480
1,514
4,121
3,200
6,514
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000
E5-2690v414 Core
M4(FP32)
M40(FP32)
P100(FP16)
P4(INT8)
P40(INT8)
All results are measured, based on GoogLenet with batch size 128Xeon uses MKL 2017 GOLD with FP32, GPU uses TensorRT internal development ver.
>35x
1.4
12.3 10.6
27.9
91.1
56.3
0
20
40
60
80
100
E5-2690v414 Core
M4(FP32)
M40(FP32)
P100(FP16)
P4(INT8)
P40(INT8)
>60x
P40 For Max Inference Throughput,
img/sec
P4 For Max Inference Efficiency,
img/sec/watt
Deep Learning Everywhere
• developer.nvidia.com/tensorrt
• developer.nvidia.com/deep-learning
• developer.nvidia.com/cuda-zone
Backup slides
Tesla Products Decoder