1. 1. Chainer Preferred Networks oono@preferred.jp 2015/9/18 GTC Japan @
2. 2. (@delta2323_) 2012.3 PFI 2014.10 PFN Chainer http://delta2323.github.io NIPS2014ICML2015 1! 2
3. 3. git clone https://github.com/pfnet/chainer.git
4. 4. Chainer http://chainer.org PFNPFI 201569 1.3.0201592 1.3.1 (9/16) 1.4.0 (9/30) MIT (Expat) HPhttp://chainer.org https://github.com/pfnet/chainer Twitter@ChainerOfficial Google GroupChainer Uesr Group Contribution Guide http://docs.chainer.org/en/stable/contribution. html PowerfulCUDAGPUcuDNN Flexible IntuitivePython4
5. 5. ChainerChainer + CuPy Chainer Python CuPyGPUNumPy* NumPy CPU GPU BLAS CUDA Toolkit cuDNN NumPy CuPy Chainer Python * NumPyPython PythonNumPy 5
6. 6. OSLinux Ubuntu 14.04 Python(Cpython) 2.7+/3.4+ Numpy1.9+Six1.9+ CUDACUDA6.5+ pip install chainer 6 pip* : Python Python
7. 7. Github Stars20155 PyLearn2 https://twitter.com/fchollet/status/635 891305084796929 7
8. 8. Github Stars20158 PyLearn2 8 https://twitter.com/fchollet/status/635 891305084796929
9. 9. Chainer Deep Q Network* * Mnih, Volodymyr, et al. "Human-level control through deep reinforcement learning." Nature 518.7540 (2015): 529-533. ** CaffeDeep Q-Network http://d.hatena.ne.jp/muupan/20141021/1413850461 *** PFI2014 http://www.ustream.tv/recorded/53153399 9
10. 10. Chainer 10 https://research.preferred.jp/2015/06/distributed-deep- reinforcement-learning/ 273 600 400 200 100 50 5 273 600 400 200 100 50 5 273 600 400 200 100 50 5
11. 11. Chainer Variational AE Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in Neural Information Processing Systems. 2014. CaffeVGG A Neural Algorithm of Artistic Style [Gatys+'15] (chainer-goch) 11
12. 12. Deep Learning Caffe Chainer n Blob Variable Layer Function Net (FunctionSet) Solver Optimizer 12 (DAG)
13. 13. GoogLeNet, NTM, Recursive Net, LSTM Chainer Caffe 167 2058 GoogleNet (2012)AlexNet*, 7 (2014) GoogLeNet**, 22 13 * ImageNet Classification with Deep Convolutional Neural Networks http://www.image-net.org/challenges/LSVRC/2012/supervision.pdf ** Szegedy, Christian, et al. "Going deeper with convolutions." arXiv preprint arXiv:1409.4842 (2014). ChainerDefine-by-Run
14. 14. Define-and-Run vs. Define-by-Run 14 f g x f g x yf x = chainer.Variable(...) y = f(x) z = g(x) zg = Define-and-Run Define-by-Run
15. 15. Forward x = chainer.Variable(np.array(1)) y = chainer.Variable(np.array(1)) z = x**2 + 2*x*y + y z.backward() Split x y _ ** 2 2 * _ _ * _ _ + _ z _ + _ chainer.Variable chainer.Function 15
16. 16. Variable.backward() Split x y _ ** 2 2 * _ _ * _ _ + _ z _ + _ x = chainer.Variable(np.array(1)) y = chainer.Variable(np.array(1)) z = x**2 + 2*x*y + y z.backward() 16
17. 17. Python (if / for / while etc) ForRNN def forward(x, t, train=True): h = F.relu(model.l1(x)) y = model.l2(h) if train: loss = F.softmax_cross_entropy(y, t) return loss else: prob = F.softmax(y) acc = F.accuracy(prob, t) return acc y sce lo ss y sm pr ob acc ac c 17
18. 18. MNIST # (1) Model definition model = FunctionSet( l1=F.Linear(784, 100), l2=F.Linear(100, 100), l3=F.Linear(100, 10)).to_gpu() # (2) Optimizer Setup opt = optimizers.SGD() opt.setup(model) # (3) Forward computation def forward(x, t): h1 = F.relu(model.l1(x)) h2 = F.relu(model.l2(h1)) y = model.l3(h2) return F.softmax_cross_entropy(y, t) # (4) Training loop for epoch in xrange(n_epoch): for i in xrange(0, N, batchsize): x = Variable(to_gpu(...)) t = Variable(to_gpu(...)) opt.zero_grads() loss = forward(x, t) loss.backward() opt.update() 784 100 100 10 0:2% 1:5% 2:90% 9:1%18
19. 19. (1) model = FunctionSet( l1=F.Linear(784, 100), l2=F.Linear(100, 100), l3=F.Linear(100, 10)).to_gpu() opt = optimizers.SGD() opt.setup(model) Function FunctionSet 19
20. 20. (2) Optimizer model = FunctionSet( l1=F.Linear(784, 100), l2=F.Linear(100, 100), l3=F.Linear(100, 10)).to_gpu() opt = optimizers.SGD() opt.setup(model) Optimizer 20
21. 21. (3) Training Loop for epoch in xrange(n_epoch): for i in xrange(0, N, batchsize): x = Variable(to_gpu(...)) t = Variable(to_gpu(...)) opt.zero_grads() loss = forward(x, t) loss.backward() opt.update() 1 21
22. 22. (3) def forward(x, t): h1 = F.relu(model.l1(x)) h2 = F.relu(model.l2(h1)) y = model.l3(h2) return F.softmax_cross_entropy(y, t) x t 22
23. 23. (4) Training Loop for epoch in xrange(n_epoch): for i in xrange(0, N, batchsize): x = Variable(to_gpu(...)) t = Variable(to_gpu(...)) opt.zero_grads() loss = forward(x, t) loss.backward() opt.update() 23
24. 24. y y truncated BPTT** x f y g z y g z y.unchain_backward() x = Variable() y = f(x) z = g(y) y.unchain_backward() * BPTTBack Propagation Through Time RNN ** truncated BPTT BPTT 24
25. 25. Caffe 25 * Caffe C++
26. 26. * hERG potassium channels and cardiac arrhythmia, Michael C. Sanguinetti & Martin Tristani-Firouzi, Nature 440, 463-469(23 March 2006) (doi:10.1038/nature04710) Fig. 5
27. 27. (QSAR) Quantitative Structure-Activity Relationship MerckQSAR (2012) * 27 QSAR HTS * Dahl, George E., Navdeep Jaitly, and Ruslan Salakhutdinov. "Multi- task Neural Networks for QSAR Predictions." arXiv preprint arXiv:1406.1231 (2014). ** http://blog.kaggle.com/2012/10/31/merck-competition-results- deep-nn-and-gpus-come-out-to-play/ **
28. 28. [Dahl+14] 28 2-3 500-2500/ Dropout Minibatch SGD 10010011010100010 1 0 Active!! 1 0 Active!! 0 1 Inactive!! PubChem 19 20 DNN B
29. 29. Community Learning TSUBAME H27 29 PubChem 2 100 SoftTarget SoftTarget Data-Parallel SoftTarget MPI TSUBAME 3GPU(K40), 4GB/ >100
30. 30. Community Learning Community Learning8 30 () () 1 10.5719 1 0.0318 2 5.2267 x 2.022 0.1377 3 3.9455 x 2.679 0.1284 4 2.5978 x 4.070 0.1367 8 1.5417 x 6.857 0.1281 NNChainer mshadow
31. 31. Community Learning 5Community Learning 31 ID [Dahl+ '14] Community Learning 1851 (1a2) 0.926 0.938 0.9387 1851 (2c19) 0.897 0.903 0.9413 1851 (2c9) 0.889 0.907 0.9274 1851 (2d6) 0.863 0.861 0.8913 1851 (3a4) 0.895 0.897 0.9214 NNChainer mshadow
32. 32. ChainerPython Define-by-Run Chainer HPhttp://chainer.org https://github.com/pfnet/chainer Twitter@ChainerOfficial Google GroupChainer Uesr Group Contribution Guide http://docs.chainer.org/en/stable/contribution.html Your Contribution is Welcome!! 32 And You are Welcome!! We are hiring :)
33. 33. Copyright 2014- Preferred Networks All Right Reserved.