Preferred NetworksChainer, SensorBee

2016 7 23 ACM SIGMOD 62( 25 i i Web )

Preferred Networksg

oono@preferred.jp tanakad@preferred.jp

• (twitter: @delta2323_)– h → 2012.4 PFI → 2014.10 PFN– h g gChainer

• (twitter: @disktnk)– h → SIer → 2015.4 PFN– h eSensorBee e

2

• Preferred Networks 4• Chainer

–– i i– Chainer t

• SensorBee– i i– i i SensorBee– Chainer SensorBee

3

Preferred Networks

• 2014 3 kPreferred Infrastructurex l• h e h ו h 40 ku 35 g i il•

– 2014 10 k lk2 l– 2015 8 FANUCk9 l– 2015 10 k10 l

• g– eFANUCe ePanasoniceNVIDIAeCiRA

4

LML L L V R

• VD # y z v –i

•a y za t e z – z ™

•a – za t ™e ™

•a e i e i –

e ea – te x

-

AutomotiveHumanoid Robot

Preferred Networks Industrial IoT – AI

6

Consumer Industrial

Cloud

Device

PhotoGameText

Speech

Infrastructure

FactoryRobot

Automotive

Healthcare

Smart City

Industry4.0

Industrial IoT

PFN

DIMo i

• 2 OSS te i1. SensorBee: x g i2. Chainer: x t t i i3. DIMo i : g – i

:IoT i i

: i i i i

Dee

p In

telli

genc

ein

-Mot

ion

(DIM

o)

…

???

…

7

• Preferred Networks– t• Chainer

–– i i– Chainer t

• SensorBee– i i– i i SensorBee– Chainer SensorBee

8

=

• L y t y u–• ™

• L i t• w i y i

argminw∑(x, y) L(x, y; w)wh ixhyhLh

h k x – y l

9

Neural Network as a Computational Graph

• i i i i i i x DAGkDirected Acyclic Graphl z

y = x1 * x2z = y - x3

x1 mul suby

x3

z

x2

data node

operator node10

kForward Propagationl

– i

ve –

x

w

_ ** 2

2 * _ _ * _

SCE L

_ + _ z

t

11

x = chainer.Variable(np.array(…))

# same as w, t

z = x ** 2 + 2 * x * w

L = SoftmaxCrossEntropy(z, t)

i k l

• 2• k L i w– e�wLl• w – x

initialize wuntil convergence:

get data (x, y)w← w - η�w L(x, y; w)

ηh

← – i ys

i – – z

12

kChain Rulel

i

x f y

x f y

6 7

Forward Propagation

L e

13

kBackward Propagationl

• x ik l

x

w

_ ** 2

2 * _ _ * _

SCE L

_ + _ z

�z L

�w L

�x L

�L L = 1

t

�t L

14

1. = + i→ i i – t

2.→ t –

3. –e i→ t –

15

L(x, y, w)

�w L

w← w - η�w L

• Preferred Networks– t• Chainer

–– d– Chainer t

• SensorBee– i i– i i SensorBee– Chainer SensorBee

16

i i

17

Prepare the training dataset

Repeat until meeting some criterionPrepare for the next (mini) batch

Compute the loss (forward prop)

Initialize the NN parameters

Save the NN parameters

Define how to compute the loss of this batch

Compute the gradient (backprop)Update the NN parameters

automated

i i

name functions example

E D BG E (- - BFCE C E

B EB B CE CA B A BG

( G F G B A BGE B B /CCD

. E F / F BC F / EB

CADHG G CB E DA B A BG

H CADHG G CB E DCE E DECD DECD

BC BFCE CCE BB

H G A BF CBEE L E EL

/ B E E HA L H L) B GCE CE

HA E CADHG G CBD

GE K CD E G CBCBIC HG CB

/ H / H(

, E E

18

i i

• NN i• NN i – 2 k l

– t i → i e y– i t g t → –

• t – i y– t i i es t y tNN

Deep Networks with Stochastic DepthG. Huang et al. 2016

Ladder Network A. Rasmus et. al. 2015

End-to-End Memory NetworkS. Sukhbaatar et. al. 2015

– k ly“ ” ” ” –

19

Chainer

• h 9 e 9

• h*()- . 1

• i

• a

.

• h= D 8

• NN

ü NN Pythonü i u NN

ü NN kDefine-by-Runlü Python i

ü i – NNü CuPy– CPU/GPU agnostic i

20

cuDNN

Chainer

NumPy CuPy

BLAS cuBLAScuRAND

CPU GPU

name

E D BG E

B EB B CE CA B A BGCADHG G CB E D

A B A BGH G A BF CB

EE L E ELHA E CADHG G CB

D

, E E

* NumPyhPython f yw e Python i i yNumPy i – t

21

i i

Python C++ Lua Python C++/Python

n Preferred Networks /Infrastructure

BVLC Idiap Research Institute, DeepMind

Univ. of Montreal

Google

2( n) o n n pjn

RNN/LSTM 1 c n n n

0 c c c c

0 p n n n

DSL(Python) (prototxt)

DSL(YAML)

DSLkPythonl

LuaJIT–i

i

× GPUg×i i

TensorBoard–

22

f g

x f g

i i

x yf

x = chainer.Variable(...)y = f(x)z = g(x)

zg

i i3

7LMPUL B U 7LMPUL B U

v• Define-and-Run = NN• Define-by-Run = NN i23

i

yfor xi, yi in data:

x = Variable(xi)y = Variable(yi)z = x + 2 * y

7LMPUL B U7LMPUL B U

x = Variable(‘x’)y = Variable(‘y’)z = x + 2 * y

for xi, yi in data:eval(z, x=xi, y=yi))

24

h i –MNIST

class MLP(Link):def __int__(self):super(MLP, self).__init__(

l1=Linear(784, 1000),l2=Linear(1000, 1000),l3=Linear(1000, 10))

def __call__(x):h1 = F.relu(self.l1(x))h2 = F.relu(self.l2(l1))return self.l3(h2)

model = L.Classifier(MLP()) optimizer = Adam()optimizer.setup(model)

Linear l1r

W bias

ReLU

Linear l2h1

W bias

ReLU

Linear l3h2

W bias

25

h i – MNISTtrain, test = get_mnist()train_iter = ShuffledIterator(train, 128)test_iter = SequentialIterator(test, 128, False)

updater = StandardUpdater(train_iter, optimizer)trainer = Trainer(updater, (10, 'epoch'))

trainer.extend(Evaluator(test_iter, model))trainer.extend(dump_graph('main/loss'))trainer.extend(snapshot())trainer.extend(LogReport())trainer.extend(PrintReport(

'epoch', 'main/accuracy','validation/main/accuracy']))

trainer.extend(ProgressBar())

trainer.run()

26

Debuggubility

class MLP(Link):def __int__(self):super(MLP, self).__init__(

l1=Linear(700, 1000), # Oops!l2=Linear(1000, 1000),l3=Linear(1000, 10))

27

Chainer –w

• t – ze y t• NVIDIAy i i i• Python NumPy i i y x v• th e e ie• (PFN ) ve i y t

NVIDIA CEONVIDIA cuDNN 28

• Preferred Networks– t• Chainer

–– i i– Chainer 4

• SensorBee– i i– i i SensorBee– Chainer SensorBee

29

t i

30

CES 2016– eNTT

t –

(iREX) i –

31

–

t

x i

(iREX) i –

32

–

b

)-

(iREX) i –

33

– ™

Panasonic

34

–

http://qiita.com/advent-calendar/2015/chainer#day-13

35

• Preferred Networks– t• Chainer

–– Chainer– Chainer t

• SensorBee–– i i SensorBee– Chainer SensorBee

36

l i y u–a ) )((D5e i e

IoT

37

l i –a i e ia i 2 i i tg z ta 2 i – i t

IoT i

38

l i x – t ta i i ia y – e u

i i

39

i i

• – i– – i y– 7 x 7

• i– i i e e e i etc– i e u

- m m- m × m i

- x→

• x i – y i→ b

40

• Preferred Networks– t• Chainer

–– Chainer– Chainer t

• SensorBee– i i– SensorBee– Chainer SensorBee

41

i – t

• – ™ e i–

- 1- – (= i) z ( )

–- (Continuous Queries)

- t y- tx– – x

– 1 event (=tuple) t

• i i i– Strom, Spark Streaming, Samza, Kafka Streaming, Flink, Heron...

( e y t t)

42

i

• Data Stream Management System (DSMS)

• C L T 7 S MV T

6VU PU V A L PLC L TPUN U C L TPUN

L T

)

+

,*

http://www.confluent.io/kafka-summit-2016-keynote-the-rise-of-real-time-jay-kreps p10

R :VS I = D TL Özsu. Data Stream Management, 2010.

43

SensorBee: Intelligent Streaming ETL for IoT

• i s t i (Extract) w(Transform) v i

• (*1) x t fGo f• BQL tu SQL – i (*2)

6 TL

FLOP SL

BV L

CLU V

MS LU

B75=C

CL O8UNPUL

VCA

7 CV L 7 7L PU PVU

CLU V 5LL

) O 2 P V VT T LUG VS PVU LU PV V VT PUN V L PL M * O 2 SP L O L UL P R UR LU V ILL -10/(0// ),

44

SensorBee: Flexibility and Extensibility

• jubatus ( g)

• opencv epymlstateetc...

6 TL

FLOP SL

BV L

CLU V

MS LU

B75=C

CL O8UNPUL

VCA

7 CV L 7 7L PU PVUCLU V 5LLd

S N PU S N PU

E79

E L 7LMPUL 9 U PVU E79 E L7LMPUL C L E7C –

e y

E7C

= OPUL SL UPUNb 6S PMP PVUb BLN L PVUb IQL L VNUP PVUb 4UVT S_ L L PVU

45

• Preferred Networks– t• Chainer

–– Chainer– Chainer t

• SensorBee– i i– i i SensorBee– Chainer SensorBee y

46

1: Twitter • z

•

a D V P SO 2 LU V ILL L OL V V N LU S L V P S O TS PUNT OPUL SL UPUN

a Elasticsearch –http://www.slideshare.net/nobu_k/elasticsearch-59627321

D P L

:LUL

4NL

9V TPUN ILSP

UN MS LU

47

6

CLU V 5LL

2: t i

4B×i i

4B×i i

DD i

DD i

6

CLU V 5LL

48

3:

SensorBee

6VT L P PVU7L L D R BL VNUP L

http://monoist.atmarkit.co.jp/mn/articles/1512/07/news083.html ( i )http://techon.nikkeibp.co.jp/atcl/news/16/041801679/?i_cid=nbptec_sied_rel ( i )http://www.fanuc.co.jp/ja/whatsnew/notice/osirase20160418.html ( i )49

4: –

SensorBee

DPTL L PLB BL L LU PVU

50

• Chainer define-by-run i iflexibility debuggabilityy

• sensorbee UDFgUDS– IoT i

• PFN Chainer– SensorBee– –eR&D t

51

52