of 19 /19
Spray(Akka)JVMCPUスケー ルさせるために行った事

Jvm operation casual talks

Embed Size (px)

DESCRIPTION

 

Citation preview

Page 1: Jvm operation casual talks

Spray(Akka)でJVMをCPUスケールさせるために行った事

Page 2: Jvm operation casual talks

渡辺 雄作(35) / @waysaku

ソフトウェアエンジニア

株式会社サイバーエージェント アドテク本部

(Webシステム)

Page 3: Jvm operation casual talks

Spray(Akka)でJVMをCPUスケールさせるために行った事

Page 4: Jvm operation casual talks

spray is an open-source toolkit for building REST/HTTP-based integration layers on top of Scala and Akka. Being asynchronous, actor-based, fast, lightweight, modular and

testable it's a great way to connect your Scala applications to the world.

Page 5: Jvm operation casual talks

Scalaコードを読めない人(技術的、もしくはポリシー的に)がAkkaアプリケーション運用を任されて

しまった

JVM上で動くノンブロッキング、非同期なアーキテクチャを採用しているアプリケーション

対象システム

対象者

Page 6: Jvm operation casual talks

リリース数日前の負荷試験、、、

何が起きたか

スループットが全然出ない!

Page 7: Jvm operation casual talks

• Akkaの仕組みを理解する

解決までの道のり

• Linuxの基本的な情報を取得• Thread Dump取得

Page 8: Jvm operation casual talks

Actor

Dispacher

Actor

ActorActor

Thread ThreadThread Thread

Actor

Actor

ひもづける

• Akkaの仕組みを理解する

Page 9: Jvm operation casual talks

Dispacher

ThreadThread

Akkaの基本的な考え方

Thread

Actor

Actor Actor

Actor

Actor

Thread

Actor!!

Page 10: Jvm operation casual talks

Cpu0 : 2.3%us, 0.7%sy, 0.0%ni, 97.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu1 : 0.3%us, 0.3%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 92.0%us, 6.0%sy, 0.0%ni, 2.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 5.4%us, 1.7%sy, 0.0%ni, 93.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

スループットが全然出ないと思ったら、CPU一個しか使ってない!

そのくせロードアベレージが無駄に高い

• Linuxの基本的な情報を取得

Page 11: Jvm operation casual talks

Thread Dumpを取得する

"Attach Listener" daemon prio=5 tid=0x00007f8de5852000 nid=0x3b07 waiting on condition [0x0000000000000000] "DestroyJavaVM" prio=5 tid=0x00007f8de280b800 nid=0x1903 waiting on condition [0x0000000000000000] "default-akka.actor.default-dispatcher-4" prio=5 tid=0x00007f8de28d8000 nid=0x5f03 waiting on condition [0x000000011fb12000] "default-akka.actor.default-dispatcher-3" prio=5 tid=0x00007f8de28e2800 nid=0x5d03 waiting on condition [0x000000011fa0f000] "default-akka.actor.default-dispatcher-2" prio=5 tid=0x00007f8de3158000 nid=0x5b03 waiting on condition [0x000000011f90c000] "default-scheduler-1" prio=5 tid=0x00007f8de3854800 nid=0x5903 waiting on condition [0x000000011f809000] "Service Thread" daemon prio=5 tid=0x00007f8de5001000 nid=0x5303 runnable [0x0000000000000000] "C2 CompilerThread1" daemon prio=5 tid=0x00007f8de4805000 nid=0x5103 waiting on condition [0x0000000000000000] "C2 CompilerThread0" daemon prio=5 tid=0x00007f8de303c800 nid=0x4f03 waiting on condition [0x0000000000000000] "Signal Dispatcher" daemon prio=5 tid=0x00007f8de303b800 nid=0x4d03 runnable [0x0000000000000000] "Finalizer" daemon prio=5 tid=0x00007f8de3803800 nid=0x3903 in Object.wait() [0x000000011e8a2000] "Reference Handler" daemon prio=5 tid=0x00007f8de3034800 nid=0x3703 in Object.wait() [0x000000011e79f000] "VM Thread" prio=5 tid=0x00007f8de3033800 nid=0x3503 runnable "GC task thread#0 (ParallelGC)" prio=5 tid=0x00007f8de281a800 nid=0x2503 runnable "GC task thread#1 (ParallelGC)" prio=5 tid=0x00007f8de4800000 nid=0x2703 runnable "GC task thread#2 (ParallelGC)" prio=5 tid=0x00007f8de281b800 nid=0x2903 runnable "GC task thread#3 (ParallelGC)" prio=5 tid=0x00007f8de281c000 nid=0x2b03 runnable "GC task thread#4 (ParallelGC)" prio=5 tid=0x00007f8de281c800 nid=0x2d03 runnable "GC task thread#5 (ParallelGC)" prio=5 tid=0x00007f8de5000000 nid=0x2f03 runnable "GC task thread#6 (ParallelGC)" prio=5 tid=0x00007f8de4801000 nid=0x3103 runnable "GC task thread#7 (ParallelGC)" prio=5 tid=0x00007f8de5800000 nid=0x3303 runnable "VM Periodic Task Thread" prio=5 tid=0x00007f8de480d800 nid=0x5503 waiting on condition

Dynamic Attach機能に利用される(jmap, jstackなどのJVM内部の情報取得や挙動の操作など)

VMをアンロードするためのリスナー(JVMアプリケーション以外からJVMを利用している場合にVMをDestroyできるように)

JIT(HotSpot)で使うネイティブコンパイルスレッド()

GCを走らせるなどのVMのオペレーションを行うスレッド

マイナーGC用のスレッド

finalizeメソッドが定義されているオブジェクトの実行スレッド

OSからのシグナルを受け付ける

Page 12: Jvm operation casual talks

"Attach Listener" daemon prio=5 tid=0x00007f8de5852000 nid=0x3b07 waiting on condition [0x0000000000000000] "DestroyJavaVM" prio=5 tid=0x00007f8de280b800 nid=0x1903 waiting on condition [0x0000000000000000] "default-akka.actor.default-dispatcher-4" prio=5 tid=0x00007f8de28d8000 nid=0x5f03 waiting on condition [0x000000011fb12000] "default-akka.actor.default-dispatcher-3" prio=5 tid=0x00007f8de28e2800 nid=0x5d03 waiting on condition [0x000000011fa0f000] "default-akka.actor.default-dispatcher-2" prio=5 tid=0x00007f8de3158000 nid=0x5b03 waiting on condition [0x000000011f90c000] "default-scheduler-1" prio=5 tid=0x00007f8de3854800 nid=0x5903 waiting on condition [0x000000011f809000] "Service Thread" daemon prio=5 tid=0x00007f8de5001000 nid=0x5303 runnable [0x0000000000000000] "C2 CompilerThread1" daemon prio=5 tid=0x00007f8de4805000 nid=0x5103 waiting on condition [0x0000000000000000] "C2 CompilerThread0" daemon prio=5 tid=0x00007f8de303c800 nid=0x4f03 waiting on condition [0x0000000000000000] "Signal Dispatcher" daemon prio=5 tid=0x00007f8de303b800 nid=0x4d03 runnable [0x0000000000000000] "Finalizer" daemon prio=5 tid=0x00007f8de3803800 nid=0x3903 in Object.wait() [0x000000011e8a2000] "Reference Handler" daemon prio=5 tid=0x00007f8de3034800 nid=0x3703 in Object.wait() [0x000000011e79f000] "VM Thread" prio=5 tid=0x00007f8de3033800 nid=0x3503 runnable "GC task thread#0 (ParallelGC)" prio=5 tid=0x00007f8de281a800 nid=0x2503 runnable "GC task thread#1 (ParallelGC)" prio=5 tid=0x00007f8de4800000 nid=0x2703 runnable "GC task thread#2 (ParallelGC)" prio=5 tid=0x00007f8de281b800 nid=0x2903 runnable "GC task thread#3 (ParallelGC)" prio=5 tid=0x00007f8de281c000 nid=0x2b03 runnable "GC task thread#4 (ParallelGC)" prio=5 tid=0x00007f8de281c800 nid=0x2d03 runnable "GC task thread#5 (ParallelGC)" prio=5 tid=0x00007f8de5000000 nid=0x2f03 runnable "GC task thread#6 (ParallelGC)" prio=5 tid=0x00007f8de4801000 nid=0x3103 runnable "GC task thread#7 (ParallelGC)" prio=5 tid=0x00007f8de5800000 nid=0x3303 runnable "VM Periodic Task Thread" prio=5 tid=0x00007f8de480d800 nid=0x5503 waiting on condition

Akkaのスレッド

Page 13: Jvm operation casual talks

"default-akka.actor.default-dispatcher-24" prio=5 tid=0x00007fef1e054800 nid=0x9003 waiting for monitor entry [0x0000000114423000] java.lang.Thread.State: BLOCKED (on object monitor) at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:211) - waiting to lock <0x0000000701d22d00> (a ch.qos.logback.core.spi.LogbackLock) at ch.qos.logback.core.rolling.RollingFileAppender.subAppend(RollingFileAppender.java:175) at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:103) at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:88) at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48) at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:272) at ch.qos.logback.classic.Logger.callAppenders(Logger.java:259) at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:441) at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:395)

Blockしてる

Page 14: Jvm operation casual talks

"default-akka.actor.default-dispatcher-24" prio=5 tid=0x00007fef1e054800 nid=0x9003 waiting for monitor entry [0x0000000114423000] java.lang.Thread.State: BLOCKED (on object monitor) at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:211) - waiting to lock <0x0000000701d22d00> (a ch.qos.logback.core.spi.LogbackLock) at ch.qos.logback.core.rolling.RollingFileAppender.subAppend(RollingFileAppender.java:175) at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:103) at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:88) at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48) at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:272) at ch.qos.logback.classic.Logger.callAppenders(Logger.java:259) at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:441) at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:395)

Lockを取得しようとして待ってる

"default-akka.actor.default-dispatcher-18" prio=5 tid=0x00007fef1a99f800 nid=0x7f03 waiting for monitor entry [0x0000000113e11000] java.lang.Thread.State: BLOCKED (on object monitor) at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:211) - locked <0x0000000701d22d00> (a ch.qos.logback.core.spi.LogbackLock) at ch.qos.logback.core.rolling.RollingFileAppender.subAppend(RollingFileAppender.java:175)

Lockしてる

Page 15: Jvm operation casual talks

ch.qos.logback.core.OutputStreamAppender#subAppend(E event)

synchronized (lock) { writeOut(event); }

Block !

Page 16: Jvm operation casual talks
Page 17: Jvm operation casual talks

• Do the blocking call within an actor (or a set of actors managed by a router [Java, Scala]), making sure to configure a thread pool which is either dedicated for this purpose or sufficiently sized.

!• Do the blocking call within a Future, ensuring an upper bound on the

number of such calls at any point in time (submitting an unbounded number of tasks of this nature will exhaust your memory or thread limits).

!• Do the blocking call within a Future, providing a thread pool with an

upper limit on the number of threads which is appropriate for the hardware on which the application runs.

!• Dedicate a single thread to manage a set of blocking resources (e.g. a

NIO selector driving multiple channels) and dispatch events as they occur as actor messages.

The non-exhaustive list of adequate solutions to the “blocking problem”

Page 18: Jvm operation casual talks

ノンブロッキングアーキテクチャは I/Oを伴う処理は非常にセンシティブ

まとめ

Akkaにはログを非同期で出力できるユーティリティが用意されているのでそれを使おう

おかしいなと思ったらまずは Thread dump

Page 19: Jvm operation casual talks

おわり