Jvm tuning in a rush! - Lviv JUG

Strojenie Wirtualnej Maszyny Java

JVM Tuning in a rush

Tomasz Borek, Symentis

Infected in childhood From me parents

Amstrad, ElWro Junior

Games! Doh!

Mem pages in DOS anyone?

In IT ETL, crawlers, web-app, archi

IAAS, SAAS, own servers

Java 4 7, GNU/Linux, SQLs

Ardent activist

Tomasz Borek

Can be found in the net! :P

https://lafkblogs.wordpress.com/https://twitter.com/lafk_plhttps://pl.linkedin.com/in/tjborek

GeeCON, ChamberConf, Confitura, Lambda Days, LambdaCon, Java Developer Days, JavaDay Lviv, JavaDay Kiev

Tomek in IT groups

.comhttp://java.plKrakwPoznaPragaSopothttp://geecon.org

Prod hits rock bottom

Everybody is kung-fu fighting

Must be quick like lightning

So! Being a dev, you need:

To know your traffic and your app and your infra



To know recent changes




To know your limits (sometimes comes from point 1)





To have access to monitoring






To adjust pooling (both thread and connection)







To peruse logs and pull out anomalies








TO TUNE JVM








TO TUNE JVM

JVM? Meaning?

Sun Hotspot

IBM

JRockit

IceTea

OpenJDK

TCK decides

JVM? Meaning?

Sun Hotspot

IBM

JRockit

IceTea

OpenJDK

TCK decides

JVM? Meaning?

Sun Hotspot but which version?

IBM

JRockit

IceTea

OpenJDK

TCK decides

JVM? Meaning?

Sun Hotspot but which version? And architecture?

IBM

JRockit

IceTea

OpenJDK

TCK decides

So! JVM tuning?

To adjust GC, based on it's logs

So! JVM tuning?


To adjust (native) heap

So! JVM tuning?



Tweaking JVM starting flags

So! JVM tuning?




To use TOOLS

Today!

JVM tuning

Diagnosing performance problems

Tools

All in a rush

JVM tuning

Takeaway #1

JVM is a process

Being a process means

OS architecture enforces JVM architecture

Your memory is limited by OS and other processesHeap is one

C-heap is another!

IO-wise and thread-wise: Your threads and files opened are limited by ulimitFile descriptors!

OOM flavours?

OOM flavours

Out of HeapSpace

OOM flavours

Out of HeapSpace

PermGen error

OOM flavours

Out of HeapSpace

PermGen error

Unable to create native thread

API says OutOfMemoryError is

Thrown when the Java Virtual Machine cannot allocate an object because it is out of memory, and no more memory could be made available by the garbage collector. OutOfMemoryError objects may be constructed by the virtual machine as if suppression were disabled and/or the stack trace was not writable.


Thrown when the Java Virtual Machine cannot allocate an object because it is out of memory, and no more memory could be made available by the garbage collector. OutOfMemoryError objects may be constructed by the virtual machine as if suppression were disabled and/or the stack trace was not writable.


Due to failure in allocating an object after garbage collection.

Nothing about threads.

Nothing about PermGen.

How much memory JVM uses?

Say you start it with Xmx == Xms == 1GB



1GB?



1GB?

Around 1GB?



1GB?

Around 1GB?

If we talk about Java HEAP

Memory management

Entire RAM

32 bits: can address less than 4GB.

64 bits: can address theoretically 16 EB.Still much more!

There's much more to 32vs64 but that's for another time

Memory management

Entire RAMWhat's left of RAM C-HEAPJVM heap-Xmx

-Xms minimal heap size

-Xmx maximal heap size

When both should be set to same value?

Memory management

Entire RAMWhat's left of RAM C-HEAPJVM heapWhat's left C-HEAPJVM opsJVM heap-Xmx

PermGen

PermGen

Permanent GenerationEarly Java

for JDK classes

and Strings

Back then good idea

Now removedJDK 7 Strings no longer in PermGen

JDK 8 no PermGen at all

PermGen

Permanent GenerationEarly Java

for JDK classes

and Strings

Back then good idea

Now removedJDK 7 Strings no longer in PermGen

JDK 8 no PermGen at all

Quick fix?Increase PermGen size.

Memory management

What's left C-HEAPJVM opsJVM heap-Xmx

PermGen

GC

When it runs?

GC

When it runs?

Minor collection?

GC

When it runs?

Minor collection?

Major collection?

GC

When it runs?

Minor collection?

Major collection?

How can a dev tune GC?

Generational GC

Tenured where long-lived are promotedEden infants that die quickly

Studies showed:most objects die young

Some live really long

Ergo: short- and long-lived division

Minor collection: Eden

Major collection: whole heap

Generational GC - mechanics

In fact Eden has also 2 Survivor spacesTo handle locality

Helps compress after freeing space

First promotion to survivor, then to tenured

Flags tell:How many GCs object must survive to be promoted

How large Eden / Tenured / Survivors spaces are

What is logged (how details GC logs are)

Many, MANY more

Memory management trade-offs

Large heap large full GC small native and C-heap?

Smaller heap minor GC may be enough

Make sure your objects die young and only really long lived reach tenured

JVM tuning




To use TOOLS later

Takeaway #2

Log GC

Takeaway #3

GC tuning is a trade-off

Diagnosing performance problems

The Box

Heinz Kabutz, Kirk Pepperdine

Top - bottom

The Box

TRAFFIC: ?

TRAFFIC

The Box

TRAFFIC: how is app used?

TRAFFIC: people, automatic

The Box


CODE: ?

TRAFFIC: people, automaticApplication CODE

The Box


CODE: threads, algos, data structures...

TRAFFIC: people, automaticCODE: threads, data structs, algo

The Box



JVM: ?

TRAFFIC: people, automaticCODE: threads, data structs, algoJVM

The Box



JVM: ?

TRAFFIC: people, automaticCODE: threads, data structs, algoJVM: flags, GC

The Box



JVM: starting flags, GC

OS: ?

TRAFFIC: people, automaticCODE: threads, data structs, algoJVM: flags, GCOS

The Box




OS: ulimit, FS, config, archi, other procs

TRAFFIC: people, automaticCODE: threads, data structs, algoJVM: flags, GCOS: FS, config, limits

The Box





VIRT: ?

TRAFFIC: people, automaticCODE: threads, data structs, algoJVM: flags, GCOS: FS, config, limitsVIRT

The Box





VIRT: hell depends!

TRAFFIC: people, automaticCODE: threads, data structs, algoJVM: flags, GCOS: FS, config, limitsVIRT

The Box





VIRT: hell depends!

HARDWARE: ?

TRAFFIC: people, automaticCODE: threads, data structs, algoJVM: flags, GCOS: FS, config, limitsVIRTHARDWARE

The Box





VIRT: hell depends!

HARDWARE: 32 vs 64, RAM, BIOS, drivers

TRAFFIC: people, automaticCODE: threads, data structs, algoJVM: flags, GCOS: FS, config, limitsVIRTHARDWARE

Brendan Gregg USE

Utilization, Saturation, Errors

Use how much in use is your resource

Saturation how many requests await (waiting queue)

Errors what errors are thrown by resource

http://www.brendangregg.com/usemethod.html

Create a checklist for each important resource, for finding out Utilization, Saturation and Errors and you'll know how to soon find out what is going on with resource

Takeaway #1

The Box

Takeaway #2

GNU/Linux perf? Brendan Gregg

TOOLS

Takeaway #1

GNU/Linux surely has a tool for that

How to find your java process?

ps | grep java


ps | grep java

pgrep java


ps | grep java

pgrep java

jps

How to change flags on-the-fly?

jinfo

How to dump threads or heap?

kill -3


kill -3

jstack


kill -3

jstack

jhat (heap)

jvisualvm

Deserves it's own slide

Profiler, sampler

Monitor (heap, threads, etc.)

Calls GC, does dumps

SUMMARIZING

For being rushed prepare in advanceMonitoring, logs, ceilings, etc.

Log GC

JVM is a process all process limits in your OS apply

The Box

Brendan Gregg

GNU/Linux tools unparalleled

?

Kliknij, aby edytowa styl

Kliknij, aby edytowa style wzorca tekstuDrugi poziomTrzeci poziomCzwarty poziomPity poziom

04/08/15


Kliknij, aby edytowa styl wzorca podtytuu

04/08/15


Kliknij, aby edytowa style wzorca tekstu
Drugi poziom
Trzeci poziom
Czwarty poziom
Pity poziom

04/08/15



04/08/15


Drugi poziom
Trzeci poziom
Czwarty poziom
Pity poziom

Drugi poziom
Trzeci poziom
Czwarty poziom
Pity poziom

04/08/15



Drugi poziom
Trzeci poziom
Czwarty poziom
Pity poziom


Drugi poziom
Trzeci poziom
Czwarty poziom
Pity poziom

04/08/15


04/08/15

04/08/15


Drugi poziom
Trzeci poziom
Czwarty poziom
Pity poziom


04/08/15



04/08/15



04/08/15



04/08/15

Kliknij, aby edytowa format tekstu konspektuDrugi poziom konspektuTrzeci poziom konspektuCzwarty poziom konspektuPity poziom konspektuSzsty poziom konspektuSidmy poziom konspektusmy poziom konspektuDziewity poziom konspektu

@LAFK_pl

Technology

Jvm tuning in a rush! - Lviv JUG