JMH

AgendaBackground

Types of Benchmarking

Factors in benchmarking

Why are hand-written Benchmarks bad

Hands On

JMH Modes, Time Unit and Benchmark State

The Why of JMH tool

The How part of JMH

Benchmark Mode , Time unit and state

What’s next

BackgroundWhen software developers are concerned with performance of their system.They may resort to these options

Performance Testing to determine the performance of an already built system. MSDN provides a very thorough guide on the subject.

Profiling to analyze and investigate bottlenecks when a system is running.

Benchmarking to compare the relative performance of systems.

Analysis to determine the algorithmic complexity (Big-O notation).

Types of BenchmarkingThis leads to two commonly known types of benchmarks:

Macrobenchmarks are used to test entire system configurations. Macro benchmarks are done between different platforms to compare the efficiency between them.

Microbenchmarks are used to compare different implementations in an isolated context, for example a single component. Micro benchmarks are done within the same platform for a small snippet of code.

MicroBenchmarkingMicro benchmarks are generally done for two reasons.

To compare different approaches of code which implements the same logic and choose the best one to use.

To identify any bottlenecks in a suspected area of code during performance optimization.

So benchmarking is used for comparisons.Benchmark is the process of recording the performance of a system

Factors in benchmarkingBenchmark candidate: What piece of software do we benchmark?

Comparison against a baseline:determined by customer requirements or you might be just looking for the best relative performance in a specific scenario among a set of benchmark candidates.

Metrics: Which metrics do we use to determine performance? Like Throughput or Average time

Benchmarking scenario: Do we consider single-threaded or multi-threaded performance? How does a data structure behave when accessed concurrently by multiple writers?

Benchmarking duration

Why are hand-written Benchmarks badBecause you need to take into account these factors

JVM consists of three main components that work together: the runtime including the interpreter, the garbage collector and the JIT-compiler. Due to these components, we neither know in advance which machine code will be executed nor how it behaves exactly at runtime

Oracle's HotSpot JVM applies a vast amount of optimizations on Java code(more than 70 optimization technique)Some compiler optimizations like dead code elimination, loop unrolling, lock coalescing and in-lining. You might be benchmarking a different code than what you are thinking.

Why are hand-written Benchmarks bad

Each method is executed in interpreted mode at first. The Java interpreter requests that it should be JIT-compiled. Consequently, we have to run the benchmarked code often enough before the actual measurement starts to ensure that all benchmarked code has been JIT-compiled beforehand.. You should not see any JIT-compiler activity after the warmup phase.

Benchmark code falls victim to dead code elimination: In certain circumstances the JIT-compiler may be able to detect that the benchmark does not do anything and eliminates large parts or even the whole benchmark code.

False sharing: In multithreaded microbenchmarks, false sharing can severely affect measured performance. See false sharing

Why are hand-written Benchmarks badReliance on a specific environment: The JVM version, the OS and the

hardware could be different in a microbenchmark and an application. whether its is single core or multi-core or hyper threaded and its impact on your program to benchmark.

When running on the same environment, we need to remember to switch off all other programs. Machine should be silent. Background processes can compete for resources and cause delay.

Warm up phase in BenchmarkingBefore recording the numbers, do multiple runs of the code snippet to

warm up the environment. This is to initialize the environment. Java JIT takes time to analyze and optimize the code on initial runs. We should give enough number of iterations for it to stabilize otherwise we will end up adding the JIT overheads to the performance.

Similarly we may not get the caching benefits that happens at different levels.

Creating your first benchmark mvn archetype:generate \ -DinteractiveMode=false \ -DarchetypeGroupId=org.openjdk.jmh \ -DarchetypeArtifactId=jmh-java-benchmark-archetype \ -DgroupId=org.sample \ -DartifactId=test \ -Dversion=1.0

If you want to benchmark an alternative JVM language, use another archetype artifact ID from the list of existing ones,

Creating your first benchmark Building the benchmarks. After the project is generated, you can build it with the following Maven command:

$ cd test/$ mvn clean install

Running the benchmarks: $ java -jar target/benchmarks.jar

Archetypes for kotlin, groovy, scala and java are provided.

Understanding JMH code We have already completed the first step by annotating a method

with @Benchmark.

JMH implements multiple annotation processors that generate the final microbenchmark class. This generated class contains setup and measurement code as well as code that's required to minimize unwanted optimizations of the JIT compiler in the microbenchmark.

JMH contains a Runner class somewhat similar to JUnit so it is possible to run embedded microbenchmarks using the JMH Java API.

Understanding JMH

You can see that JMH creates multiple JVM forks. For each for fork, it runs n warmup iterations (shown in blue in the picture below), which do not get measured and are just needed to reach steady state before m iterations are run (shown in red in the picture below).

BenchMark Modes

Throughput: Rate at which the processing is done. @BenchmarkMode({Mode.Throughput}) calculates the operations per second. The timebound can be configured.

Average Time: Measures the average execution time. @BenchmarkMode({Mode.AverageTime}) calculates seconds by operations. The timebound can be configured. Its the reciprocal of throughput.

Benchmark Time Unit

JMH enables you to specify what time units you want the benchmark results printed in. The time unit will be used for all benchmark modes your benchmark is executed in.

You specify the benchmark time unit using the JMH annotation @OutputTimeUnit. The @OutputTimeUnit annotation takes a java.util.concurrent.TimeUnit as parameter to specify the actual time unit to use.

Benchmark StateSometimes you way want to initialize some variables that your benchmark code needs, but

which you do not want to be part of the code your benchmark measures. Such variables are called "state" variables. State variables are declared in special state classes, and an instance of that state class can then be provided as parameter to the benchmark method. @State annotation signals to JMH that this is a state class.

A state object can be reused across multiple calls to your benchmark method. JMH provides different "scopes" that the state object can be reused in. There state scope is specified in the parameter of the @State annotation.

Benchmark StateState Scopes

A state object can be reused across multiple calls to your benchmark method. JMH provides different "scopes" that the state object can be reused in. Their state scope is specified in the parameter of the @State annotation.The Scope class contains the following scope constants:

Thread - Each thread running the benchmark will create its own instance of the state object.

Benchmark-All threads running the benchmark share the same state object.

Benchmark State class RequirmentsA JMH state class must obey the following rules:

The class must be declared public

If the class is a nested class, it must be declared static (e.g. public static class ...)

The class must have a public no-arg constructor (no parameters to the constructor).

When these rules are obeyed you can annotate the class with the @State annotation to make JMH recognize it as a state class.

http://openjdk.java.net/projects/code-tools/jmh/

http://daniel.mitterdorfer.name/articles/2014/benchmarking-hello-jmh/

http://tutorials.jenkov.com/java-performance/jmh.html

http://javapapers.com/java/java-micro-benchmark-with-jmh/

https://github.com/nilskp/jmh-charts

https://github.com/melix/jmh-gradle-plugin

Thanks

Github : https://github.com/ackhare/JMHDemoForSession

Presented By :- chetan khare