Haben Sie Ihre Web Logic Umgebung im Griff Daniel Joray

2015 © Trivadis

BASEL BERN BRUGG GENF LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN

KOPENHAGEN

2015 © Trivadis

Haben Sie Ihre

WebLogic Umgebung im Griff?

Daniel Joray

2015 © Trivadis

Trivadis ist führend bei der IT-Beratung, der Systemintegration,

dem Solution-Engineering und der Erbringung von IT-Services

mit Fokussierung auf und Technologien

im D-A-CH-Raum.

Unsere Leistungen erbringen wir aus den strategischen Geschäftsfeldern:

Trivadis Services übernimmt den korrespondierenden Betrieb

Ihrer IT Systeme.

Unser Unternehmen

08.11.2015

Trivadis – das Unternehmen

B E T R I E B

2

2015 © Trivadis

Mit über 600 IT- und Fachexperten bei Ihnen vor Ort

3

12 Trivadis Niederlassungen mit

über 600 Mitarbeitenden

200 Service Level Agreements

Mehr als 4'000 Trainingsteilnehmer

Forschungs- und Entwicklungs-

budget: CHF 5.0 Mio. / EUR 4.0

Mio.

Finanziell unabhängig und

nachhaltig profitabel

Erfahrung aus mehr als 1'900

Projekten pro Jahr bei über 800

Kunden

Stand 12/2013

Hamburg

Düsseldorf

Frankfurt

FreiburgMünchen

Wien

Basel

ZürichBern

Lausanne

3

Stuttgart

08.11.2015


3

Brugg

2015 © Trivadis


Trivadis an der DOAG

Ebene 3 - gleich neben der Rolltreppe

Wir freuen uns auf Ihren Besuch.

Denn mit Trivadis gewinnen Sie immer.

2015 © Trivadis

AGENDA

1. About Monitoring

2. JVM

3. JDBC DataSources

4. JMS

5. Application

6. Logfiles

7. Tools

8. Summary

Haben Sie Ihre WebLogic Umgebung im Griff

2015 © Trivadis


About Monitoring

2015 © Trivadis

About Monitoring - a Definition


Monitoring is a proactive - not reactive - approach to supervise a

system

You have to ensure that…

All components of your applications are up and running

Your applications do not produce errors

Your application setup is optimal to get out the maximum performance

In a WebLogic Domain, there are hundreds of MBeans with thousands

of metrics attributes

It is not possible to monitor all those metrics

2015 © Trivadis

About Monitoring - Components

Main components in a WebLogic environment are:

JVM Java Virtual Machine

no JVM running, no J2EE application available

JDBC DataSource

no JDBC DataSource, no data

JMS Java Messaging System

no JMS, no message exchange

The Application(s)

Logfiles


2015 © Trivadis

About Monitoring - Components / Metrics


For each components of Weblogic you can access a lot of monitoring

metrics

2015 © Trivadis


But which are really

important for your

application?

2015 © Trivadis


JVM

2015 © Trivadis

JVM – General

A Java virtual machine (JVM) is a virtual machine capable of executing

Java bytecode, it is the code execution component of the Java software

platform

WebLogic can use HotSpot JVM from Oracle, JRockit from BEA, IBM

JVM and HP JVM

Not all JVMs are created equal

The Java virtual machine heap is the area of memory used by the JVM

for dynamic memory allocation, the heap is split up into “generations”

The memory management is not a task of WebLogic server, this is the

work of the Java Virtual Machine (JVM)

The Java Virtual Machine (JVM) you use to run your WebLogic Server–

based application is a key factor in server performance


2015 © Trivadis

JVM - Memory

Garbage collection (GC) is the technique a JVM uses to free memory

occupied by objects that are no longer being used by the application

The memory management configuration is set at JVM startup


$ java –Xms128 –Xmx256M -XX :NewRatio=3 -XX:MaxPermSize=128M

2015 © Trivadis

JVM - Memory Key Metrics

Percent use of the heap size

Percent of time in Garbage Collection

GC is a stop the world process

It is very important to verify the system is

not spending too much time in this state

Max/ Mean time for the garbage

collection


2015 © Trivadis

JVM- Threads

A thread is an independent path of

execution through program code.

A thread pool is a group of threads

initially created that waits for jobs and

executes them

The value of the ThreadPoolSize

attribute equals the number of

simultaneous operations that can be

performed by WebLogic Server.

You can degrade performance by

increasing this value unnecessarily.


2015 © Trivadis

JVM – Threads Key Metrics

Idle Thread Counts

This count does not include standby

threads and stuck threads. The count

indicates threads that are ready to pick up

new work when it arrives

Thread Pool Queue length

The number of pending requests in the

priority queue

Hogging Thread Counts

Hogging threads are candidates for stuck

threads.

Threads that “might” get stuck. These

threads will be declared “stuck” after

StuckThreadMaxTimeout seconds which

usually is 600secs.


2015 © Trivadis

JVM - Threads Key Metrics

Server Running Time

This is a great metric to catch servers that crash and get restarted by

Nodemanager


2015 © Trivadis

JVM - Threads Key Metrics

Keep in mind monitoring requires system resources, you need to

place the focus on the really important metrics

Not optimal monitoring configuration will impact the application

performance


2015 © Trivadis


JDBC DataSource

2015 © Trivadis

JDBC DataSource – General

The application connects to the database with

a JDBC DataSource

A JDBC DataSource contains a pool of

database connections which are ready-to-use

A JBCD DataSource can be configured with

different behaviours e.g.

initial-capacity

min-capacity

max-capacity

etc.


2015 © Trivadis

JDBC DataSource - Key Metrics

Number Available

allows you to validate that you have the correct

amount of resources available to service the

clients needs

Number Active Connections

allows you to validate that you have not two

many amount of resources available to service

the client’s needs

Number Unavailable Connections

The number of connections currently in use by

applications or being tested in this instance of

the data source


2015 © Trivadis


JMS

2015 © Trivadis

JMS – General

The Java Message Service (JMS) is a standard API for accessing

enterprise messaging systems. Specifically, JMS:

Enables Java applications sharing a messaging system to exchange

messages

Simplifies application development by providing a standard interface for

creating, sending, and receiving messages


2015 © Trivadis

JMS - Key Metrics

JMS Pending Message

A message present on a JMS destination can be

either current or pending. A current message is

available for immediate consumption by

consumers

Pending JMS messages means the messages

that were:

- Sent in a transaction but not committed

- Received but not acknowledged

- Received but not committed

- Subject to a redelivery delay

- Subject to a delivery time

if you see an unusual increase in the number of

pending messages, it does point to something

not working correctly in the application or the

WebLogic Server


2015 © Trivadis

JMS - Key Metrics

JMS Oldest Message Age

Normally it doesn’t matter how many messages are on the queue, it is a

good idea to pay close attention to how old the oldest message is

This is normally a key indication of issues, or shows that the system is

being affected by an excessive message dump on the queue – something

the system cannot keep up with the load

JMS Consumers

If no consumers are visible, then we

don’t process messages.


2015 © Trivadis


Application

2015 © Trivadis

Application – General

A WebLogic Server application consists of one of the following modules or

applications running on WebLogic Server:

Web application modules - HTML pages, servlets, JavaServer Pages,

and related files

Enterprise Java Beans (EJB) modules - entity beans, session beans,

and message-driven beans

Connector modules - resource adapters


2015 © Trivadis

Application - Key Metrics

Application health and deployment status

Validate all deployments are deployed to the correct servers and are in

an active state


2015 © Trivadis

Web Application - Key Metrics

Open Sessions Current

Specifies the number of sessions that are

currently active in the application

Open Session High

Refers to the highest number of sessions

that were open for an Application, this is

great data for capacity planning


2015 © Trivadis

EJB Application - Key Metrics

Bean Access Total

Total number of times an attempt was

made to get a bean instance from the free

pool

Waiter Current Count

Number of threads currently waiting for

an available bean instance from the pool


2015 © Trivadis


Logfiles

2015 © Trivadis

Logfiles – General

In a WebLogic Domain, most components have logfiles

Domain

Managed Server

Instance Log File stderr/stdout of the JVM server_name.out

Server Log File server_name.log

HTTP access log File access.log

JDBC subsystem

DataSource.log

JMS subsystem jmsServers\JMSServer_server\jms.messages.log


2015 © Trivadis

Logfiles – Key Metrics

The messages for all WebLogic Server subsystems contain a consistent

set of fields the most important are

Severity INFO, WARNING, ERROR, NOTICE, CRITICAL, ALERT, EMERGENCY

Subsystem

Message ID

Message Text


2015 © Trivadis


Tools

2015 © Trivadis

Tools – First Place

The Weblogic Console is probably the first place see the Weblogic

metrics

Almost all of the components has a monitoring tab


2015 © Trivadis

Tools – Low Cost Solutions

The jconsole is a JMX-compliant graphical tool for monitoring a Java

virtual machine

The jvisualvm is a tool that provides a visual interface for viewing

detailed information about Java applications while they are running on a

Java Virtual Machine (JVM), command allows you to query various

aspects of specific virtual machines

The Java Mission Control jmc and Java Flight Recorder together create

a complete tool chain to continuously collect low level and detailed

runtime information enabling after-the-fact incident analysis

DomainHealth is a J2EE application, which is deployed on a

AdminServer, it shows the most key metrics

GCViewer analyses logfile from the JVM garbage collection actions

WLST – The WebLogic Scripting Tool


2015 © Trivadis

Tools – Professional

Dynatrace / Appdynamics

Those applications manage the availability and performance of software

applications and the impact on user experience in the form of deep

transaction tracing, synthetic monitoring, real user monitoring, and network

monitoring.

EM12c Middleware Monitoring Plugin

The WebLogic Server Management Pack Enterprise Edition greatly improves

server as well as application performance by providing unique functionality to

automatically detect performance bottlenecks; quickly diagnose these

performance problems, and identify their root cause.


2015 © Trivadis

Tools – jvisualvm


Java VisualVM is an intuitive graphical user interface that provides

detailed information on a specified Java Virtual Machine

Other functionality, such as some that offered by the jconsole

command, can be added as optional plug-ins.

Java VisualVM combines several

monitoring, troubleshooting, and

profiling utilities into a single tool.

Most of the functionality offered by

the standalone tools jmap, jinfo,

jstat, and jstack were integrated

2015 © Trivadis

Tools – DomainHealth


DomainHealth (Paul Done)

https://github.com/ccristian/domainhealth/wiki/Introduction-to-the-Domain-Health-monitoring-tool

DomainHealth is an open source "zero-config" monitoring tool for

WebLogic. It collects important server metrics over time, archives these

into CSV files and provides a simple web interface for viewing graphs of

current and historical statistics.

Base on WLS Diagnostic

Framework (WLDF)

Show statistics for

- Core Server

- JDBC Data Source

- JMS Destination

- Work Managers

- Channels (Server Sockets).

2015 © Trivadis

Tools – GCViewer (1)


Enable logging for GCViewer

export JAVA_OPTIONS=" -Xloggc:jvm_${SERVER_NAME}.log

–verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps

-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M "

Java HotSpot(TM) 64-Bit Server VM (24.76-b04) for linux-amd64 JRE (1.7.0_76-b13), built on Dec 18 2014 16:58:39

by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)Memory: 4k page, physical 32751136k(12598616k free),

swap 8388604k(4898056k free)CommandLine flags: -XX:InitialHeapSize=805306368 -

XX:MaxHeapSize=1610612736 -XX:MaxPermSize=805306368 -XX:PermSize=268435456 -XX:+PrintGC -

XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedOops -

XX:+UseParallelGC

2015-08-13T10:35:57.602+0200: 4.014: [GC [PSYoungGen: 196608K->24008K(229376K)] 196608K-

>24080K(753664K), 0.0293670 secs] [Times: user=0.05 sys=0.00, real=0.04 secs]









2015 © Trivadis

Tools – GC Viewer (2)


Generated logs can be later analysed with GCViewer or with the IBM

Pattern Modeling and Analysis Tool for Java Garbage Collector

2015 © Trivadis

Tools – WLST


WLST(WebLogic Scripting Tool) is the best friend of the administrator#Definition for connecting to a server

def connectToServer():

connect(…, …, …)

#Definition to print a running servers heap details

def printHeapDetails(server_name):

domainRuntime()

cd('/ServerRuntimes/'+server_name+'/JVMRuntime/'+server_name)

hf = float(get('HeapFreeCurrent'))/1024/1024

hs = float(get('HeapSizeCurrent'))/1024/1024

print 'HeapFreeCurrent - ' + `hf` + 'MB'

print 'HeapSizeCurrent - ' + `hs` + 'MB'

#Definition to disconnect from a server

def disconnectFromServer():

disconnect()

exit()

#Calling connectToServer definition with no arguments

connectToServer()

#Calling printHeapDetails with arguments

printHeapDetails('wlsAdmin')

printHeapDetails('wls1')

#Calling disconnectFromServer definition with no arguments

disconnectFromServer()

2015 © Trivadis

Tools – EM12c Cloud Control


Full Integration -

2015 © Trivadis

Tools – Dynatrace


End-to-End Monitoring

Deep diving in the application code

2015 © Trivadis

Tools – Distributed Log Collectors


Logs are a critical part of any system, they give you insight into what a

system is doing as well what happened

When your system grows to multiple hosts, managing the logs and

accessing them can get complicated

Searching for a particular error across hundreds of log files on hundreds

of servers is difficult without good tools

A common approach to this problem is to setup a centralized logging

solution

Distributed Log Collectors

2015 © Trivadis

Tools – Log Analysis, Viewer and Search


Graylog is a fully integrated open source log management platform for

collecting, indexing, and analyzing both structured and unstructured data

from almost any source. Non Graylog-authored components Include

MongoDB for metadata and Elasticsearch for log file storage and text

search

2015 © Trivadis

Tools – Log Analysis, Viewer and Search


Splunk Light / Enterprise / Cloud makes it simple to collect, analyze and

act upon the untapped value of the big data generated by your

technology infrastructure, security systems and business applications

Collect and Index Data

Search and Investigate

Correlate and Analyze

Visualize and Report

Monitor and Alert

2015 © Trivadis

Summary

Each components from a WebLogic Domain should be monitored, but

not each metrics

Probably you need more than one tool

Not all monitored metrics should send alarm, but in problem case or if

you want plan the capacity of your system for the next months you

should have enough information to do your job


“This is as much an art as it is a science.” (John Mounjoy, WebLogic The Definitive Guide)

2015 © Trivadis

Fragen und Antworten...

2015 © Trivadis

BASEL BERN BRUGG GENF LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN

KOPENHAGEN

Daniel Joray

Principal Consultant

Tel. +41 58 459 50 26

[email protected]


Presentations & Public Speaking

Haben Sie Ihre Web Logic Umgebung im Griff Daniel Joray