Archive | Performance Analysis RSS feed for this section

Thread/Heap dumping using MBean Toolkit

8 Sep

Introduction

Thread and heap dumps can be used to analyse JVM concurrency and performance issues

Thread dump:
A file dump that shows all live threads currently living in the JVM

Heap dump:
A file dump that physically contains all objects on the JVM heap (usually large files as it is a snapshot of the heap itself)

Taking Heap/Thread dumps remotely using MBean Toolkit

The jar file at the github repository https://github.com/techshare1337/MBeanToolkit contains a jar file which is an app that can be launched as a normal executable. This app will allow you to take heap/thread dumps from your local PC by connecting to a JVM on another machine. In order to connect to a JVM on another machine the following flags need to be set on that JVM which can be done by following the guide below ‘Extracting the heap dump from the server’. The source code for this project can be checked out at https://github.com/techshare1337/MBeanToolkit.

-Dcom.sun.management.jmxremote # allow remote jmx connections
-Dcom.sun.management.jmxremote.port=8802 # port to connect on (doesn't have to be 8802)
-Dcom.sun.management.jmxremote.ssl=false # disable ssl authentication
-Dcom.sun.management.jmxremote.authenticate=false # set authenticate to false
-Djava.rmi.server.hostname=volrd10.int.corp.sun # set hostname to hostname or ip of machine that JVM is hosted on
  • Connect to JVM using [hostname]:[port] e.g. ‘voldrd10:8802’ from example above
  • To take a heap dump
    • Give the heap dump a filename and click heapdump. Heap dump should be named with filename + timestamp details and located in JVM directory on the remote server depending on JVM flags for heap dump output
  • To take a thread dump
    • Click thread dump. Application should collect dump info returned and output to file in same directory as the toolkit

Using the scheduler:
The scheduler can be used to take dumps at a set time, on a set repeation period and for a set duration. This is useful for diagnosing issues that can occur from the build up of something overtime.
To do this, check the scheduler checkbox and enter in the start delay (time to wait for a dump to be taken), repeat period (time to wait before taking another dump), duration (period to keep taking dumps for.
E.g. start delay=3600, repeat every=60, duration of=1800
Will wait for 1 hour to take a dump and repeat taking a dump every minute after this for half an hour.
Finally click the dump button to start the scheduler.

Extracting the Heap Dump from the server

1. On the machine with said heap dump, login using PuTTY and use the command 

sudo su - [superuser]

this will then allow you to modify permissions which you will need to apply to modify and extract files. The following command will give you full permissions on the ‘wsdev’ group which you should be added to from the previous step and applies to the directory where heap dumps and settings for them are set.

chmod -Rf 774 /apps/tc6

Setting the flags for remote JMX connections:

1. Using WinSCP, go to the directory /apps/tc6/[nameOfAppForHeapDump]/script

2. Modify flags to be as shown in the tutorial above in the file which contains flags or the config. The one I have noticed that always needs to be changed (from true to false) is

-Dcom.sun.management.jmxremote.authenticate=false

Also, if you want a heap dump to be invoked on out of memory errors add the following flag to the file

-XX:+HeapDumpOnOutOfMemoryError"

so would be 

CATALINA_OPTS="$CATALINA_OPTS -XX:+HeapDumpOnOutOfMemoryError"

You may get an error that reads something along the lines of ‘timestamp cannot be modified’ after saving, just skip this message. Alternatively go to WinSCP preferences and check the box for ‘Ignore permission errors’ to suppress this warning from now on as shown below.

You must now stop and start the jvm again to apply the changes.

Extracting heap dumps

Drag and drop the heap dump file to your local pc which should be able to be done if you’ve set the permissions correctly

Advertisements

Garbage Collection Analysis with HPJmeter

2 Sep

Introduction

HPJmeter is a tool that can be used to analyse garbage collection logs. HPJmeter is a useful tool to assist in identifying JVM issues.

How to use

  1. Download the tool from here to your machine
  2. Execute the jar file to open the application. Retrieve the garbage collection log (gc.log)  and save it somewhere to be opened by HPjmeter
  3. Go to Open File and select the gc file. You should get something that looks like this:

hpjmeter1

The summary view shows the overall statistics to do with GC and the heap. The screenshot below is the other main view ‘Heap Usage After GC’:hpjmeter2

This shows heap usage after GC and the garbage collection details. Pink indicates the JVM going into full GC.

General Trends

Typical

The below is an example of a typical/expected graph where memory increases but then reaches a point where heap usage drops again and this continues forming a ‘zig-zag’ or ‘sawtooth’ type patternhpjmeter3

Memory leak indicative

The below is an example indicative of memory issues where garbage collection goes and stays into full GC never able to release objects to decrease memory usage. When this happens there should be out of memory errors visible in the logs themselves.hpjmeter4