Skip navigation
Currently Being Moderated

JVM Settings and Debugging

VERSION 16  Click to view document history
Created on: Aug 20, 2007 1:39 PM by LG - Last Modified:  Mar 28, 2009 6:13 AM by LG

Useful Links

Virtual Machines

Tools

Settings for Openfire

Openfire Installation Guide (Visit this page if you wonder how to set these values.)

Disable IPv6

As still some users do not use IPv6, it is save to disable it by adding as the third line in bin/openfire:

INSTALL4J_ADD_VM_PARAMS="-Djava.net.preferIPv4Stack=true"

Heap Settings

Add "-Xms32m -Xmx128m -Xss128k -Xoss128k -XX:ThreadStackSize=128" to INSTALL4J_ADD_VM_PARAMS to specify an initial heap of 32 MB, a maximum heap of 128 MB and a stack size of 128 kB for native and java threads. Do not append a k to the ThreadStackSize value.

One may also need to set "-XX:MaxPermSize=128m", but before doing this one should consult the gc.log file.

Monitor the JVM Memory Usage

Add "-XX:+PrintGCDetails -Xloggc:/tmp/gc.log" to INSTALL4J_ADD_VM_PARAMS to track every garbage collecion and allocation failure.

Heap dump on OutOfMemory error

Add "-XX:+HeapDumpOnOutOfMemoryError" to INSTALL4J_ADD_VM_PARAMS to create a heap dump every time an OutOfMemory error occurs. There should never occur one, but if it does the dump contains useful information to debug the issue.

Note: If you did not adjust the heap settings and try to connect 10000 users to Openfire then you will also get a heap dump which is not really interesting.

DNS Settings

If s2s connections to servers with dynamic IP addresses are possible one may want to change the DNS cache to a low value instead of forever. One can set the "networkaddress.cache.ttl=1800" value in

"path-to-JRE\lib\security\java.security" (Windows) / "path-to-JRE/lib/security/java.security" (Linux) or add "-Dsun.net.inetaddr.ttl=1800" as an additional INSTALL4J_ADD_VM_PARAMS parameter.

"1800" seconds are 30 minutes, this should be a fine value.

JM-898 should fix this issue.

UTF-8 Settings

Add "-Dfile.encoding=UTF-8" to the INSTALL4J_ADD_VM_PARAMS parameter. This fixes a few issues: JM-1351, JM-1190, JM-1101.

Debugging

To identify deadlocks or looping threads you need to create a stack trace.

Creating a stack trace

Use `jstack openfire-pid >/tmp/javacore.txt´ to write one into /tmp/javacore.txt.

On Linux `kill -3 openfire-pid´ will cause the JVM to write one to STDOUT, one will likely find in in nohup.out, stdout.log or somewhere else, depending on your start script.

One can also use StackTrace (graphical UI) to create one, the link is at the top of the page.

On Windows all methods may fail if the process runs as SYSTEM and your user does not have SYSTEM rights.

Thread identification

To identify looping threads you should also get one or more stack traces and try to identify the looping thread.

Linux native to java thread mapping

This short manual is for Linux and it may not work on your Linux installation, depending on your kernel and ps versions:

You may want to try to run `ps -T -p openfire-pid -o pid,tid,pri,time | grep -v '00:00:00'´ - this could (depending on your linux ps implementation) display the thread which is using a lot of CPU just like:

PID TID PRI TIME

8313 8313 23 00:00:01

8313 8314 21 00:13:23 <== high "time" value, look at the TID, I use 8314 as a sample value.

8313 8315 23 00:00:04

... very long list, run the command above or `ps -T -p openfire-pid -o pid,tid,pri,time | grep 8314´ again to make sure that this value increases.

 

Then you would have to look for the "tid" with the highest "time" value and convert it from dec to hex (8314=0x207A ; Linux: `printf "%X\n" 8314´; Windows: `calc´ with scientific view).

 

You should find somewhere in the javacore a line with ".. nid=0x207A .." - this should be the looping thread.

Windows native to java thread mapping

Get "pslist" from MS Technet, Process Utilities, PsList.

Run "pslist.exe -dmx Openfire >c:\temp\openfire.threads.txt" to create a list of threads within Openfire. Also the Openfire PID will be written to the file (you can alternatively look in the Task Manager for it).

Run "jstack openfire-pid >c:\temp\openfire.javacore.txt" to create a core file. This may fail depending on service permissions - for Spark it usually works fine.

Then you need to map the interesting "Tid" in `openfire.threads.txt´ to a "nid" in `openfire.javacore.txt´ by converting it to hex(Tid 252 ==> nid=0xFC).

More advanced debugging and profiling

Probably only useful for developers. Do not use these options unless you know what you are doing.

Tools

Remote debugging of Openfire

Start Openfire with these JVM options: "-Xdebug -agentlib:jdwp=transport=dt_socket,address=8000"

Details:http://java.sun.com/j2se/1.5.0/docs/guide/jpda/conninv.html

Remote profiling of Openfire with Profiler4J

Download the CVS source and build profiler4j-1.0-beta3. Add "-javaagent:/path-to/agent.jar=waitconn=false" to the JVM options. Then you need to start also the Profiler4J console to get profiling information.

Random disconnects on high load or during load tests

Use the loadstats plugin, either compile it with Openfire or download it here Load Stats Plugin

The wiki document which describes its output is as far as I can tell no longer available.

Comments (6)