Openfire 3.6.4 memory leak with Empathy

Most of the people that are responding to this thread are reporting that Openfire runs out of memory (causing OutOfMemoryExceptions) after at least one user started to use the Empathy client. I’ve created a new bug report for the Empathy issue specificly: OF-82

So far, I’ve not been able to identify where things go wrong. We need your help!

I would like to receive a couple of thread dumps (or possibly, a memory dump) of an instance of Openfire that is about to run out of memory. If you can provide these dumps, please contact me (contact details can be found in my profile).

Most likely, the memory leak is linked to specific functionality. Are there any clues as to what functionality causes this problem?

I’ll set up an Openfire instance in a Solaris zone on our private network specifically to test this. Keen to help if I can.

Cheers,

Dave

Additionally, a list of all plugins that your system is running (if you didn’t provide this list yet) could also be helpful.

I’m noting now that at a number of users that are experiencing this problem are using the monitoring plugin. This could be coincidence, of course, but lets check, to be sure. Can you guys reproduce the problem without the monitoring plugin? Can you stop the problem by unloading the monitoring plugin (keep an eye on java-monitors memory usage graphs after you do this!)

Similar checks can be used to eliminate other plugins too.

Right, I have the following set up on OpenSolaris snv_127

Server Properties
Version: Openfire 3.6.4

Environment
Java Version: 1.6.0_15 Sun Microsystems Inc. – Java HotSpot™ Server VM
Appserver: jetty-6.1.x
OS / Hardware: SunOS / x86
Java Memory 36.20 MB of 494.69 MB (7.3%) used

Using the embedded DB and local accounts only.

I’ve disabled anonymous login, inbound account registration, and the server-to-server service. All other settings are at defaults.

The only plug-in present is:

Search Provides support for Jabber Search (XEP-0055) 1.4.3

Next step, log in using Empathy on an Ubuntu 9.10 Virtualbox VM to see what happens…

Hi all,

I think I may have started some of this - I’m on holiday at the moment so this may be the last post for a while but just to sum up some of our findings when running quite an intensive load test on Openfire:

  • We needed to add the stalled session property that Guus has mentioned. When our clients dropped off for various reasons (out of memory exceptions on clients, network errors etc) Openfire did not deal with it that efficiently.

  • We set our heap size wrong on the Openfire server itself, we were under the impression it had more memory than it did so set our heap size too high.

  • The settings for the logging of room conversations by default does not really lend itself to heavy MUC usage (but fine for IM). We were generating pretty high traffic (2000 users, something like 3 chats per second). If you look at the source code I believe the default is to log in batches of 50. I can’t remember the interval off the top of my head but basically our log of messages stored in memory was way too high. We’ve now upped the batch size and shortened the interval size and it’s behaving much better, logging 250 messages per minute, although as it’s not a real SQL batch statement (it’s individual inserts) I would imagine the database may be getting a bit of a hammering.

Anyway, I believe our Openfire is up for the time being - thanks for all your help all : )

Hi Guss,

Everything appears fine on the nascent test system I set up (above)…so out of curiosity I logged in to our production server using Empathy 2.28.1.1 from an Ubuntu 9.10 virtual machine. And bingo - it’s all on. Empathy indeed appears to be the kiss of death for OpenFire.

I can confirm that what I’m seeing is without the monitoring plugin installed.

The production system details:

Version: Openfire 3.6.4

Environment
Java Version: 1.6.0_07 Sun Microsystems Inc. – Java HotSpot™ Server VM
Appserver: jetty-6.1.x
OS / Hardware: Linux / i386
Java Memory 171.62 MB of 253.19 MB (67.8%) used

Other differences from the above system I posted:

  • uses an external MySQL DB

  • users authenticate using our coporate AD installation

I haven’t got JConsole hooked up so am monitoring by refreshing the OF admin console page

Over the course of an hour I have observed JVM memory usage creep from 20 percent with half a dozen logged on users to at the time of writing 80 percent consumed.

All other users are on Spark 2.5.8 or Pidgin. I am the only user with Empathy.

The only plug-ins installed on this system are:

Red5 v0.1.11

Search v1.4.3

We are not using Red5 Sparkweb functionality - i.e we have it set on a trial basis for users if they wish to tinker, but as yet no-one is actually using it.

Another observation - if you really want to exacerbate the behaviour, quit Empathy and fire it back up in rapid sucession…

I’m not going to be able to get you thread dumps with this particular setup (sorry), but will return to my test system tomorrow and try and get some concrete data for you.

Cheers!
Dave

EDIT: JVM heap size just hit 90 percent of capacity…so I’m gonna do a quick bounce of the server before anyone notices…

Message was edited by: davenz

Hi Guus,

Matter of fact I did get a thread dump on our production system, easy when you know how I guess

I’ve replied to Gato in this thread:

And I’m attaching the dump as well.

In this case I was able to expend JVM memory completely in about five (5) minutes from bringing the OF server up - simply by repeatedly exiting and launching Empathy rapidly. This was with one user (me), with the remaining three or so users (at this late time of the day hehe) using Spark or Pidgin. If I am reading this right then I can easily see how in large deployments several Empathy users would create headaches.

JVM memory status: 252.80 MB of 253.19 MB (99.8%) used

ps -ef | grep -i java

daemon 30532 1 14 19:05 pts/0 00:01:33 /usr/lib/jvm/java-1.6.0-sun-1.6.0.u7/jre/bin/java -server -Xms128m -Xmx256m -DopenfireHome=/opt/openfire -Dopenfire.lib.dir=/opt/openfire/lib -classpath /opt/openfire/lib/startup.jar -jar /opt/openfire/lib/startup.jar

kill -3 30532

cd /opt/openfire/logs/

more nohup.out

Output in the attached.

Lemme know if this helps at all
nohup-out-20091117.txt.zip (5706 Bytes)

Same problem here, but we are not using empathy. Our server only allow connections from Spark and Pidgin clients.

We don’t use Sun JDK, but OpenJDK instead

Java Version:
1.6.0_0 Sun Microsystems Inc. – OpenJDK Server VM
Appserver:
jetty-6.1.x
Host Name:
openfire-im
OS / Hardware:
Linux / i386
Locale / Timezone:
en / Central European Time (1 GMT)

These are our enabled plugins:

Client Control 1.0.3

Email Listener 1.0.0

Kraken IM 1.1.2 (with Yahoo, MSN and Gtalk enabled)

Monitoring Service 1.1.1

Registration 1.4.1

Search 1.4.3

User Import/Export 2.2.0

Out of memory happens randomly. It can take up to 15 days… or 15 hours.

Not necessarily the same problem then - there seems to be a growing body of evidence implicating Empathy in this particular issue, and in my case it appears I have a solid test case I can reproduce this consistently on.

I agree with Dave here. Given the amount of noise surrounding the Empathy client, something must be up there. You are probably not running into the same problem, but into something different with similar effects.

I think there are at least 3-4 different issues discussed on this thread and this gets very confusing, for Guus too, i think. Would be better to discuss Empathy issue on Empathy thread. For random out of memory issues a separate threads should be started too.

Yes, it could be empathy, of course.

But I have sent you my own problem in order to find a possible pattern not related with empathy.

It probably will be empathy and our problem is another one, but perhaps is a plugin we are all using. I’m only trying to apply a “Dr House differential diagnostic”. :wink:

I’ve also found some exceptions in log regarding email listener and some others related to Gtalk personal groups.

I’ve changed the thread title to more acurately reflect the issue being discussed here.

I too am happy to find this thread.

We have a very modest Openfire installation, just 46 users, which ran untended and uninterupted for hundreds of days until November 5th, when we had the first “out of Java memory” crash. We took the enforced downtime as an opportunity to upgrade to 3.6.4.

It too crashed seven days later - out of Java resources…

Today (five days later) Java memory usage was at nearly 90% (of 1GB) so we restarted Openfire to preempt another crash.

Most of our clients are Pidgin (on Windows & Ubuntu), plus a smattering of OS X iChat. A couple of the Ubuntu users switched to Empathy on release of Ubuntu 9.10 (29 October). This does seem to have coincided the onset of our problems.

We have requested Empathy users to switch back to Pidgin and are monitoring closely.

Regards,

I am running Openfire 3.6.2 on Ubuntu Server 8.10.

Java Version:
1.6.0_0 Sun Microsystems Inc. – OpenJDK Server VM
Appserver:
jetty-6.1.x

I started having this same exact issue after I setup a laptop with Ubuntu Desktop 9.10 with the Empathy IM client. Once I saw this post I turned off the Ubuntu laptop and rebooted the Ubuntu Server. The problem has not come back since.

Could this be related? http://www.igniterealtime.org/community/thread/40410?tstart=0

I have noticed those connections too. Empathy queries possible proxy servers, which is why you see a lot of server-to-server connections being created. Although not very nice, it should be of no concequence. I’m running some quick tests to make sure.

This bug has been fixed the day before yesterday by the telepathy-gabble developers. An update should roll out soon.

I think I’ve uncovered the cause of the problem. It appears to be Openfire’s implementation of XEP-0163 “Personal Eventing Protocol.”

As a workaround, you can disable PEP by adding this Openfire System Property (you can add/modify properties through the Openfire Admin console):

The property xmpp.pep.enabled should be set to: false

3 Likes