Openfire 3.6.4 memory leak with Empathy

Hi Guus,

Matter of fact I did get a thread dump on our production system, easy when you know how I guess

I’ve replied to Gato in this thread:

And I’m attaching the dump as well.

In this case I was able to expend JVM memory completely in about five (5) minutes from bringing the OF server up - simply by repeatedly exiting and launching Empathy rapidly. This was with one user (me), with the remaining three or so users (at this late time of the day hehe) using Spark or Pidgin. If I am reading this right then I can easily see how in large deployments several Empathy users would create headaches.

JVM memory status: 252.80 MB of 253.19 MB (99.8%) used

ps -ef | grep -i java

daemon 30532 1 14 19:05 pts/0 00:01:33 /usr/lib/jvm/java-1.6.0-sun-1.6.0.u7/jre/bin/java -server -Xms128m -Xmx256m -DopenfireHome=/opt/openfire -Dopenfire.lib.dir=/opt/openfire/lib -classpath /opt/openfire/lib/startup.jar -jar /opt/openfire/lib/startup.jar

kill -3 30532

cd /opt/openfire/logs/

more nohup.out

Output in the attached.

Lemme know if this helps at all
nohup-out-20091117.txt.zip (5706 Bytes)

Same problem here, but we are not using empathy. Our server only allow connections from Spark and Pidgin clients.

We don’t use Sun JDK, but OpenJDK instead

Java Version:
1.6.0_0 Sun Microsystems Inc. – OpenJDK Server VM
Appserver:
jetty-6.1.x
Host Name:
openfire-im
OS / Hardware:
Linux / i386
Locale / Timezone:
en / Central European Time (1 GMT)

These are our enabled plugins:

Client Control 1.0.3

Email Listener 1.0.0

Kraken IM 1.1.2 (with Yahoo, MSN and Gtalk enabled)

Monitoring Service 1.1.1

Registration 1.4.1

Search 1.4.3

User Import/Export 2.2.0

Out of memory happens randomly. It can take up to 15 days… or 15 hours.

Not necessarily the same problem then - there seems to be a growing body of evidence implicating Empathy in this particular issue, and in my case it appears I have a solid test case I can reproduce this consistently on.

I agree with Dave here. Given the amount of noise surrounding the Empathy client, something must be up there. You are probably not running into the same problem, but into something different with similar effects.

I think there are at least 3-4 different issues discussed on this thread and this gets very confusing, for Guus too, i think. Would be better to discuss Empathy issue on Empathy thread. For random out of memory issues a separate threads should be started too.

Yes, it could be empathy, of course.

But I have sent you my own problem in order to find a possible pattern not related with empathy.

It probably will be empathy and our problem is another one, but perhaps is a plugin we are all using. I’m only trying to apply a “Dr House differential diagnostic”. :wink:

I’ve also found some exceptions in log regarding email listener and some others related to Gtalk personal groups.

I’ve changed the thread title to more acurately reflect the issue being discussed here.

I too am happy to find this thread.

We have a very modest Openfire installation, just 46 users, which ran untended and uninterupted for hundreds of days until November 5th, when we had the first “out of Java memory” crash. We took the enforced downtime as an opportunity to upgrade to 3.6.4.

It too crashed seven days later - out of Java resources…

Today (five days later) Java memory usage was at nearly 90% (of 1GB) so we restarted Openfire to preempt another crash.

Most of our clients are Pidgin (on Windows & Ubuntu), plus a smattering of OS X iChat. A couple of the Ubuntu users switched to Empathy on release of Ubuntu 9.10 (29 October). This does seem to have coincided the onset of our problems.

We have requested Empathy users to switch back to Pidgin and are monitoring closely.

Regards,

I am running Openfire 3.6.2 on Ubuntu Server 8.10.

Java Version:
1.6.0_0 Sun Microsystems Inc. – OpenJDK Server VM
Appserver:
jetty-6.1.x

I started having this same exact issue after I setup a laptop with Ubuntu Desktop 9.10 with the Empathy IM client. Once I saw this post I turned off the Ubuntu laptop and rebooted the Ubuntu Server. The problem has not come back since.

Could this be related? http://www.igniterealtime.org/community/thread/40410?tstart=0

I have noticed those connections too. Empathy queries possible proxy servers, which is why you see a lot of server-to-server connections being created. Although not very nice, it should be of no concequence. I’m running some quick tests to make sure.

This bug has been fixed the day before yesterday by the telepathy-gabble developers. An update should roll out soon.

I think I’ve uncovered the cause of the problem. It appears to be Openfire’s implementation of XEP-0163 “Personal Eventing Protocol.”

As a workaround, you can disable PEP by adding this Openfire System Property (you can add/modify properties through the Openfire Admin console):

The property xmpp.pep.enabled should be set to: false

3 Likes

Good news! I have added this setting to our server, I will monitor the memory use now. Thanks a lot for your work!

What I forgot to mention: you’ll most likely have to restart the server for the setting to take effect.

I’ve been experiencing memory utilization problems as well: http://www.igniterealtime.org/community/message/197613#197613

I’ve seen usage of Empathy go up and instead of running out of memory once a month, it’s almost daily now. I can make my heap dumps available to the developers if they are interested in profiling it.

The fix for http://bugs.freedesktop.org/show_bug.cgi?id=21151 (Should only query SOCK5 proxies when needed) has made it into Ubuntu Jaunty-Proposed repositories now.

It’ll stop the random servers being called from Empathy, although doesn’t seem related to Guus’s PEP theory.

To enable Proposed updates in Ubuntu, goto “Software Sources” in Administration, goto the updates tab, and selected “Proposed”. Then reload your sources. telepathy-gabble should then appear in update-manager.

Couple Empathy clients connected, no problem so far, the memory use doesn’t show any suddent increase like before. I guess u did it Guus, great job! I will try to add some more Empathy clients to see if it’s stable.

Hi Guus,

Are there plans for this to be fixed in the next Openfire release (and if so is there a timescale on this)? Or will we have to stick with disabling the PEP?

Thanks for your help.

Hi Dave,

Sorry that it has taken so long to get back to you.

I’ve been trying to identify the cause of the problem, but so far, have been unable to do so. I’ve asked other developers to have a look too, but none of them have been successful either. Sadly, my attempts are severely hindered by lack of time - I’m doing this in my spare time, which is limited.

I’m not comfortable at all releasing the next version of Openfire with this bug in it, but there’s going to be a point in the near future where I feel we should be pragmatic, and move forward. The release has been postponed for to long now.

In the meantime, the lead developer of the Empathy client has confirmed that updates have been released that should dramatically reduce the impact of the bug. Thanks, Sjoerd! I haven’t tested the new client yet though.

Although I’m having trouble identifying the exact cause, I did manage to make some progress. While investigating, I’ve discovered a number of smaller and bigger issues in the PEP / pubsub routines of Openfire. There’s no obvious direct link between these issues and the memory leak that we’ve been discussing here, but the issues that I’m addressing now are likely candidates, in the sense that these kind of (concurrency-related) bugs are known to cause these kind of problems. I’m currently busy rewriting parts of the PEP routines. I’m hoping that my general improvements make this problem go away (or at least help me to identify the cause). This borders on the edges of educated guessing and wishful thinking, but hey, it’s the holiday season.

Regards,

Guus