Openfire 3.6.4 memory leak with Empathy

I too am happy to find this thread.

We have a very modest Openfire installation, just 46 users, which ran untended and uninterupted for hundreds of days until November 5th, when we had the first “out of Java memory” crash. We took the enforced downtime as an opportunity to upgrade to 3.6.4.

It too crashed seven days later - out of Java resources…

Today (five days later) Java memory usage was at nearly 90% (of 1GB) so we restarted Openfire to preempt another crash.

Most of our clients are Pidgin (on Windows & Ubuntu), plus a smattering of OS X iChat. A couple of the Ubuntu users switched to Empathy on release of Ubuntu 9.10 (29 October). This does seem to have coincided the onset of our problems.

We have requested Empathy users to switch back to Pidgin and are monitoring closely.

Regards,

I am running Openfire 3.6.2 on Ubuntu Server 8.10.

Java Version:
1.6.0_0 Sun Microsystems Inc. – OpenJDK Server VM
Appserver:
jetty-6.1.x

I started having this same exact issue after I setup a laptop with Ubuntu Desktop 9.10 with the Empathy IM client. Once I saw this post I turned off the Ubuntu laptop and rebooted the Ubuntu Server. The problem has not come back since.

Could this be related? http://www.igniterealtime.org/community/thread/40410?tstart=0

I have noticed those connections too. Empathy queries possible proxy servers, which is why you see a lot of server-to-server connections being created. Although not very nice, it should be of no concequence. I’m running some quick tests to make sure.

This bug has been fixed the day before yesterday by the telepathy-gabble developers. An update should roll out soon.

I think I’ve uncovered the cause of the problem. It appears to be Openfire’s implementation of XEP-0163 “Personal Eventing Protocol.”

As a workaround, you can disable PEP by adding this Openfire System Property (you can add/modify properties through the Openfire Admin console):

The property xmpp.pep.enabled should be set to: false

3 Likes

Good news! I have added this setting to our server, I will monitor the memory use now. Thanks a lot for your work!

What I forgot to mention: you’ll most likely have to restart the server for the setting to take effect.

I’ve been experiencing memory utilization problems as well: http://www.igniterealtime.org/community/message/197613#197613

I’ve seen usage of Empathy go up and instead of running out of memory once a month, it’s almost daily now. I can make my heap dumps available to the developers if they are interested in profiling it.

The fix for http://bugs.freedesktop.org/show_bug.cgi?id=21151 (Should only query SOCK5 proxies when needed) has made it into Ubuntu Jaunty-Proposed repositories now.

It’ll stop the random servers being called from Empathy, although doesn’t seem related to Guus’s PEP theory.

To enable Proposed updates in Ubuntu, goto “Software Sources” in Administration, goto the updates tab, and selected “Proposed”. Then reload your sources. telepathy-gabble should then appear in update-manager.

Couple Empathy clients connected, no problem so far, the memory use doesn’t show any suddent increase like before. I guess u did it Guus, great job! I will try to add some more Empathy clients to see if it’s stable.

Hi Guus,

Are there plans for this to be fixed in the next Openfire release (and if so is there a timescale on this)? Or will we have to stick with disabling the PEP?

Thanks for your help.

Hi Dave,

Sorry that it has taken so long to get back to you.

I’ve been trying to identify the cause of the problem, but so far, have been unable to do so. I’ve asked other developers to have a look too, but none of them have been successful either. Sadly, my attempts are severely hindered by lack of time - I’m doing this in my spare time, which is limited.

I’m not comfortable at all releasing the next version of Openfire with this bug in it, but there’s going to be a point in the near future where I feel we should be pragmatic, and move forward. The release has been postponed for to long now.

In the meantime, the lead developer of the Empathy client has confirmed that updates have been released that should dramatically reduce the impact of the bug. Thanks, Sjoerd! I haven’t tested the new client yet though.

Although I’m having trouble identifying the exact cause, I did manage to make some progress. While investigating, I’ve discovered a number of smaller and bigger issues in the PEP / pubsub routines of Openfire. There’s no obvious direct link between these issues and the memory leak that we’ve been discussing here, but the issues that I’m addressing now are likely candidates, in the sense that these kind of (concurrency-related) bugs are known to cause these kind of problems. I’m currently busy rewriting parts of the PEP routines. I’m hoping that my general improvements make this problem go away (or at least help me to identify the cause). This borders on the edges of educated guessing and wishful thinking, but hey, it’s the holiday season.

Regards,

Guus

Thanks for the detailed update Guus. Much appreciated. If I could buy you a beer I would

Have a great holiday

I have attached a patch that should fix this (and related) problems in the JIRA issue (OF-82). I’d be greatful if someone would give it a test.

Anyone?

Your patch did not patch cleanly against the 3.6.4 source. At least not for me.

jsu2,

Did you get the same error I mentioned in the Jira ticket? If so, you need to convert two of the files from CRLF to unix (dos2unix) and then try to apply the patch

daryl

I have committed my patch - all of the code is in SVN trunk right now. This should simplify getting things started considerably.

We are having serious memory leak issues that are NOT related to Empathy or pep. I started another thread for that incase anyone else is experiencing these issues: http://www.igniterealtime.org/community/thread/40791

Thanks

Daniel