92 Replies Latest reply on Nov 30, 2012 6:44 AM by ewolkowicz

    Openfire 3.6.4 memory leak with Empathy

    Bronze

      Hi all,

       

      We've just installed Opefire 3.6.4 using the RPM from this site (with the bundled JRE). We have about 60 users, using a mixture of Spark, Empathy 2.28.0.1, Pidgin 2.5.5 and something called SJPhone.

       

      We seem to be getting a leak, where memory just keeps going up until the server falls over. The machine has 1GB of RAM, with 768MB Max for the JVM.

       

      The "Environment" section currently looks like this:

       

      Java Version:1.6.0_03 Sun Microsystems Inc. -- Java HotSpot(TM) Server VM
      Appserver:jetty-6.1.x
      Host Name:cammesg01.city-link.co.uk
      OS / Hardware:Linux / i386
      Locale / Timezone:en / Greenwich Mean Time                     (0 GMT)
      Java Memory
      657.56 MB of 759.50 MB (86.6%) used

       

      This seems unacceptably high. The GC from Java releases a little each time, but after about a day or so it's usually all used up and the server falls over.

       

      The plugins installed are: Broadcast, Client Control, Monitoring, Presence, User Search, User Import/Export

       

      Database is MySQL. Remote Services Disabled. Connection Manager Disabled. Media Proxy Disabled.

       

      Current Users
      Low:                                 059High:                                 60

       

       

      Active Conversations
      Low:                                 07High:                                 13

       

      Packets Per Minute
      Low:                                 8191High:                                 461

       

       

      I'd like to start using this server for more users via the http-bind functionality and strophejs, but need to get this leaky problem sorted out first. Any ideas why this could be happening?

       

      All I can think of doing is upgrading the Java version but I get a feeling this won't solve the issue.

       

      Many thanks in advance

      Dave

        • Re: Openfire 3.6.4 memory leak
          Bronze

          We are having the same issue.  What is interesting is that we did not have this problem until I started using Empathy.  The rest of the staff use PSI or Pidgin.  I was the only user using Empathy that comes with the Ubuntu 9.10 edition. 

          1 of 1 people found this helpful
          • Re: Openfire 3.6.4 memory leak
            StaticVortex Silver

            We are also experiencing a memory leak issue.

             

            We have 3GB dedicated to Openfire and the memory just slowly leaks away. Right now Openfire is using 1.7GB and the cache shows that it has cached about 400MB. There are only 2400 people online (relatively quiet).

             

            We are running on Ubuntu server. In the evenings when the site is busy (around 7000 people online) the memory is totally used up and the GC is running continuously. It will crash after a few days as there won't be enough RAM. If I restart openfire the memory usage will drop right back to a few hundred MB like it should be.

             

            Can anyone advise a good way to hunt down what is going on with all our memory?

             

            Thanks

            Daniel

            • Re: Openfire 3.6.4 memory leak
              Bronze

              I've filed a bug upstream against Empathy too: https://bugzilla.gnome.org/show_bug.cgi?id=598522

              • Re: Openfire 3.6.4 memory leak
                Michael Bronze

                Hi folks,

                 

                I just noticed the same problem with 3.6.4.  I was using 3.6.3 before this until the pubsub problem (I've started a thread on this) surfaced.  I disabled pubsub in 3.6.4 and gave it a 1024M max heap.  With ~200 sessions, the heap was up to 1000M within a few hours.  None of the users are on Empathy though.

                • Re: Openfire 3.6.4 memory leak
                  Bronze

                  Thank god I found this thread - ever since I moved to Empathy in Ubuntu 9.10, my server has been crashing with 100% CPU usage. Changed server-to-server setting, increased java memory size, but no good. I just switched back to Linux Spark.

                  • Re: Openfire 3.6.4 memory leak
                    Bronze

                    I've been having this problem for awhile, making it more difficult to use Openfire in a Ubuntu environment. Is there any development being made on this?

                    • Re: Openfire 3.6.4 memory leak
                      Guus der Kinderen KeyContributor

                      Most of the people that are responding to this thread are reporting that Openfire runs out of memory (causing OutOfMemoryExceptions) after at least one user started to use the Empathy client. I've created a new bug report for the Empathy issue specificly: OF-82

                       

                      So far, I've not been able to identify where things go wrong. We need your help!

                       

                      I would like to receive a couple of  thread dumps (or possibly, a memory dump) of an instance of Openfire that is about to run out of memory. If you can provide these dumps, please contact me (contact details can be found in my profile).

                       

                      Most likely, the memory leak is linked to specific functionality. Are there any clues as to what functionality causes this problem?

                        • Re: Openfire 3.6.4 memory leak
                          davenz Bronze

                          I'll set up an Openfire instance in a Solaris zone on our private network specifically to test this. Keen to help if I can.

                           

                          Cheers,

                          Dave

                            • Re: Openfire 3.6.4 memory leak
                              davenz Bronze

                              Right, I have the following set up on OpenSolaris snv_127

                               

                              Server Properties
                              Version:     Openfire 3.6.4

                               

                              Environment
                              Java Version:     1.6.0_15 Sun Microsystems Inc. -- Java HotSpot(TM) Server VM
                              Appserver:     jetty-6.1.x
                              OS / Hardware:     SunOS / x86
                              Java Memory     36.20 MB of 494.69 MB (7.3%) used

                               

                              Using the embedded DB and local accounts only.

                               

                              I've disabled anonymous login, inbound account registration, and the server-to-server service. All other settings are at defaults.

                               

                              The only plug-in present is:

                               

                              Search Provides support for Jabber Search (XEP-0055) 1.4.3

                               

                              Next step, log in using Empathy on an Ubuntu 9.10 Virtualbox VM to see what happens....

                            • Re: Openfire 3.6.4 memory leak
                              Guus der Kinderen KeyContributor

                              Additionally, a list of all plugins that your system is running (if you didn't provide this list yet) could also be helpful.

                                • Re: Openfire 3.6.4 memory leak
                                  Guus der Kinderen KeyContributor

                                  I'm noting now that at a number of users that are experiencing this problem are using the monitoring plugin. This could be coincidence, of course, but lets check, to be sure. Can you guys reproduce the problem without the monitoring plugin? Can you stop the problem by unloading the monitoring plugin (keep an eye on java-monitors memory usage graphs after you do this!)

                                   

                                  Similar checks can be used to eliminate other plugins too.

                                    • Re: Openfire 3.6.4 memory leak
                                      mikeycmccarthy Silver

                                      Hi all,

                                       

                                      I think I may have started some of this - I'm on holiday at the moment so this may be the last post for a while but just to sum up some of our findings when running quite an intensive load test on Openfire:

                                       

                                      - We needed to add the stalled session property that Guus has mentioned. When our clients dropped off for various reasons (out of memory exceptions on clients, network errors etc) Openfire did not deal with it that efficiently.

                                      - We set our heap size wrong on the Openfire server itself, we were under the impression it had more memory than it did so set our heap size too high.

                                      - The settings for the logging of room conversations by default does not really lend itself to heavy MUC usage (but fine for IM). We were generating pretty high traffic (2000 users, something like 3 chats per second). If you look at the source code I believe the default is to log in batches of 50. I can't remember the interval off the top of my head but basically our log of messages stored in memory was way too high. We've now upped the batch size and shortened the interval size and it's behaving much better, logging 250 messages per minute, although as it's not a real SQL batch statement (it's individual inserts) I would imagine the database may be getting a bit of a hammering.

                                       

                                      Anyway, I believe our Openfire is up for the time being - thanks for all your help all : )

                                      • Re: Openfire 3.6.4 memory leak
                                        davenz Bronze

                                        Hi Guss,

                                         

                                        Everything appears fine on the nascent test system I set up (above)....so out of curiosity I logged in to our production server using Empathy 2.28.1.1 from an Ubuntu 9.10 virtual machine. And bingo - it's all on. Empathy indeed appears to be the kiss of death for OpenFire.

                                         

                                        I can confirm that what I'm seeing is without the monitoring plugin installed.

                                         

                                        The production system details:

                                         

                                        Version:             Openfire 3.6.4

                                         

                                        Environment
                                        Java Version:     1.6.0_07 Sun Microsystems Inc. -- Java HotSpot(TM) Server VM
                                        Appserver:     jetty-6.1.x
                                        OS / Hardware:     Linux / i386
                                        Java Memory     171.62 MB of 253.19 MB (67.8%) used

                                         

                                        Other differences from the above system I posted:

                                         

                                        - uses an external MySQL DB

                                        - users authenticate using our coporate AD installation

                                         

                                        I haven't got JConsole hooked up so am monitoring by refreshing the OF admin console page

                                         

                                        Over the course of an hour I have observed JVM memory usage creep from 20 percent with half a dozen logged on users to at the time of writing 80 percent consumed.

                                         

                                        All other users are on Spark 2.5.8 or Pidgin. I am the only user with Empathy.

                                         

                                        The only plug-ins installed on this system are:

                                         

                                        Red5 v0.1.11

                                        Search v1.4.3

                                         

                                        We are not using Red5 Sparkweb functionality - i.e we have it set on a trial basis for users if they wish to tinker, but as yet no-one is actually using it.

                                         

                                        Another observation - if you really want to exacerbate the behaviour, quit Empathy and fire it back up in rapid sucession...

                                         

                                        I'm not going to be able to get you thread dumps with this particular setup (sorry), but will return to my test system tomorrow and try and get some concrete data for you.

                                         

                                        Cheers!
                                        Dave

                                         

                                         

                                        EDIT: JVM heap size just hit 90 percent of capacity...so I'm gonna do a quick bounce of the server before anyone notices...

                                         

                                        Message was edited by: davenz

                                          • Re: Openfire 3.6.4 memory leak
                                            Bronze

                                            I too am happy to find this thread.

                                             

                                            We have a very modest Openfire installation, just 46 users, which ran untended and uninterupted for hundreds of days until November 5th, when we had the first "out of Java memory" crash. We took the enforced downtime as an opportunity to upgrade to 3.6.4.

                                             

                                            It too crashed seven days later - out of Java resources..

                                             

                                            Today (five days later) Java memory usage was at nearly 90% (of 1GB) so we restarted Openfire to preempt another crash.

                                             

                                            Most of our clients are Pidgin (on Windows & Ubuntu), plus a smattering of OS X iChat. A couple of the Ubuntu users switched to Empathy on release of Ubuntu 9.10 (29 October). This does seem to have coincided the onset of our problems.

                                             

                                            We have requested Empathy users to switch back to Pidgin and are monitoring closely.

                                             

                                            Regards,

                                      • Re: Openfire 3.6.4 memory leak
                                        davenz Bronze

                                        Hi Guus,

                                         

                                        Matter of fact I did get a thread dump on our production system, easy when you know how I guess

                                         

                                        I've replied to Gato in this thread:

                                         

                                        http://www.igniterealtime.org/community/message/198160#198160

                                         

                                        And I'm attaching the dump as well.

                                         


                                        In this case I was able to expend JVM memory completely in about five (5) minutes from bringing the OF server up - simply by repeatedly exiting and launching Empathy rapidly. This was with one user (me), with the remaining three or so users (at this late time of the day hehe) using Spark or Pidgin. If I am reading this right then I can easily see how in large deployments several Empathy users would create headaches.

                                         

                                         

                                         

                                        JVM memory status: 252.80 MB of 253.19 MB (99.8%) used

                                         

                                        # ps -ef | grep -i java


                                        daemon   30532     1 14 19:05 pts/0    00:01:33 /usr/lib/jvm/java-1.6.0-sun-1.6.0.u7/jre/bin/java -server -Xms128m -Xmx256m -DopenfireHome=/opt/openfire -Dopenfire.lib.dir=/opt/openfire/lib -classpath /opt/openfire/lib/startup.jar -jar /opt/openfire/lib/startup.jar

                                         

                                        # kill -3 30532

                                         

                                        # cd /opt/openfire/logs/

                                        # more nohup.out

                                         

                                        Output in the attached.

                                         

                                        Lemme know if this helps at all

                                      • Re: Openfire 3.6.4 memory leak
                                        Bronze

                                        Same problem here, but we are not using empathy. Our server only allow connections from Spark and Pidgin clients.

                                         

                                        We don't use Sun JDK, but OpenJDK instead

                                         

                                        Java Version:1.6.0_0 Sun Microsystems Inc. -- OpenJDK Server VM
                                        Appserver:jetty-6.1.x
                                        Host Name:openfire-im
                                        OS / Hardware:Linux / i386
                                        Locale / Timezone:en / Central European Time                     (1 GMT)

                                         

                                        These are our enabled plugins:

                                        Client Control 1.0.3

                                        Email Listener 1.0.0

                                        Kraken IM 1.1.2 (with Yahoo, MSN and Gtalk enabled)

                                        Monitoring Service 1.1.1

                                        Registration 1.4.1

                                        Search 1.4.3

                                        User Import/Export 2.2.0

                                         

                                        Out of memory happens randomly. It can take up to 15 days... or 15 hours.

                                          • Re: Openfire 3.6.4 memory leak
                                            davenz Bronze

                                            Not necessarily the same problem then - there seems to be a growing body of evidence implicating Empathy in this particular issue, and in my case it appears I have a solid test case I can reproduce this consistently on.

                                              • Re: Openfire 3.6.4 memory leak
                                                Guus der Kinderen KeyContributor

                                                I agree with Dave here. Given the amount of noise surrounding the Empathy client, something must be up there. You are probably not running into the same problem, but into something different with similar effects.

                                                  • Re: Openfire 3.6.4 memory leak
                                                    Bronze

                                                    Yes, it could be empathy, of course.

                                                    But I have sent you my own problem in order to find a possible pattern not related with empathy.

                                                    It probably will be empathy and our problem is another one, but perhaps is a plugin we are all using. I'm only trying to apply a "Dr House differential diagnostic". ;-)

                                                    I've also found some exceptions in log regarding email listener and some others related to Gtalk personal groups.

                                                  • Re: Openfire 3.6.4 memory leak
                                                    wroot KeyContributor

                                                    I think there are at least 3-4 different issues discussed on this thread and this gets very confusing, for Guus too, i think. Would be better to discuss Empathy issue on Empathy thread. For random out of memory issues a separate threads should be started too.

                                                • Re: Openfire 3.6.4 memory leak
                                                  Bronze

                                                  I've changed the thread title to more acurately reflect the issue being discussed here.

                                                  • Re: Openfire 3.6.4 memory leak
                                                    Bronze

                                                    I am running Openfire 3.6.2 on Ubuntu Server 8.10.

                                                     

                                                    Java Version:1.6.0_0 Sun Microsystems Inc. -- OpenJDK Server VM
                                                    Appserver:jetty-6.1.x

                                                     


                                                    I started having this same exact issue after I setup a laptop with Ubuntu Desktop 9.10 with the Empathy IM client.  Once I saw this post I turned off the Ubuntu laptop and rebooted the Ubuntu Server.  The problem has not come back since.

                                                    • Re: Openfire 3.6.4 memory leak with Empathy
                                                      Guus der Kinderen KeyContributor

                                                      I think I've uncovered the cause of the problem. It appears to be Openfire's implementation of XEP-0163 "Personal Eventing Protocol."

                                                       

                                                      As a workaround, you can disable PEP by adding this Openfire System Property (you can add/modify properties through the Openfire Admin console):

                                                       

                                                      The property xmpp.pep.enabled should be set to: false

                                                      1 of 1 people found this helpful
                                                        • Re: Openfire 3.6.4 memory leak with Empathy
                                                          Bronze

                                                          Good news! I have added this setting to our server, I will monitor the memory use now. Thanks a lot for your work!

                                                          • Re: Openfire 3.6.4 memory leak with Empathy
                                                            Bronze

                                                            Hi Guus,

                                                             

                                                            Are there plans for this to be fixed in the next Openfire release (and if so is there a timescale on this)? Or will we have to stick with disabling the PEP?

                                                             

                                                            Thanks for your help.

                                                              • Re: Openfire 3.6.4 memory leak with Empathy
                                                                Guus der Kinderen KeyContributor

                                                                Hi Dave,

                                                                 

                                                                Sorry that it has taken so long to get back to you.

                                                                 

                                                                I've been trying to identify the cause of the problem, but so far, have been unable to do so. I've asked other developers to have a look too, but none of them have been successful either. Sadly, my attempts are severely hindered by lack of time - I'm doing this in my spare time, which is limited.

                                                                 

                                                                I'm not comfortable at all releasing the next version of Openfire with this bug in it, but there's going to be a point in the near future where I feel we should be pragmatic, and move forward. The release has been postponed for to long now.

                                                                 

                                                                In the meantime, the lead developer of the Empathy client has confirmed that updates have been released that should dramatically reduce the impact of the bug. Thanks, Sjoerd! I haven't tested the new client yet though.

                                                                 

                                                                Although I'm having trouble identifying the exact cause, I did manage to make some progress. While investigating, I've discovered a number of smaller and bigger issues in the PEP / pubsub routines of Openfire. There's no obvious direct link between these issues and the memory leak that we've been discussing here, but the issues that I'm addressing now are likely candidates, in the sense that these kind of (concurrency-related) bugs are known to  cause these kind of problems. I'm currently busy rewriting parts of the PEP routines. I'm hoping that my general improvements make this problem go away (or at least help me to identify the cause). This borders on the edges of educated guessing and wishful thinking, but hey, it's the holiday season.

                                                                 

                                                                Regards,

                                                                 

                                                                  Guus

                                                              • Re: Openfire 3.6.4 memory leak with Empathy
                                                                plastikat Bronze

                                                                Hi, Guus and All!

                                                                 

                                                                My configuration:

                                                                LDAP (AD) users and groups (around 120 users), rosters are distributed via shared groups, around six various plugins (I've been trying disabling all of them while digging this problem, so the list is not important).

                                                                My environment:

                                                                Server (current): Openfire 3.7.0 release (started using with Openfire 3.6.4 release and further through some nightly builds and beta).

                                                                Clients (current): Miranda IM 0.9.19.0 (started using with Miranda IM 0.9.10.0 and further through stable releases).

                                                                 

                                                                I've started suffering from daily memory runouts since putting 3.6.4 in production, Java memory configuration was default, min=max=64 MB. Then, I've tried increasing memory to min=64 MB max=256 MB, but it extended total memory runout period only (to about 2 days). My temporary solution was scheduling a nightly server restart - and even if it was pretty acceptable in my (corporative) environment, it is not good for any server at all.

                                                                 

                                                                I've read about Empathy/PEP/Java-memory some time ago, but: firstly, we don't use Empathy at all; and secondly, it was not clear to me what actually PEP is. So, I've been waiting for Openfire 3.7.0 release, hoping that it would solve my memory problem - and when 3.7.0 was finally released - and the problem was not solved - I've started analyzing and digging actively. I must say, that Java Monitor is really a great tool for troubleshooting Java apps & servers (thank Guus for telling about it), it helped to see what's going on inside my server (and keeps telling me that).

                                                                 

                                                                When influence of all of my plugins has been excluded, I've recalled this PEP thing, and have carefully read about it. I've found out that it actually intercrosses with my other issue, but no one (not even guru's ) have pointed that out to me. Now, I confirm that disabling PEP solved my long-suffered memory runouts (leak) problem, and here are a couple of related findings:

                                                                 

                                                                     1. I think that this problem is not related to Empathy only, I suppose it is related to any client, that supports PEP (i.e. sending/receiving Mood/Activity/Tunes information or other "personal events") - and this aspect should be pointed out here or in some other place - so that people would be clearly aware of it. If "personal events" are not actually being sent by clients (i.e. clients are not capable of doing so) - it doesn't matter if PEP is enabled or not, the problem comes out only when PEP is actually used. And, until PEP realisation in Openfire is a known memory hog (until bugs are found and fixed) - wouldn't making it disabled by default (out-of-box) be a good idea? I guess there are very few folks that use it consciously (and very few clients that support it).

                                                                 

                                                                     2. As far as I understand setting property xmpp.pep.enabled to false doesn't actually disable PEP capability advertising by server, it simply disables processing of events of this given type - becase after setting this property and restarting server I still get (while connecting):

                                                                 

                                                                <iq type="result" id="****" from="****" to="****@****/****">
                                                                     <query xmlns="http://jabber.org/protocol/disco#info">
                                                                          <identity category="server" name="Openfire Server" type="im" />
                                                                          <identity category="pubsub" type="pep" />
                                                                          <feature var="http://jabber.org/protocol/pubsub#manage-subscriptions" />

                                                                          ...

                                                                          <feature var="http://jabber.org/protocol/rsm" />
                                                                     </query>
                                                                </iq>

                                                                 

                                                                So, is there a way to disable PEP-advertising (i.e. completely disable PEP)? If there is no such way - it would be desirable.

                                                                The point is that Miranda IM clients (I don't know about other clients, including Empathy) show "personal events" selecting/setting menus only when this capability is advertised by server, and if it's not - there are simply no such menus (I've found this out long ago, by trying to connecting to some public non-PEP non-Openfire servers). Thus, it's not obvious to people why they have these menus and they can't set "personal events".

                                                              • Re: Openfire 3.6.4 memory leak with Empathy
                                                                Guus der Kinderen KeyContributor

                                                                I have attached a patch that should fix this (and related) problems in the JIRA issue (OF-82). I'd be greatful if someone would give it a test.

                                                                • Re: Openfire 3.6.4 memory leak with Empathy
                                                                  StaticVortex Silver

                                                                  We are having serious memory leak issues that are NOT related to Empathy or pep. I started another thread for that incase anyone else is experiencing these issues: http://www.igniterealtime.org/community/thread/40791

                                                                   

                                                                  Thanks

                                                                  Daniel

                                                                  • Openfire 3.6.4 memory leak with Empathy
                                                                    Michael Adams Bronze

                                                                    I figured out how to capture a memory leak report on the latest build (September 30, 2011). Memory usage was heavily concentrated in PEPService, TaskQueue, and the language modules.

                                                                     

                                                                    http://community.igniterealtime.org/servlet/JiveServlet/previewBody/2207-102-1-2 490/leak_report_09302011.pdf

                                                                    • Openfire 3.6.4 memory leak with Empathy
                                                                      Aaron Evans Bronze

                                                                      I can confirm that setting the property xmpp.pep.enabled set to false prevents a memory leak in openfire 3.7.

                                                                       

                                                                      We had some folks using a few different clients (adium, pidgin, etc) and at least one of them was definitely causing a memory leak and we'd have to restart openfire at least once a week with a heap size of around 1 GB.

                                                                       

                                                                      Now that we've put that property in place, it's been smooth sailing ever since.  BTW, we have 3 other servers where we tightly controlled which clients were allowed to connect and they did not suffer from the memory leak presumably because the client used didn't use PEP features.

                                                                       

                                                                      In my opinion, this is a MAJOR bug with openfire - a client program should not be able to take down your server by generating a memory leak.  The server should correctly handle whatever the client does to prevent this.

                                                                       

                                                                      I'm rather surprised that this issue has persisted for so long and that it hasn't been addressed...

                                                                      • Openfire 3.6.4 memory leak with Empathy
                                                                        Rodrigo Torres Bronze

                                                                        I want to report that the memory leak still exists on 3.7.1 and setting xmpp.pep.enabled=false seems to solve the problem.

                                                                          • Openfire 3.6.4 memory leak with Empathy
                                                                            rcollier KeyContributor

                                                                            Yes, that is already known.  I was referring to the next version, the as of yet unreleased 3.7.2, which is why I was mentioning that you would need to download the nightly build.

                                                                             

                                                                            There are instructions related to that in the bottom of this thread.  Unfortunately, the nightly build from the downloads page has not been updated since December, so that isn't of any use.

                                                                          • Openfire 3.6.4 memory leak with Empathy
                                                                            ewolkowicz Bronze

                                                                            Not sure if this will help anyone but I was able to fix memory issues I was having by pointing openfire to a different, 64 bit JRE.  The OS is RHEL 6.1.  Here is my config file from /etc/sysconfig/openfire.

                                                                             

                                                                            JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre

                                                                             

                                                                            OPENFIRE_OPTS="-Xms256m -Xmx1024m -Xss128k -Xoss128k -XX:ThreadStackSize=128 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:NewRatio=2 -XX:+PrintGCDetails -Xloggc:/opt/openfire/logs/gc.log"

                                                                             

                                                                            I run openfire 3.7.0.  Other than a few administrative changes yesterday, I haven't had to restart Openfire in many many months after using that JRE.  I keep the gc log enabled since it barely takes up any space (now that there's no memory errors ).  That was how I was able to figure out where the problem was originally.