Skip navigation
8708 Views 20 Replies Latest reply: Sep 30, 2011 9:41 AM by Michael Adams RSS
StaticVortex Silver 138 posts since
Jun 22, 2006
Currently Being Moderated

Jan 13, 2010 6:15 PM

Memory Leak Issues - Anyone?

I am starting another thread because we are still experiencing a serious memory leak and can't get to the bottom of it.

 

We have PEP disabled and don't actually have any Empathy clients - all our clients are XIFF.

 

By increading the memory significantly we can keep Openfire running for about 24 hours before it crashes from out of memory exceptions.

 

We are running on 64bit Ubuntu 9.04 server with 4GB RAM - 32bit JAVA (Had issues with 64bit version)

 

We have an average of between 5000 and 8000 people online at any time. CPU and DB load is all fine.

 

The memory slowly leaks away until it is in the red zone and then eventually it dies.

 

Is anyone else experiencing a similar issue? I appreciate any help on this issue.

 

Thanks

Daniel

  • Guus der Kinderen KeyContributor 836 posts since
    Sep 8, 2005
    Currently Being Moderated
    Jan 13, 2010 10:59 PM (in response to StaticVortex)
    Re: Memory Leak Issues - Anyone?

    Hey Daniel,

     

    Well, as in other threads that describe similar issues, my first suggestions are:

    • monitor the JVM, for example with a tool like the Java-monitor plugin;
    • create and analyze thread dumps;
    • create and analyze a heap dump.

     

    I'd be happy to help, if you can provide me with the data.

     

    Regards,

     

      Guus

        • Guus der Kinderen KeyContributor 836 posts since
          Sep 8, 2005
          Currently Being Moderated
          Jan 20, 2010 10:37 AM (in response to StaticVortex)
          Re: Memory Leak Issues - Anyone?

          Disclaimer: I hardly had any sleep in the last few days (but my baby boy is doing great ). I've noticed that this does affect my concentration considerably, so what I'm about to type might not make any sense at all, but:

           

          Why all the Jetty-related threads? I'll be the first one to admit that I haven't looked at the Jetty Continuation thingy yet, but isn't the idea behind async servlet handling not having to keep all of those threads open? There appear to be two pretty big threadpools - are those required. Can the sizes of those pools be monitored and/or capped, perhaps?

  • Walter Ebeling KeyContributor 689 posts since
    Oct 25, 2006
    Currently Being Moderated
    Jan 13, 2010 11:36 PM (in response to StaticVortex)
    Re: Memory Leak Issues - Anyone?

    Hi Daniel,

     

    same situation around here. Roughly 2000 concurrent users, Kraken as IM gateway in use, Spark 2.5.8 deployed, PEP disabled.

     

    The server dies within 2-3 weeks, Nagios reports Java memory errors prior the server failure. Usually the first sign of a imminent server failure is the loss of the admin interface. Furtunately, our SLA for Spark is low and we don't have to provide 24x7.

     

    Walter

    • Guus der Kinderen KeyContributor 836 posts since
      Sep 8, 2005
      Currently Being Moderated
      Jan 21, 2010 2:36 AM (in response to StaticVortex)
      Re: Memory Leak Issues - Anyone?

      Excellent find, and thanks for documenting a workaround too. I've created OF-333 to track this issue. If you could provide a patch - please! We're quite short-handed at the moment - any help is welcome.

      • Guus der Kinderen KeyContributor 836 posts since
        Sep 8, 2005
        Currently Being Moderated
        Jan 27, 2010 10:17 AM (in response to Guus der Kinderen)
        Re: Memory Leak Issues - Anyone?

        I made some modifications to Roster, RosterItem and CacheSizes. These should improve your milage quite a bit. Could you verify if this works for you?

         

        The long term solution would be to move out the "Cacheable" interface completely. As far as I see, we need it only for the DefaultCache implementation that Jive wrote - we should switch that for an open source alternative, which will give us the opportunity to move Coherence out too. That should give us a clustering solution that's completely open source.

  • kot Bronze 9 posts since
    Sep 28, 2009
    Currently Being Moderated
    Jan 28, 2010 5:35 PM (in response to StaticVortex)
    Re: Memory Leak Issues - Anyone?

    Hello everyone.

     

    I have memory leak to.

     

    What I think for...

     

    I think that MAIN memory leak results from bad tcp sessions (except low memory leak in enother modules).

     

    Java as .Net has a GC that clean unused datas. But GC doesn`t clean threads if they was not closed in code. Also one thread occupy memory in java virtual memory and OS native memory simultaneously.

     

    Reasons of unclosing of threads may be cause of hang up tcp session (some threads for one tcp session): when tcp (acs and other error) is appear for session, OF can`t see it (if you use switch for network) and OF client (or transports servers like icq, msn and others) establish new connection with him. But the old session is not closed and hang ups reducing the java and NATIVE mamory.

     

    So that is main memory leak that appers in transport and OF connection.

     

    I wrote plugin that disconnect idle sessions after 60 min. but the problem that it`s clean logical sessions of OF but not network.

     

    So, how to select idle network sessions ,I do not know yet.

    Also I use 128k of stack for threads in JVM settings - that`s increase time for out of memory exception come 

  • Michael Adams Bronze 20 posts since
    Dec 23, 2008
    Currently Being Moderated
    Sep 30, 2011 9:41 AM (in response to StaticVortex)
    Memory Leak Issues - Anyone?

    I figured out how to capture a memory leak report on the latest build (September 30, 2011). Memory usage was heavily concentrated in PEPService, TaskQueue, and the language modules.

     

    http://community.igniterealtime.org/servlet/JiveServlet/previewBody/2207-102-1-2 490/leak_report_09302011.pdf

More Like This

  • Retrieving data ...

Bookmarked By (2)

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points