Skip navigation
Up to Discussions in Openfire Support
8716 Views 36 Replies Latest reply: Jan 13, 2010 8:44 AM by Guus der Kinderen RSS
ddiggler Bronze
ddiggler
17 posts since
Nov 11, 2009
Currently Being Moderated

Dec 16, 2009 1:09 PM

Openfire service crashing daily

The service will run for a day, maybe two, then it will stop with the error message below.  The error is identical to post http://www.igniterealtime.org/community/message/198731#198731.  I posted a response in that forum 6 days ago with no response.  I figured I try creating my own thread.  I also posted a thread in http://kraken.blathersource.org/node/266.  I have a feeling the problems may be related.

 

We are running Openfire 3.6.4 w/ 1.6.0_17 JVM

Windows Server 2003 SP2 (Virtual Server in VMware ESX)

Dual 2.3Ghz Procs / 3GB of RAM

-Xms1280m -Xmx1280m  (You can see all arguments in the error msg below.  We tried 256/256, 512/512, 1024/1024, 1024/1536 and 512/1024 as recommended in the linked article..same problems)

 

Plugins:

Client Control: Only allow Spark client

Kraken IM Gateway 1.1.2: We control IM Gateway access through groups.  Only 170 users are allowed to used it.

Monitoring Service:  Full archiving is enabled.  User to User and group chats

Red5:  Used for video conferencing test (3 users) and where JWCHAT (~200 JWCHAT users) is setup .  HTTP Binding is enabled for this.

Search: Used for easy searching...duh

User Import Export:  Used for migration of user from old IM solution

 

 

We have 3000 accounts w/ a max of 1500 concurrent users.  We average 1300 to 1400 concurrent users.

 

Running SQL 2005 DB (on a dedicated SQL server)

Microsoft SQL Server Management Studio  9.00.4035.00
Microsoft Analysis Services Client Tools 2005.090.4035.00
Microsoft Data Access Components (MDAC)  2000.086.3959.00 (srv03_sp2_rtm.070216-1710)
Microsoft MSXML    2.6 3.0 6.0
Microsoft Internet Explorer  7.0.5730.13
Microsoft .NET Framework  2.0.50727.3082
Operating System    5.2.3790

 

Error Message...I excluded the process list due to length.  If this will help, I can post it.  If anyone has a solution, I am in desperate need of help. Everything worked great during our testing of 400-500 concurrent users.

 

#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 835968 bytes for Chunk::new. Out of swap space?
#
#  Internal Error (414C4C4F434154494F4E0E43505000C7), pid=3188, tid=3168
#
# Java VM: Java HotSpot(TM) Server VM (1.6.0_03-b05 mixed mode)
# If you would like to submit a bug report, please visit:
#  http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x4801f800):  JavaThread "CompilerThread1" daemon [_thread_in_native,]

Stack: [0x483b0000,0x48400000)
[error occurred during error reporting, step 110, id 0xc0000005]


Current CompileTask:
C2:2692      org.jivesoftware.openfire.plugin.SearchPlugin.replyDataFormResult(Ljava/util/Co llection;Lorg/xmpp/packet/IQ;)Lorg/xmpp/packet/IQ; (416 bytes)

 

VM Arguments:
jvm_args: -Dexe4j.isInstall4j=true -Dexe4j.isService=true -Dexe4j.moduleName=C:\Program Files\Openfire\bin\openfire-service.exe -Dexe4j.processCommFile=C:\WINDOWS\TEMP\e4j_p3188.tmp -Dexe4j.tempDir= -Dexe4j.unextractedPosition=0 -Dexe4j.consoleCodepage=cp0 -Xrs -Xms1024m -Xmx1024m
java_command: <unknown>
Launcher Type: generic

Environment Variables:
PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;c:\program files\openfire\jre\bin
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 15 Stepping 8, GenuineIntel

 

---------------  S Y S T E M  ---------------

OS: Windows Server 2003 family Build 3790 Service Pack 2

CPU:total 2 (4 cores per cpu, 1 threads per core) family 6 model 15 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, physical 2096584k(1255816k free), swap 4194303k(4194303k free)

vm_info: Java HotSpot(TM) Server VM (1.6.0_03-b05) for windows-x86, built on Sep 24 2007 22:20:35 by "java_re" with unknown MS VC++:1310

 

 

 

We get the errors I posted below for a few hours then the OpenFire admin console will stop responding for 30-45 minutes, then the service crashes.

 

------------------------------------------------------------------error--------- ---------------------------------------------------

2009.12.16 11:36:00 [org.jivesoftware.openfire.nio.ConnectionHandler.exceptionCaught(ConnectionHand ler.java:110)
]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at net.kano.joscar.flap.AsynchronousFlapProcessor.<init>(AsynchronousFlapProcessor .java:29)
at net.kano.joscar.flap.ClientFlapConn.init(ClientFlapConn.java:82)
at net.kano.joscar.flap.ClientFlapConn.<init>(ClientFlapConn.java:73)
at net.sf.kraken.protocols.oscar.AbstractFlapConnection.<init>(AbstractFlapConnect ion.java:102)
at net.sf.kraken.protocols.oscar.LoginConnection.<init>(LoginConnection.java:41)
at net.sf.kraken.protocols.oscar.OSCARSession.logIn(OSCARSession.java:120)
at net.sf.kraken.protocols.oscar.OSCARTransport.registrationLoggedIn(OSCARTranspor t.java:95)
at net.sf.kraken.BaseTransport.processPacket(BaseTransport.java:398)
at net.sf.kraken.BaseTransport.processPacket(BaseTransport.java:199)
at org.jivesoftware.openfire.component.InternalComponentManager$RoutableComponents .process(InternalComponentManager.java:619)
at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.jav a:260)
at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:164)
at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:70)
at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76)
at org.jivesoftware.openfire.net.StanzaHandler.processPresence(StanzaHandler.java: 337)
at org.jivesoftware.openfire.net.ClientStanzaHandler.processPresence(ClientStanzaH andler.java:85)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:254)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:176)
at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:133)
at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)
at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Unknown Source)

2009.12.16 11:35:51 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:92)
] error sending msg: MSG 59 U 92
====================
     Chunk Debug   
====================
MIME-Version: 1.0
Content-Type: text/x-msmsgscontrol
TypingUser: <user>@hotmail.com

 

====================
Binary Chunk Debug
====================
00000000h: 4D 49 4D 45 2D 56 65 72 73 69 6F 6E 3A 20 31 2E ; MIME-Version: 1.
00000010h: 30 0D 0A 43 6F 6E 74 65 6E 74 2D 54 79 70 65 3A ; 0..Content-Type:
00000020h: 20 74 65 78 74 2F 78 2D 6D 73 6D 73 67 73 63 6F ;  text/x-msmsgsco
00000030h: 6E 74 72 6F 6C 0D 0A 54 79 70 69 6E 67 55 73 65 ; ntrol..TypingUse
00000040h: 72 3A 20 6A 72 61 31 39 37 36 40 68 6F 74 6D 61 ; r: <user>@hotma
00000050h: 69 6C 2E 63 6F 6D 0D 0A 0D 0A 0D 0A             ; il.com......

java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at java.io.ByteArrayOutputStream.writeTo(Unknown Source)
at net.sf.jml.net.Session.sendMessage(Session.java:511)
at net.sf.jml.net.Session.access$1300(Session.java:30)
at net.sf.jml.net.Session$MsgSender.run(Session.java:476)
at java.lang.Thread.run(Unknown Source)

------------------------------------------------------------------error--------- ---------------------------------------------------

  • LG KeyContributor
    LG
    6,132 posts since
    Dec 13, 2005
    Currently Being Moderated
    Dec 19, 2009 10:12 AM (in response to ddiggler)
    Re: Openfire service crashing daily

    Hi,

     

    your "unable to create new native thread" error shows that the Java heap (Xmx) is not your problem. You may want to try to set "-XX:ThreadStackSize=128" to have more space for native threads.

     

    LG

    • ddiggler Bronze
      ddiggler
      17 posts since
      Nov 11, 2009
      Currently Being Moderated
      Dec 22, 2009 1:37 PM (in response to LG)
      Re: Openfire service crashing daily

      Would this be set in the openfire-service.vmoptions file also?  I found this article (http://www.igniterealtime.org/community/docs/DOC-1033).  What would be recommended settings for "-Xss128k -Xoss128k -XX:ThreadStackSize=128" for 2000 users.  I understand it depends on plugins and what the users are doing..  I already have -Xms1280 and -Xmx1280 set.  What would be required to set - Xss or -Xoss based on our load/configuration?  If this is to subjective to suggest settings, how would I determine what's appropriate

       

      I have this posted in the Kraken forums also.  Just in case it's a problem with the plugin.

      http://kraken.blathersource.org/node/266

       

      We where running 1.1.2 and updated it to 1.1.3.  The service ran for the entire following day and the weekend (less load).  Come Monday (yesterday) morning the Openfire console wasn't working but the port was responding.  IM was still working...   The server CPU was pegged to 100%.  I was off so a co-worker just rebooted the server.  I would assumed it would have crashed again if he wouldn't have..  ?

       

      Today so far, so good.  1500 connections under 18% Java memory (-Xms1280m, -Xmx1280m).

       

      If you need me to post my logs or JVM graphs, please let me know.  Our users are not to happy with the constant service crashes.

       

      We've been getting tons of these messages..

       

      2009.12.21 10:39:58 [org.jivesoftware.openfire.event.SessionEventDispatcher.dispatchEvent(SessionEv entDispatcher.java:87)
      ]
      java.lang.IllegalStateException: Timer already cancelled.
      at java.util.Timer.sched(Unknown Source)
      at java.util.Timer.schedule(Unknown Source)
      at org.jivesoftware.openfire.plugin.spark.TaskEngine.schedule(TaskEngine.java:98)
      at org.jivesoftware.openfire.plugin.spark.SparkManager$SparkSessionListener.sessio nCreated(SparkManager.java:373)
      at org.jivesoftware.openfire.event.SessionEventDispatcher.dispatchEvent(SessionEve ntDispatcher.java:63)
      at org.jivesoftware.openfire.SessionManager.addSession(SessionManager.java:528)
      at org.jivesoftware.openfire.session.LocalClientSession.setAuthToken(LocalClientSe ssion.java:589)
      at org.jivesoftware.openfire.handler.IQBindHandler.handleIQ(IQBindHandler.java:141 )
      at org.jivesoftware.openfire.handler.IQHandler.process(IQHandler.java:49)
      at org.jivesoftware.openfire.IQRouter.handle(IQRouter.java:351)
      at org.jivesoftware.openfire.IQRouter.route(IQRouter.java:101)
      at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:68)
      at org.jivesoftware.openfire.net.StanzaHandler.processIQ(StanzaHandler.java:319)
      at org.jivesoftware.openfire.net.ClientStanzaHandler.processIQ(ClientStanzaHandler .java:79)
      at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:284)
      at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:176)
      at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:133)
      at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)
      at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
      at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
      at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
      at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)
      at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
      at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
      at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
      at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)
      at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)
      at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
      at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
      at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
      at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)
      at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
      at java.lang.Thread.run(Unknown Source)

      • vidgizmo Bronze
        vidgizmo
        1 posts since
        Dec 18, 2009
        Currently Being Moderated
        Dec 22, 2009 12:45 PM (in response to ddiggler)
        Re: Openfire service crashing daily

        We resolved this issue by telling people to not use Empathy/Telepathy based IM clients for now (which are apparently the default with the latest release of Ubuntu Linux). Since doing this we've been running w/o problem for almost a week.

         

        I'm monitoring threads the Empathy/Telepathy and Openfire forums to watch for a fix, but none yet.

         

        This search shows lots of threads in the forums related to this topic:

        http://www.igniterealtime.org/community/search.jspa?peopleEnabled=true&userID=&c ontainerType=&container=&q=empathy

        • ddiggler Bronze
          ddiggler
          17 posts since
          Nov 11, 2009
          Currently Being Moderated
          Dec 22, 2009 12:54 PM (in response to vidgizmo)
          Re: Openfire service crashing daily

          Thanks for the reply..  I have seen those postings and we run only the Spark and JWChat web client.

          • sixthring KeyContributor
            sixthring
            3,797 posts since
            Apr 2, 2007
            Currently Being Moderated
            Dec 22, 2009 3:15 PM (in response to ddiggler)
            Re: Openfire service crashing daily

            I would suggest disabling some of the plugins you are running.  They may be the cause of your issues.  Then enable them one at a time.  Or remove them one at a time.  Start with removing client control.

          • LG KeyContributor
            LG
            6,132 posts since
            Dec 13, 2005
            Currently Being Moderated
            Dec 23, 2009 12:33 PM (in response to ddiggler)
            Re: Openfire service crashing daily

            Hi,

             

            the thread stack size applies to every thread and 128k may be too small - you'll get Java errors in your log file if this is the case. Xss should be the same as ThreadStackSize and thus for the native stack size while Xoss should set the java stack size for every thread. As you have a problem with the native thread count setting the ThreadStackSize should be enough.

            You may want to create some stack traces while the server is running - there you may be able to see an increasing number of java threads (every java thread can be mapped to a native thread). Maybe the name helps you to identify which part of Openfire or plugins does start these threads.

             

            LG

            • ddiggler Bronze
              ddiggler
              17 posts since
              Nov 11, 2009
              Currently Being Moderated
              Dec 30, 2009 8:12 AM (in response to LG)
              Re: Openfire service crashing daily

              I added "-XX:ThreadStackSize=128" and set xmpp.pep.enabled=false (just incase).  The service appears to have ran longer (may be due to the holiday) but we still are having problems.  Today the service didn't crash but you couldn't get into the web client or the admin console.  The server CPU was pegged at 100% and the following event was logged in the eventvwr.

               

              Event Type: Error
              Event Source: Srv
              Event Category: None
              Event ID: 2019
              Date:  12/30/2009
              Time:  9:08:59 AM
              User:  N/A
              Computer: <servername>
              Description:
              The server was unable to allocate from the system nonpaged pool because the pool was empty.

              For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
              Data:
              0000: 00 00 04 00 01 00 54 00   ......T.
              0008: 00 00 00 00 e3 07 00 c0   ....ã..À
              0010: 00 00 00 00 9a 00 00 c0   ....&#154;..À
              0018: 00 00 00 00 00 00 00 00   ........
              0020: 00 00 00 00 00 00 00 00   ........
              0028: 08 00 00 00               ....  

              I also noticed out Statistics page is not showing up...it's there, but empty (http://www.igniterealtime.org/community/thread/36238 and http://www.igniterealtime.org/community/message/172876#172876).  We are going to attempt to repair the ofRRDs table this weekend.   Could this be the source of all my crashes?

               

              Here is my error.log around that time..

               

              2009.12.30 09:01:32 [org.jivesoftware.openfire.spi.PresenceManagerImpl.userUnavailable(PresenceMana gerImpl.java:271)
              ] Error storing offline presence of user: <username>
              java.sql.SQLException: Violation of PRIMARY KEY constraint 'ofPresence_pk'. Cannot insert duplicate key in object 'dbo.ofPresence'.
              at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:368)
              at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2816)
              at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2254)
              at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:631)
              at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:584)
              at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:546)
              at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.execute(JtdsPreparedStatement.j ava:559)
              at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
              at java.lang.reflect.Method.invoke(Unknown Source)
              at org.logicalcobwebs.proxool.ProxyStatement.invoke(ProxyStatement.java:100)
              at org.logicalcobwebs.proxool.ProxyStatement.intercept(ProxyStatement.java:57)
              at $java.sql.Wrapper$$EnhancerByProxool$$b2f44fdd.execute(<generated>)
              at org.jivesoftware.openfire.spi.PresenceManagerImpl.userUnavailable(PresenceManag erImpl.java:268)
              at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:145)
              at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:112)
              at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:176)
              at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:134)
              at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:70)
              at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76)
              at org.jivesoftware.openfire.SessionManager$ClientSessionListener.onConnectionClos e(SessionManager.java:1133)
              at org.jivesoftware.openfire.nio.NIOConnection.notifyCloseListeners(NIOConnection. java:202)
              at org.jivesoftware.openfire.nio.NIOConnection.close(NIOConnection.java:185)
              at org.jivesoftware.openfire.nio.ConnectionHandler.sessionClosed(ConnectionHandler .java:87)
              at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.sessionClosed(A bstractIoFilterChain.java:550)
              at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
              at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
              at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
              at org.apache.mina.common.IoFilterAdapter.sessionClosed(IoFilterAdapter.java:65)
              at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
              at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
              at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
              at org.apache.mina.filter.codec.ProtocolCodecFilter.sessionClosed(ProtocolCodecFil ter.java:259)
              at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
              at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
              at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
              at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :249)
              at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
              at java.lang.Thread.run(Unknown Source)
              2009.12.30 09:12:51 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:92)
              ] error sending msg: REM 18 FL 55a97183-9ec9-47d1-960a-d6555bd5c2c7 f530dfb6-9a55-483b-8bbc-cc04b2669407

              java.net.SocketException: Software caused connection abort: socket write error
              at java.net.SocketOutputStream.socketWrite0(Native Method)
              at java.net.SocketOutputStream.socketWrite(Unknown Source)
              at java.net.SocketOutputStream.write(Unknown Source)
              at java.io.ByteArrayOutputStream.writeTo(Unknown Source)
              at net.sf.jml.net.Session.sendMessage(Session.java:511)
              at net.sf.jml.net.Session.access$1300(Session.java:30)
              at net.sf.jml.net.Session$MsgSender.run(Session.java:476)
              at java.lang.Thread.run(Unknown Source)
              2009.12.30 09:12:54 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:92)
              ] error sending msg: REM 19 FL 177032fc-a811-4c18-878d-c388d7041c91 f530dfb6-9a55-483b-8bbc-cc04b2669407

              java.net.SocketException: Software caused connection abort: socket write error
              at java.net.SocketOutputStream.socketWrite0(Native Method)
              at java.net.SocketOutputStream.socketWrite(Unknown Source)
              at java.net.SocketOutputStream.write(Unknown Source)
              at java.io.ByteArrayOutputStream.writeTo(Unknown Source)
              at net.sf.jml.net.Session.sendMessage(Session.java:511)
              at net.sf.jml.net.Session.access$1300(Session.java:30)
              at net.sf.jml.net.Session$MsgSender.run(Session.java:476)
              at java.lang.Thread.run(Unknown Source)
              2009.12.30 09:13:26 [org.jivesoftware.openfire.spi.PresenceManagerImpl.userUnavailable(PresenceMana gerImpl.java:271)
              ] Error storing offline presence of user: <username>
              java.sql.SQLException: Violation of PRIMARY KEY constraint 'ofPresence_pk'. Cannot insert duplicate key in object 'dbo.ofPresence'.
              at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:368)
              at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2816)
              at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2254)
              at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:631)
              at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:584)
              at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:546)
              at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.execute(JtdsPreparedStatement.j ava:559)
              at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
              at java.lang.reflect.Method.invoke(Unknown Source)
              at org.logicalcobwebs.proxool.ProxyStatement.invoke(ProxyStatement.java:100)
              at org.logicalcobwebs.proxool.ProxyStatement.intercept(ProxyStatement.java:57)
              at $java.sql.Wrapper$$EnhancerByProxool$$b2f44fdd.execute(<generated>)
              at org.jivesoftware.openfire.spi.PresenceManagerImpl.userUnavailable(PresenceManag erImpl.java:268)
              at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:145)
              at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:112)
              at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:176)
              at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:134)
              at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:70)
              at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76)
              at org.jivesoftware.openfire.SessionManager$ClientSessionListener.onConnectionClos e(SessionManager.java:1133)
              at org.jivesoftware.openfire.nio.NIOConnection.notifyCloseListeners(NIOConnection. java:202)
              at org.jivesoftware.openfire.nio.NIOConnection.close(NIOConnection.java:185)
              at org.jivesoftware.openfire.nio.ConnectionHandler.sessionClosed(ConnectionHandler .java:87)
              at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.sessionClosed(A bstractIoFilterChain.java:550)
              at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
              at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
              at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
              at org.apache.mina.common.IoFilterAdapter.sessionClosed(IoFilterAdapter.java:65)
              at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
              at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
              at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
              at org.apache.mina.filter.codec.ProtocolCodecFilter.sessionClosed(ProtocolCodecFil ter.java:259)
              at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
              at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
              at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
              at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :249)
              at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
              at java.lang.Thread.run(Unknown Source)
              2009.12.30 09:15:06 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:92)
              ] error sending msg: ADC 14 FL C=86d9264b-67aa-468a-8c8d-d696cf3fa67a 5f088445-3fe7-43ca-a863-52b6990d659e

              java.net.SocketException: Software caused connection abort: socket write error
              at java.net.SocketOutputStream.socketWrite0(Native Method)
              at java.net.SocketOutputStream.socketWrite(Unknown Source)
              at java.net.SocketOutputStream.write(Unknown Source)
              at java.io.ByteArrayOutputStream.writeTo(Unknown Source)
              at net.sf.jml.net.Session.sendMessage(Session.java:511)
              at net.sf.jml.net.Session.access$1300(Session.java:30)
              at net.sf.jml.net.Session$MsgSender.run(Session.java:476)
              at java.lang.Thread.run(Unknown Source)

              • Guus der Kinderen KeyContributor
                Guus der Kinderen
                771 posts since
                Sep 8, 2005
                Currently Being Moderated
                Dec 30, 2009 1:11 PM (in response to ddiggler)
                Re: Openfire service crashing daily

                Would you mind loading a java-monitor plugin in your setup? That will give us more information, and will possibly allow you to anticipate a bit.

                 

                More information on java-monitor in this blogpost: New Openfire monitoring plugin

              • Guus der Kinderen KeyContributor
                Guus der Kinderen
                771 posts since
                Sep 8, 2005
                Currently Being Moderated
                Dec 30, 2009 1:22 PM (in response to ddiggler)
                Re: Openfire service crashing daily

                Alhough you're writing that you're using Java 1.6.0_17, your crashreport says otherwise:

                 

                Java HotSpot(TM) Server VM (1.6.0_03-b05) for windows-x86, built on Sep 24 2007 22:20:35 by "java_re" with unknown MS VC++:1310

                 

                03 is pretty old. It shouldn't hurt to update.

                • ddiggler Bronze
                  ddiggler
                  17 posts since
                  Nov 11, 2009
                  Currently Being Moderated
                  Dec 30, 2009 1:39 PM (in response to Guus der Kinderen)
                  Re: Openfire service crashing daily

                  Thanks for the quick reponse.

                   

                  This is a crash report from when we where running 1.6.0_03-b05.  We have updated since then thinking it may help solve our problems.  We are running

                  1.6.0_17 Sun Microsystems Inc. -- Java HotSpot(TM) Client VM.

                   

                  I will look into Java-Monitor.  In the mean time I have been running the JVMMonitor (http://www.igniterealtime.org/community/docs/DOC-1712).  I have been making copies of the graphs directories when we have problems.  I attached the graphs from when the system was running at 100% CPU and the admin console was not responding.  (We get the logon page but a "Page can't be displayed" upon logging in)

                  Attachments:
                • ddiggler Bronze
                  ddiggler
                  17 posts since
                  Nov 11, 2009
                  Currently Being Moderated
                  Dec 31, 2009 7:17 AM (in response to Guus der Kinderen)
                  Re: Openfire service crashing daily

                  It just crashed again.  This time the CPU didn't peg to 100% but the service just stopped.  I was in the console when it happened, memory was showing 78% of the 1280M for just over 900 users

                   

                  #
                  # A fatal error has been detected by the Java Runtime Environment:
                  #
                  # java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?
                  #
                  #  Internal Error (allocation.cpp:117), pid=1268, tid=1404
                  #  Error: ChunkPool::allocate
                  #
                  # JRE version: 6.0_17-b04
                  # Java VM: Java HotSpot(TM) Client VM (14.3-b01 mixed mode windows-x86 )
                  # If you would like to submit a bug report, please visit:
                  #   http://java.sun.com/webapps/bugreport/crash.jsp
                  #

                  ---------------  T H R E A D  ---------------

                  Current thread (0x56fd9800):  JavaThread "CompilerThread0" daemon [_thread_in_native, id=1404, stack(0x57100000,0x57150000)]

                  Stack: [0x57100000,0x57150000],  sp=0x5714f4b8,  free space=317k
                  Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
                  V  [jvm.dll+0x1e66b7]
                  V  [jvm.dll+0xa0c9c]
                  V  [jvm.dll+0x27e6]
                  V  [jvm.dll+0x29e2]
                  V  [jvm.dll+0x2be3]
                  V  [jvm.dll+0x52101]
                  V  [jvm.dll+0x523b8]
                  V  [jvm.dll+0x53ffc]
                  V  [jvm.dll+0x54ef6]
                  V  [jvm.dll+0x5513a]
                  V  [jvm.dll+0x26a67]
                  V  [jvm.dll+0x26b0a]
                  V  [jvm.dll+0x26c71]
                  V  [jvm.dll+0x26d84]
                  V  [jvm.dll+0x84df0]
                  V  [jvm.dll+0x858cf]
                  V  [jvm.dll+0x1d0414]
                  V  [jvm.dll+0x173e4c]
                  C  [MSVCR71.dll+0x9565]
                  C  [kernel32.dll+0x2482f]


                  Current CompileTask:
                  C1:4873 s!   org.jivesoftware.util.log.output.io.rotate.RotatingFileTarget.write(Ljava/lang/ String;)V (46 bytes)

                   


                  VM Arguments:
                  jvm_args: -Dexe4j.isInstall4j=true -Dexe4j.isService=true -Dexe4j.moduleName=C:\Program Files\Openfire\bin\openfire-service.exe -Dexe4j.processCommFile=C:\WINDOWS\TEMP\e4j_p1268.tmp -Dexe4j.tempDir= -Dexe4j.unextractedPosition=0 -Dexe4j.consoleCodepage=cp0 -Xrs -Xms1280m -Xmx1280m -XX:ThreadStackSize=128
                  java_command: <unknown>
                  Launcher Type: generic

                  Environment Variables:
                  PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\The Open Group\WMI Mapper\\bin;c:\program files\openfire\jre\bin
                  OS=Windows_NT
                  PROCESSOR_IDENTIFIER=x86 Family 6 Model 15 Stepping 8, GenuineIntel

                   

                  ---------------  S Y S T E M  ---------------

                  OS: Windows Server 2003 family Build 3790 Service Pack 2

                  CPU:total 2 (4 cores per cpu, 1 threads per core) family 6 model 15 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

                  Memory: 4k page, physical 2097151k(1158652k free), swap 4194303k(4194303k free)

                  vm_info: Java HotSpot(TM) Client VM (14.3-b01) for windows-x86 JRE (1.6.0_17-b04), built on Oct 11 2009 00:52:06 by "java_re" with MS VC++ 7.1

                  time: Thu Dec 31 10:04:08 2009
                  elapsed time: 84400 seconds

                  -----------------------------------------------------------Last couple error.log errors------------------------------------------

                   

                  2009.12.31 09:54:48 [org.jivesoftware.openfire.nio.ConnectionHandler.exceptionCaught(ConnectionHand ler.java:110)
                  ]
                  java.lang.OutOfMemoryError: unable to create new native thread
                  at java.lang.Thread.start0(Native Method)
                  at java.lang.Thread.start(Unknown Source)
                  at net.kano.joscar.flap.AsynchronousFlapProcessor.<init>(AsynchronousFlapProcessor .java:29)
                  at net.kano.joscar.flap.ClientFlapConn.init(ClientFlapConn.java:82)
                  at net.kano.joscar.flap.ClientFlapConn.<init>(ClientFlapConn.java:73)
                  at net.sf.kraken.protocols.oscar.AbstractFlapConnection.<init>(AbstractFlapConnect ion.java:102)
                  at net.sf.kraken.protocols.oscar.LoginConnection.<init>(LoginConnection.java:41)
                  at net.sf.kraken.protocols.oscar.OSCARSession.logIn(OSCARSession.java:120)
                  at net.sf.kraken.session.TransportSession.sessionDisconnected(TransportSession.jav a:523)
                  at net.sf.kraken.protocols.oscar.LoginConnection.handleStateChange(LoginConnection .java:65)
                  at net.sf.kraken.protocols.oscar.AbstractFlapConnection$1.stateChanged(AbstractFla pConnection.java:115)
                  at net.kano.joscar.net.ClientConn.setState(ClientConn.java:354)
                  at net.kano.joscar.net.ClientConn.connect(ClientConn.java:412)
                  at net.sf.kraken.protocols.oscar.OSCARSession.logIn(OSCARSession.java:124)
                  at net.sf.kraken.protocols.oscar.OSCARTransport.registrationLoggedIn(OSCARTranspor t.java:95)
                  at net.sf.kraken.BaseTransport.processPacket(BaseTransport.java:398)
                  at net.sf.kraken.BaseTransport.processPacket(BaseTransport.java:199)
                  at org.jivesoftware.openfire.component.InternalComponentManager$RoutableComponents .process(InternalComponentManager.java:619)
                  at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.jav a:260)
                  at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:164)
                  at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:70)
                  at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76)
                  at org.jivesoftware.openfire.net.StanzaHandler.processPresence(StanzaHandler.java: 337)
                  at org.jivesoftware.openfire.net.ClientStanzaHandler.processPresence(ClientStanzaH andler.java:85)
                  at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:254)
                  at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:176)
                  at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:133)
                  at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)
                  at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
                  at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
                  at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
                  at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)
                  at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
                  at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
                  at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
                  at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)
                  at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)
                  at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
                  at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
                  at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
                  at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)
                  at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
                  at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
                  at java.lang.Thread.run(Unknown Source)
                  2009.12.31 09:55:39 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:92)
                  ] error sending msg: PNG

                  java.net.SocketException: Software caused connection abort: socket write error
                  at java.net.SocketOutputStream.socketWrite0(Native Method)
                  at java.net.SocketOutputStream.socketWrite(Unknown Source)
                  at java.net.SocketOutputStream.write(Unknown Source)
                  at java.io.ByteArrayOutputStream.writeTo(Unknown Source)
                  at net.sf.jml.net.Session.sendMessage(Session.java:511)
                  at net.sf.jml.net.Session.access$1300(Session.java:30)
                  at net.sf.jml.net.Session$MsgSender.run(Session.java:476)
                  at java.lang.Thread.run(Unknown Source)

      • Guus der Kinderen KeyContributor
        Guus der Kinderen
        771 posts since
        Sep 8, 2005
        Currently Being Moderated
        Dec 31, 2009 11:29 AM (in response to ddiggler)
        Re: Openfire service crashing daily

        The reference to swap space surprised me. I didn't see that before. I've done a bit of googling, and found this:

        3.1.4 Detail Message: request <size> bytes for <reason>. Out of swap space?

        The detail message request <size> bytes for <reason>. Out of swap space? appears to be an OutOfMemoryError. However, the HotSpot VM code reports this apparent exception when an allocation from the native heap failed and the native heap might be close to exhaustion. The message indicates the size (in bytes) of the request that failed and the reason for the memory request. In most cases the <reason> part of the message is the name of a source module reporting the allocation failure, although in some cases it indicates a reason.

        When this error message is thrown, the VM invokes the fatal error handling mechanism, that is, it generates a fatal error log file, which contains useful information about the thread, process, and system at the time of the crash. In the case of native heap exhaustion, the heap memory and memory map information in the log can be useful. See Appendix C, Fatal Error Log for detailed information about this file.

        If this type of OutOfMemoryError is thrown, you might need to use troubleshooting utilities on the operating system to diagnose the issue further. See 2.16 Operating-System-Specific Tools.

        The problem might not be related to the application, for example:

        • The operating system is configured with insufficient swap space.

        • Another process on the system is consuming all memory resources.

        If neither of the above issues is the cause, then it is possible that the application failed due to a native leak, for example, if application or library code is continuously allocating memory but is not releasing it to the operating system.

        I've found this at http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/memleaks.html#gbyvj

         

        Did you rule out causes outside of Openfire?

        • ddiggler Bronze
          ddiggler
          17 posts since
          Nov 11, 2009
          Currently Being Moderated
          Jan 4, 2010 9:21 AM (in response to Guus der Kinderen)
          Re: Openfire service crashing daily

          It just crashed again today.  I attached the JVM-Monitor graphs.  These graphs are right after the service crashed but before rebooting or restarting the service.  This server is dedicated to running Openfire.  The only other software on this system is the antivirus software, which I disabled two weeks ago to exclude as a possible cause.  This is a VMware server running on ESX so the VMtools are installed.

           

          #
          # A fatal error has been detected by the Java Runtime Environment:
          #
          # java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?
          #
          #  Internal Error (allocation.cpp:117), pid=1256, tid=1412
          #  Error: ChunkPool::allocate
          #
          # JRE version: 6.0_17-b04
          # Java VM: Java HotSpot(TM) Client VM (14.3-b01 mixed mode windows-x86 )
          # If you would like to submit a bug report, please visit:
          #   http://java.sun.com/webapps/bugreport/crash.jsp
          #
          VM Arguments:
          jvm_args: -Dexe4j.isInstall4j=true -Dexe4j.isService=true -Dexe4j.moduleName=C:\Program Files\Openfire\bin\openfire-service.exe -Dexe4j.processCommFile=C:\WINDOWS\TEMP\e4j_p1256.tmp -Dexe4j.tempDir= -Dexe4j.unextractedPosition=0 -Dexe4j.consoleCodepage=cp0 -Xrs -Xms1280m -Xmx1280m -XX:ThreadStackSize=128
          java_command: <unknown>
          Launcher Type: generic

          Environment Variables:
          PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\The Open Group\WMI Mapper\\bin;c:\program files\openfire\jre\bin
          OS=Windows_NT
          PROCESSOR_IDENTIFIER=x86 Family 6 Model 15 Stepping 8, GenuineIntel

           

          ---------------  S Y S T E M  ---------------

          OS: Windows Server 2003 family Build 3790 Service Pack 2

          CPU:total 2 (4 cores per cpu, 1 threads per core) family 6 model 15 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

          Memory: 4k page, physical 2097151k(1238272k free), swap 4194303k(4194303k free)

          vm_info: Java HotSpot(TM) Client VM (14.3-b01) for windows-x86 JRE (1.6.0_17-b04), built on Oct 11 2009 00:52:06 by "java_re" with MS VC++ 7.1

          time: Mon Jan 04 11:38:05 2010
          elapsed time: 253258 seconds

          Attachments:
          • sixthring KeyContributor
            sixthring
            3,797 posts since
            Apr 2, 2007
            Currently Being Moderated
            Jan 4, 2010 9:25 AM (in response to ddiggler)
            Re: Openfire service crashing daily

            Since your hands are tied as to disabling the possible offiending software I do not know what help you possibly think we can provide.

          • Guus der Kinderen KeyContributor
            Guus der Kinderen
            771 posts since
            Sep 8, 2005
            Currently Being Moderated
            Jan 4, 2010 9:39 AM (in response to ddiggler)
            Re: Openfire service crashing daily

            Interesting - there's a substantial spike on all graphs on the end. Can you explain this with user patterns (did everyone come into the office) or could this indicate the point where your problem starts?

             

            It's hard to tell for sure (as there's not a full history worth of data) but it looks like the amount of threads is growing out of control (see thread1.jpg). I advise you to make regular threaddumps, and compare them.

             

            HeapMem.jpg shows that the JVM is using but a fraction of the memory that you've made available to it. You can turn this back a notch or two, if you want. You'd expect more of a jigsaw-like pattern in this graph, which would be just fine.

          • Guus der Kinderen KeyContributor
            Guus der Kinderen
            771 posts since
            Sep 8, 2005
            Currently Being Moderated
            Jan 4, 2010 10:19 AM (in response to ddiggler)
            Re: Openfire service crashing daily

            I've found this whitepaper, in which you might VMWare specific hints, tips and/or clues: http://www.vmware.com/resources/techresources/1087

          • LG KeyContributor
            LG
            6,132 posts since
            Dec 13, 2005
            Currently Being Moderated
            Jan 4, 2010 10:58 AM (in response to ddiggler)
            Re: Openfire service crashing daily

            Hi,

             

            looking at Stanza1.jpg I wonder whether the users are really exchanging 1500 presence packets per minute/second. I wonder whether this is an Openfire issue caused by one client. The client may be hard to identify, anyhow "Server Settings", "Message Audit Policy" allows one to audit presence packets to verify that these packets are fine.

             

            LG

          • ddiggler Bronze
            ddiggler
            17 posts since
            Nov 11, 2009
            Currently Being Moderated
            Jan 5, 2010 7:38 AM (in response to ddiggler)
            Re: Openfire service crashing daily

            I am posting more Java-Monitor graphs.  This seems a bit odd to me.  I noticed my heap memory usage drop to about 25% of what it was, the threads where cut in half and there was a spike in Garbage2.jpg.  When I first noticed this drop we had about 1270 sessions.  At the time of me gather the graphs we had 1360 sessions.   So the session count is rising but my heap memory usage just plummeted.

             

            We will also be rebuilding this system on a physical server as recommended.

             

            Thanks for everyones suggestions so far...

            Attachments:
            • LG KeyContributor
              LG
              6,132 posts since
              Dec 13, 2005
              Currently Being Moderated
              Jan 5, 2010 8:52 AM (in response to ddiggler)
              Re: Openfire service crashing daily

              Hi,

               

              I assume that you did restart your server at 2010.01.04-16:xx (y.m.d:H:M) and that at 2010.01.05-14:xx a "Full GC" did occur. Thread1.jpg shows that the number of "runnable" and "timed waiting" threads does not change while the number of "waiting" threads drops, likely caused by the "Full GC".


              Attaching VisualVM to your Openfire process allows you to get stacktraces and heapdumps, so you really may want to install it on a client machine and connect it to Openfire.

               

              LG

          • ddiggler Bronze
            ddiggler
            17 posts since
            Nov 11, 2009
            Currently Being Moderated
            Jan 5, 2010 11:57 AM (in response to ddiggler)
            Re: Openfire service crashing daily

            Just crashed again..  Posting new graphs.  I hope I am not overwhelming everyone with information.  I just don't want to miss anything that would be relevant.

             

            #
            # A fatal error has been detected by the Java Runtime Environment:
            #
            # java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?
            #
            #  Internal Error (allocation.cpp:117), pid=1296, tid=1376
            #  Error: ChunkPool::allocate
            #
            # JRE version: 6.0_17-b04
            # Java VM: Java HotSpot(TM) Client VM (14.3-b01 mixed mode windows-x86 )
            # If you would like to submit a bug report, please visit:
            #   http://java.sun.com/webapps/bugreport/crash.jsp
            #

            ---------------  T H R E A D  ---------------

            Current thread (0x56fc0800):  VMThread [stack: 0x57050000,0x570a0000] [id=1376]

            Stack: [0x57050000,0x570a0000],  sp=0x5709fc58,  free space=319k
            Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
            V  [jvm.dll+0x1e66b7]
            V  [jvm.dll+0xa0c9c]
            V  [jvm.dll+0x27e6]
            V  [jvm.dll+0x29e2]
            V  [jvm.dll+0x2be3]
            V  [jvm.dll+0x1935c3]
            V  [jvm.dll+0x1c7f3]
            V  [jvm.dll+0x1cc28]
            V  [jvm.dll+0x1ce41]
            V  [jvm.dll+0x1e852e]
            V  [jvm.dll+0x1e887c]
            V  [jvm.dll+0x1e8ca2]
            V  [jvm.dll+0x173e4c]
            C  [MSVCR71.dll+0x9565]
            C  [kernel32.dll+0x2482f]

            VM_Operation (0x5875f2ac): BulkRevokeBias, mode: safepoint, requested by thread 0x57d21800

            Attachments:
            • LG KeyContributor
              LG
              6,132 posts since
              Dec 13, 2005
              Currently Being Moderated
              Jan 5, 2010 12:24 PM (in response to ddiggler)
              Re: Openfire service crashing daily

              Hi,

               

              you may want to monitor the VM size of your process with "pslist -m 1234" where 1234 should be your Openfire pid. "pslist -t 1234" could also be interesting but I think that it is a VM memory problem.

               

              LG

            • Guus der Kinderen KeyContributor
              Guus der Kinderen
              771 posts since
              Sep 8, 2005
              Currently Being Moderated
              Jan 5, 2010 1:02 PM (in response to ddiggler)
              Re: Openfire service crashing daily

              The thread usage pattern is a bit odd, as others also mentioned. This might indicate a problem, but also might simply be a result of object that linger for an extreme long time, awaiting for finalization. As you assigned a lot more memory to Openfire than that it needs, it can take a long, long time before stuff is cleaned out.

               

              I'd be interested in the thread dumps, as I've mentioned earlier. That might give us a clue as to what all those 'extra' threads are about.

               

              For now, I'm still going with a problem that relates to the operating system itself. Any luck preparing a non-virtual host yet?

              • ddiggler Bronze
                ddiggler
                17 posts since
                Nov 11, 2009
                Currently Being Moderated
                Jan 5, 2010 1:35 PM (in response to Guus der Kinderen)
                Re: Openfire service crashing daily

                We are currently working on moving this to a physical server.

                 

                We had the memory set so high because we thought it was a lack of JVM memory that was causing our problems.  That just isn't the case.  We haven't adjusted it back yet.

                 

                I emailed you the complete dumps from when the service crashes.  They list all the threads at the time the service crashed.  Let me know if that is what you are looking for.

                • Guus der Kinderen KeyContributor
                  Guus der Kinderen
                  771 posts since
                  Sep 8, 2005
                  Currently Being Moderated
                  Jan 5, 2010 1:50 PM (in response to ddiggler)
                  Re: Openfire service crashing daily

                  As I replied to you by mail:

                   

                  This wasn't exactly what I'm looking for, but does give me some information. At first sight, there appear to be to much Timer threads, which appear to relate to the library that implements the Yahoo Messenger protocol. Any chance that you'd be able to run a couple of days with the Yahoo gateway disabled?

  • ddiggler Bronze
    ddiggler
    17 posts since
    Nov 11, 2009
    Currently Being Moderated
    Jan 13, 2010 8:37 AM (in response to ddiggler)
    Re: Openfire service crashing daily

    I will probably jinx myself by doing this...but we have had a stable system for 62 hours.  It normally wouldn't last more then 12-14 hours.

     

    We decided to keep the server running in a virtual server but rebuilt a new 64bit OS and 64bit JVM (using this executable http://www.igniterealtime.org/community/docs/DOC-1331 to have it use the 64bit JVM) .  We followed the VMware whitepaper that was referenced in this thread, making sure to set the memory reservation for the amount of RAM allocated to the virtual server.

     

    I have been monitoring the JVM-Monitor graphs (attached) and the usage pattern has changed significantly.   We still have been seeing spikes in the presencecount...but it seems to recovery.  The last spike lasted all night, during off hours, when we have the least connections, which is a little odd.

     

    If everything stays stable, I would like to thank everyone that contributed in helping me out.   If there are any comments on the current graphs, please feel free to share.

     

    Thanks

    Attachments:

More Like This

  • Retrieving data ...

Bookmarked By (0)

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points