I would suggest disabling some of the plugins you are running. They may be the cause of your issues. Then enable them one at a time. Or remove them one at a time. Start with removing client control.
Hi,
the thread stack size applies to every thread and 128k may be too small - you’ll get Java errors in your log file if this is the case. Xss should be the same as ThreadStackSize and thus for the native stack size while Xoss should set the java stack size for every thread. As you have a problem with the native thread count setting the ThreadStackSize should be enough.
You may want to create some stack traces while the server is running - there you may be able to see an increasing number of java threads (every java thread can be mapped to a native thread). Maybe the name helps you to identify which part of Openfire or plugins does start these threads.
LG
I added “-XX:ThreadStackSize=128” and set xmpp.pep.enabled=false (just incase). The service appears to have ran longer (may be due to the holiday) but we still are having problems. Today the service didn’t crash but you couldn’t get into the web client or the admin console. The server CPU was pegged at 100% and the following event was logged in the eventvwr.
Event Type: Error
Event Source: Srv
Event Category: None
Event ID: 2019
Date: 12/30/2009
Time: 9:08:59 AM
User: N/A
Computer:
Description:
The server was unable to allocate from the system nonpaged pool because the pool was empty.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 04 00 01 00 54 00 …T.
0008: 00 00 00 00 e3 07 00 c0 …ã…À
0010: 00 00 00 00 9a 00 00 c0 …�…À
0018: 00 00 00 00 00 00 00 00 …
0020: 00 00 00 00 00 00 00 00 …
0028: 08 00 00 00 …
I also noticed out Statistics page is not showing up…it’s there, but empty (http://www.igniterealtime.org/community/thread/36238 and http://www.igniterealtime.org/community/message/172876#172876). We are going to attempt to repair the ofRRDs table this weekend. Could this be the source of all my crashes?
Here is my error.log around that time…
2009.12.30 09:01:32 [org.jivesoftware.openfire.spi.PresenceManagerImpl.userUnavailable(PresenceMana gerImpl.java:271)
] Error storing offline presence of user:
java.sql.SQLException: Violation of PRIMARY KEY constraint ‘ofPresence_pk’. Cannot insert duplicate key in object ‘dbo.ofPresence’.
at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:368)
at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2816)
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2254)
at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:631)
at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:584)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:546)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.execute(JtdsPreparedStatement.j ava:559)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.logicalcobwebs.proxool.ProxyStatement.invoke(ProxyStatement.java:100)
at org.logicalcobwebs.proxool.ProxyStatement.intercept(ProxyStatement.java:57)
at $java.sql.Wrapper$$EnhancerByProxool$$b2f44fdd.execute()
at org.jivesoftware.openfire.spi.PresenceManagerImpl.userUnavailable(PresenceManag erImpl.java:268)
at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:145)
at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:112)
at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:176)
at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:134)
at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:70)
at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76)
at org.jivesoftware.openfire.SessionManager$ClientSessionListener.onConnectionClos e(SessionManager.java:1133)
at org.jivesoftware.openfire.nio.NIOConnection.notifyCloseListeners(NIOConnection. java:202)
at org.jivesoftware.openfire.nio.NIOConnection.close(NIOConnection.java:185)
at org.jivesoftware.openfire.nio.ConnectionHandler.sessionClosed(ConnectionHandler .java:87)
at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.sessionClosed(A bstractIoFilterChain.java:550)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
at org.apache.mina.common.IoFilterAdapter.sessionClosed(IoFilterAdapter.java:65)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
at org.apache.mina.filter.codec.ProtocolCodecFilter.sessionClosed(ProtocolCodecFil ter.java:259)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :249)
at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Unknown Source)
2009.12.30 09:12:51 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:92)
] error sending msg: REM 18 FL 55a97183-9ec9-47d1-960a-d6555bd5c2c7 f530dfb6-9a55-483b-8bbc-cc04b2669407
java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at java.io.ByteArrayOutputStream.writeTo(Unknown Source)
at net.sf.jml.net.Session.sendMessage(Session.java:511)
at net.sf.jml.net.Session.access$1300(Session.java:30)
at net.sf.jml.net.Session$MsgSender.run(Session.java:476)
at java.lang.Thread.run(Unknown Source)
2009.12.30 09:12:54 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:92)
] error sending msg: REM 19 FL 177032fc-a811-4c18-878d-c388d7041c91 f530dfb6-9a55-483b-8bbc-cc04b2669407
java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at java.io.ByteArrayOutputStream.writeTo(Unknown Source)
at net.sf.jml.net.Session.sendMessage(Session.java:511)
at net.sf.jml.net.Session.access$1300(Session.java:30)
at net.sf.jml.net.Session$MsgSender.run(Session.java:476)
at java.lang.Thread.run(Unknown Source)
2009.12.30 09:13:26 [org.jivesoftware.openfire.spi.PresenceManagerImpl.userUnavailable(PresenceMana gerImpl.java:271)
] Error storing offline presence of user:
java.sql.SQLException: Violation of PRIMARY KEY constraint ‘ofPresence_pk’. Cannot insert duplicate key in object ‘dbo.ofPresence’.
at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:368)
at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2816)
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2254)
at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:631)
at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:584)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:546)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.execute(JtdsPreparedStatement.j ava:559)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.logicalcobwebs.proxool.ProxyStatement.invoke(ProxyStatement.java:100)
at org.logicalcobwebs.proxool.ProxyStatement.intercept(ProxyStatement.java:57)
at $java.sql.Wrapper$$EnhancerByProxool$$b2f44fdd.execute()
at org.jivesoftware.openfire.spi.PresenceManagerImpl.userUnavailable(PresenceManag erImpl.java:268)
at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:145)
at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:112)
at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:176)
at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:134)
at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:70)
at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76)
at org.jivesoftware.openfire.SessionManager$ClientSessionListener.onConnectionClos e(SessionManager.java:1133)
at org.jivesoftware.openfire.nio.NIOConnection.notifyCloseListeners(NIOConnection. java:202)
at org.jivesoftware.openfire.nio.NIOConnection.close(NIOConnection.java:185)
at org.jivesoftware.openfire.nio.ConnectionHandler.sessionClosed(ConnectionHandler .java:87)
at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.sessionClosed(A bstractIoFilterChain.java:550)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
at org.apache.mina.common.IoFilterAdapter.sessionClosed(IoFilterAdapter.java:65)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
at org.apache.mina.filter.codec.ProtocolCodecFilter.sessionClosed(ProtocolCodecFil ter.java:259)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextSessionClosed(Abst ractIoFilterChain.java:269)
at org.apache.mina.common.support.AbstractIoFilterChain.access$800(AbstractIoFilte rChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.sessionClosed( AbstractIoFilterChain.java:633)
at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :249)
at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Unknown Source)
2009.12.30 09:15:06 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:92)
] error sending msg: ADC 14 FL C=86d9264b-67aa-468a-8c8d-d696cf3fa67a 5f088445-3fe7-43ca-a863-52b6990d659e
java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at java.io.ByteArrayOutputStream.writeTo(Unknown Source)
at net.sf.jml.net.Session.sendMessage(Session.java:511)
at net.sf.jml.net.Session.access$1300(Session.java:30)
at net.sf.jml.net.Session$MsgSender.run(Session.java:476)
at java.lang.Thread.run(Unknown Source)
Would you mind loading a java-monitor plugin in your setup? That will give us more information, and will possibly allow you to anticipate a bit.
More information on java-monitor in this blogpost: New Openfire monitoring plugin
Alhough you’re writing that you’re using Java 1.6.0_17, your crashreport says otherwise:
Java HotSpot™ Server VM (1.6.0_03-b05) for windows-x86, built on Sep 24 2007 22:20:35 by “java_re” with unknown MS VC++:1310
03 is pretty old. It shouldn’t hurt to update.
Thanks for the quick reponse.
This is a crash report from when we where running 1.6.0_03-b05. We have updated since then thinking it may help solve our problems. We are running
1.6.0_17 Sun Microsystems Inc. – Java HotSpot™ Client VM.
I will look into Java-Monitor. In the mean time I have been running the JVMMonitor (http://www.igniterealtime.org/community/docs/DOC-1712). I have been making copies of the graphs directories when we have problems. I attached the graphs from when the system was running at 100% CPU and the admin console was not responding. (We get the logon page but a “Page can’t be displayed” upon logging in)
Graphs.zip (32373 Bytes)
have you tried running with out any plugins enabled.
No. The business will not let me remove the Krakken, Monitoring, Red5 or Search plugins. I could get away with removing the Client Control and Import/Export plugins…maybe… Unfortunately I have to work with what I have.
It just crashed again. This time the CPU didn’t peg to 100% but the service just stopped. I was in the console when it happened, memory was showing 78% of the 1280M for just over 900 users
A fatal error has been detected by the Java Runtime Environment:
java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?
Internal Error (allocation.cpp:117), pid=1268, tid=1404
Error: ChunkPool::allocate
JRE version: 6.0_17-b04
Java VM: Java HotSpot™ Client VM (14.3-b01 mixed mode windows-x86 )
If you would like to submit a bug report, please visit:
http://java.sun.com/webapps/bugreport/crash.jsp
--------------- T H R E A D ---------------
Current thread (0x56fd9800): JavaThread “CompilerThread0” daemon [_thread_in_native, id=1404, stack(0x57100000,0x57150000)]
Stack: [0x57100000,0x57150000], sp=0x5714f4b8, free space=317k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [jvm.dll+0x1e66b7]
V [jvm.dll+0xa0c9c]
V [jvm.dll+0x27e6]
V [jvm.dll+0x29e2]
V [jvm.dll+0x2be3]
V [jvm.dll+0x52101]
V [jvm.dll+0x523b8]
V [jvm.dll+0x53ffc]
V [jvm.dll+0x54ef6]
V [jvm.dll+0x5513a]
V [jvm.dll+0x26a67]
V [jvm.dll+0x26b0a]
V [jvm.dll+0x26c71]
V [jvm.dll+0x26d84]
V [jvm.dll+0x84df0]
V [jvm.dll+0x858cf]
V [jvm.dll+0x1d0414]
V [jvm.dll+0x173e4c]
C [MSVCR71.dll+0x9565]
C [kernel32.dll+0x2482f]
Current CompileTask:
C1:4873 s! org.jivesoftware.util.log.output.io.rotate.RotatingFileTarget.write(Ljava/lang/ String;)V (46 bytes)
VM Arguments:
jvm_args: -Dexe4j.isInstall4j=true -Dexe4j.isService=true -Dexe4j.moduleName=C:\Program Files\Openfire\bin\openfire-service.exe -Dexe4j.processCommFile=C:\WINDOWS\TEMP\e4j_p1268.tmp -Dexe4j.tempDir= -Dexe4j.unextractedPosition=0 -Dexe4j.consoleCodepage=cp0 -Xrs -Xms1280m -Xmx1280m -XX:ThreadStackSize=128
java_command:
Launcher Type: generic
Environment Variables:
PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\The Open Group\WMI Mapper\bin;c:\program files\openfire\jre\bin
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 15 Stepping 8, GenuineIntel
--------------- S Y S T E M ---------------
OS: Windows Server 2003 family Build 3790 Service Pack 2
CPU:total 2 (4 cores per cpu, 1 threads per core) family 6 model 15 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
Memory: 4k page, physical 2097151k(1158652k free), swap 4194303k(4194303k free)
vm_info: Java HotSpot™ Client VM (14.3-b01) for windows-x86 JRE (1.6.0_17-b04), built on Oct 11 2009 00:52:06 by “java_re” with MS VC++ 7.1
time: Thu Dec 31 10:04:08 2009
elapsed time: 84400 seconds
-----------------------------------------------------------Last couple error.log errors------------------------------------------
2009.12.31 09:54:48 [org.jivesoftware.openfire.nio.ConnectionHandler.exceptionCaught(ConnectionHand ler.java:110)
]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at net.kano.joscar.flap.AsynchronousFlapProcessor.(AsynchronousFlapProcessor .java:29)
at net.kano.joscar.flap.ClientFlapConn.init(ClientFlapConn.java:82)
at net.kano.joscar.flap.ClientFlapConn.(ClientFlapConn.java:73)
at net.sf.kraken.protocols.oscar.AbstractFlapConnection.(AbstractFlapConnect ion.java:102)
at net.sf.kraken.protocols.oscar.LoginConnection.(LoginConnection.java:41)
at net.sf.kraken.protocols.oscar.OSCARSession.logIn(OSCARSession.java:120)
at net.sf.kraken.session.TransportSession.sessionDisconnected(TransportSession.jav a:523)
at net.sf.kraken.protocols.oscar.LoginConnection.handleStateChange(LoginConnection .java:65)
at net.sf.kraken.protocols.oscar.AbstractFlapConnection$1.stateChanged(AbstractFla pConnection.java:115)
at net.kano.joscar.net.ClientConn.setState(ClientConn.java:354)
at net.kano.joscar.net.ClientConn.connect(ClientConn.java:412)
at net.sf.kraken.protocols.oscar.OSCARSession.logIn(OSCARSession.java:124)
at net.sf.kraken.protocols.oscar.OSCARTransport.registrationLoggedIn(OSCARTranspor t.java:95)
at net.sf.kraken.BaseTransport.processPacket(BaseTransport.java:398)
at net.sf.kraken.BaseTransport.processPacket(BaseTransport.java:199)
at org.jivesoftware.openfire.component.InternalComponentManager$RoutableComponents .process(InternalComponentManager.java:619)
at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.jav a:260)
at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:164)
at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:70)
at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76)
at org.jivesoftware.openfire.net.StanzaHandler.processPresence(StanzaHandler.java: 337)
at org.jivesoftware.openfire.net.ClientStanzaHandler.processPresence(ClientStanzaH andler.java:85)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:254)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:176)
at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:133)
at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)
at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Unknown Source)
2009.12.31 09:55:39 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:92)
] error sending msg: PNG
java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at java.io.ByteArrayOutputStream.writeTo(Unknown Source)
at net.sf.jml.net.Session.sendMessage(Session.java:511)
at net.sf.jml.net.Session.access$1300(Session.java:30)
at net.sf.jml.net.Session$MsgSender.run(Session.java:476)
at java.lang.Thread.run(Unknown Source)
That is unfortunate as any one of those plugins could be causing your error. I am fairly certain that it is being cuased by a plugin.
I hear ya…but my hands are tied…believe me, I have tried talking them into it.
seems rather rediculus to me that they would rather keep having it crash than try to diagnose the issue with some very basic tests.
The reference to swap space surprised me. I didn’t see that before. I’ve done a bit of googling, and found this:
3.1.4 Detail Message:
request <size> bytes for <reason>. Out of swap space?The detail message
request <size> bytes for <reason>. Out of swap space?appears to be anOutOfMemoryError. However, the HotSpot VM code reports this apparent exception when an allocation from the native heap failed and the native heap might be close to exhaustion. The message indicates the size (in bytes) of the request that failed and the reason for the memory request. In most cases the<reason>part of the message is the name of a source module reporting the allocation failure, although in some cases it indicates a reason.
When this error message is thrown, the VM invokes the fatal error handling mechanism, that is, it generates a fatal error log file, which contains useful information about the thread, process, and system at the time of the crash. In the case of native heap exhaustion, the heap memory and memory map information in the log can be useful. See Appendix C, Fatal Error Log for detailed information about this file.
If this type of
OutOfMemoryErroris thrown, you might need to use troubleshooting utilities on the operating system to diagnose the issue further. See 2.16 Operating-System-Specific Tools.
The problem might not be related to the application, for example:
- The operating system is configured with insufficient swap space.
- Another process on the system is consuming all memory resources.
If neither of the above issues is the cause, then it is possible that the application failed due to a native leak, for example, if application or library code is continuously allocating memory but is not releasing it to the operating system.
I’ve found this at http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/memleaks.html#gbyvj
Did you rule out causes outside of Openfire?
It just crashed again today. I attached the JVM-Monitor graphs. These graphs are right after the service crashed but before rebooting or restarting the service. This server is dedicated to running Openfire. The only other software on this system is the antivirus software, which I disabled two weeks ago to exclude as a possible cause. This is a VMware server running on ESX so the VMtools are installed.
A fatal error has been detected by the Java Runtime Environment:
java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?
Internal Error (allocation.cpp:117), pid=1256, tid=1412
Error: ChunkPool::allocate
JRE version: 6.0_17-b04
Java VM: Java HotSpot™ Client VM (14.3-b01 mixed mode windows-x86 )
If you would like to submit a bug report, please visit:
http://java.sun.com/webapps/bugreport/crash.jsp
VM Arguments:
jvm_args: -Dexe4j.isInstall4j=true -Dexe4j.isService=true -Dexe4j.moduleName=C:\Program Files\Openfire\bin\openfire-service.exe -Dexe4j.processCommFile=C:\WINDOWS\TEMP\e4j_p1256.tmp -Dexe4j.tempDir= -Dexe4j.unextractedPosition=0 -Dexe4j.consoleCodepage=cp0 -Xrs -Xms1280m -Xmx1280m -XX:ThreadStackSize=128
java_command:
Launcher Type: generic
Environment Variables:
PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\The Open Group\WMI Mapper\bin;c:\program files\openfire\jre\bin
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 15 Stepping 8, GenuineIntel
--------------- S Y S T E M ---------------
OS: Windows Server 2003 family Build 3790 Service Pack 2
CPU:total 2 (4 cores per cpu, 1 threads per core) family 6 model 15 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
Memory: 4k page, physical 2097151k(1238272k free), swap 4194303k(4194303k free)
vm_info: Java HotSpot™ Client VM (14.3-b01) for windows-x86 JRE (1.6.0_17-b04), built on Oct 11 2009 00:52:06 by “java_re” with MS VC++ 7.1
time: Mon Jan 04 11:38:05 2010
elapsed time: 253258 seconds
JVM.zip (178119 Bytes)
Since your hands are tied as to disabling the possible offiending software I do not know what help you possibly think we can provide.
Interesting - there’s a substantial spike on all graphs on the end. Can you explain this with user patterns (did everyone come into the office) or could this indicate the point where your problem starts?
It’s hard to tell for sure (as there’s not a full history worth of data) but it looks like the amount of threads is growing out of control (see thread1.jpg). I advise you to make regular threaddumps, and compare them.
HeapMem.jpg shows that the JVM is using but a fraction of the memory that you’ve made available to it. You can turn this back a notch or two, if you want. You’d expect more of a jigsaw-like pattern in this graph, which would be just fine.
I’ve found this whitepaper, in which you might VMWare specific hints, tips and/or clues: http://www.vmware.com/resources/techresources/1087
Hi,
looking at Stanza1.jpg I wonder whether the users are really exchanging 1500 presence packets per minute/second. I wonder whether this is an Openfire issue caused by one client. The client may be hard to identify, anyhow “Server Settings”, “Message Audit Policy” allows one to audit presence packets to verify that these packets are fine.
LG
One of my co-workers that works with VMWare in a professional capacity tells me that Java and VMWare do play nice, as long as you follow the guidelines in that whitepaper I already linked.
Thanks for the whitepaper. Most of the guidelines are VMware best practice, which we’ve been following. The only thing I noticed that was odd is they recommend setting a memory reservation for the amount of RAM that is allocated to the VM. This is something we will need to try.