Urgent: Issue with binding continuation loop?

Hi,

We are having a serious issue with http binding on our openfire server.

Generally there are about 20 people connected at any time to our openfire server and about 10% of these are httpbind connections.

The CPU generally sits at about 1% and the server certainly isn’t busy.

The problem is that after about 1 or 2 days the CPU jumps to 90% or even 100% suddenly - mostly stopping openfire from working.

The issue is clearly visible when enabling the Debug log which generates thousands of continuation messages every second.

2007.12.19 09:51:19 JettyLog: continuation {}

2007.12.19 09:51:19 JettyLog: continuation {}

2007.12.19 09:51:19 JettyLog: continuation {}

2007.12.19 09:51:19 JettyLog: continuation {}

2007.12.19 09:51:19 JettyLog: continuation {}

2007.12.19 09:51:19 JettyLog: continuation {}

2007.12.19 09:51:19 JettyLog: continuation {}

2007.12.19 09:51:19 JettyLog: continuation {}

2007.12.19 09:51:19 JettyLog: continuation {}

2007.12.19 09:51:19 JettyLog: continuation {}

2007.12.19 09:51:19 JettyLog: continuation {}

2007.12.19 09:51:19 JettyLog: continuation {}

If I go in and disable http bind and then renable it in the administrator then the CPU immediately drops again.

We are using the latest version of Openfire - but the issue has existed through all previous versions. We are running on Windows Server 2003 and using ISAPI rewrite to do the http binding port redirection.

There are no error or warning messages logged when this problem begins.

Let me know if you require more information.

Thanks

Daniel

We will look into upgrading to the latest jetty version, hopefully that will resolve the issue.

Alex

What would be timing on that? Is there any further information I can provide so you can determine the specific problem and have you seen it happen before?

Thanks for getting back to me so quickly,

Daniel

I am unable to see this issue listed on your recently closed or outstanding issues.

Have you had other customers experience a similar problem and could you think of any work arounds.

Unfortunately this is having a major impact on a product that we just released and so I am keen to get it resolved.

Thanks

Daniel

I have updated to the most recent version of openfire (which includes a Jetty update) but unfortunately it hasn’t helped the problem.

We are really desparate to solve this issue as Jabber is going up to 100% after about 30 minutes now our site is more busy. If we disable and then re-enable the http binding service in Openfire then the CPU drops right back down again to 0% (and disconnects all the httpbind connections). We have between 5 and 20 users connecting via http-bind when the problem occurs.

Any help would really be really appreciated.

Thanks

Daniel

Daniel,

Could you post some thread dumps from your instance when the cpu is topped out? I would like to see what it is doing.

Thanks,

Alex

We are using Windows server so I am unsure how to get a thread dump?

How can I run Openfire in a console?

Thanks


Seems I found some third party software called StackTrace that can do it. I will send you the dump when it crashes again.

Thanks

THREAD DUMP… The server is up to 80% CPU now. Let me know if you need more details.


Here is the correct dump…

1836 openfired.exe session:2 threads:72 parent:3472

2008-01-22 17:03:39

Full thread dump Java HotSpot™ Server VM (1.6.0_03-b05 mixed mode):

“StackTrace Remote Thread” prio=6 tid=0x29093400 nid=0x374 waiting on condition http://0x00000000…0x2b69fa80

java.lang.Thread.State: RUNNABLE

“btpool2-49” prio=6 tid=0x29248400 nid=0x8dc in Object.wait() http://0x2b44f000…0x2b44fd18

java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:490)

  • locked <0x085a8910> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

“btpool2-48” prio=6 tid=0x2a928400 nid=0xfa8 in Object.wait() http://0x2b5af000…0x2b5afd98

java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:490)

  • locked <0x08547a48> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

“btpool2-47” prio=6 tid=0x2852fc00 nid=0xb64 in Object.wait() http://0x2b3ef000…0x2b3efa18

java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:490)

  • locked <0x085475d0> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

“btpool2-44” prio=6 tid=0x28d00c00 nid=0xde8 runnable http://0x29c1f000…0x29c1fb98

java.lang.Thread.State: RUNNABLE

at org.mortbay.io.nio.SelectChannelEndPoint.updateKey(SelectChannelEndPoint.java:3 05)

  • locked <0x08541c98> (a org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint)

at org.mortbay.io.nio.SelectChannelEndPoint.undispatch(SelectChannelEndPoint.java: 180)

  • locked <0x08541c98> (a org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint)

at org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint.undispatch(Selec tChannelConnector.java:361)

at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:421)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:450)

“btpool2-42” prio=6 tid=0x28449800 nid=0xc30 runnable http://0x298ff000…0x298ffc98

java.lang.Thread.State: RUNNABLE

at org.mortbay.io.nio.SelectChannelEndPoint.updateKey(SelectChannelEndPoint.java:3 05)

  • locked <0x084fa7e8> (a org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint)

at org.mortbay.io.nio.SelectChannelEndPoint.undispatch(SelectChannelEndPoint.java: 180)

  • locked <0x084fa7e8> (a org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint)

at org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint.undispatch(Selec tChannelConnector.java:361)

at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:421)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:450)

“btpool0-21” prio=6 tid=0x2a547400 nid=0x958 in Object.wait() http://0x2b39f000…0x2b39fd98

java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:490)

  • locked <0x084d1a50> (a org.mortbay.thread.BoundedThreadPool$PoolThread)

“Timer-3” daemon prio=6 tid=0x2a3ce800 nid=0xf74 in Object.wait() http://0x298af000…0x298afa98

java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.util.TimerThread.mainLoop(Unknown Source)

  • locked <0x08131050> (a java.util.TaskQueue)

at java.util.TimerThread.run(Unknown Source)

“Timer-2” daemon prio=6 tid=0x28d56400 nid=0x8d0 in Object.wait() http://0x296cf000…0x296cfb18

java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.lang.Object.wait(Object.java:485)

at java.util.TimerThread.mainLoop(Unknown Source)

  • locked <0x0811b000> (a java.util.TaskQueue)

at java.util.TimerThread.run(Unknown Source)

“btpool2-0 - Acceptor0 SelectChannelConnector@0.0.0.0:8085” prio=6 tid=0x290fd800 nid=0xf38 runnable http://0x2967f000…0x2967fb98

java.lang.Thread.State: RUNNABLE

at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)

at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)

at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)

  • locked <0x08131c58> (a sun.nio.ch.Util$1)

  • locked <0x08131c48> (a java.util.Collections$UnmodifiableSet)

  • locked <0x08131878> (a sun.nio.ch.WindowsSelectorImpl)

at sun.nio.ch.SelectorImpl.select(Unknown Source)

at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:406)

at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:166)

at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java :124)

at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:515)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:450)

“pool-8-thread-17” prio=6 tid=0x28cf9800 nid=0xb40 waiting on condition http://0x2a34f000…0x2a34fb98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d853f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“pool-8-thread-16” prio=6 tid=0x28cf9400 nid=0xac0 waiting on condition http://0x2a2ff000…0x2a2ffc18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d853f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“pool-8-thread-15” prio=6 tid=0x283d8c00 nid=0x15c waiting on condition http://0x2a2af000…0x2a2afc98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d853f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“pool-8-thread-14” prio=6 tid=0x291acc00 nid=0x664 waiting on condition http://0x2a25f000…0x2a25fd18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d853f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“pool-8-thread-13” prio=6 tid=0x290a1400 nid=0x2d0 waiting on condition http://0x2a20f000…0x2a20fd98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d853f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“pool-8-thread-12” prio=6 tid=0x28e27c00 nid=0x344 waiting on condition http://0x2a1bf000…0x2a1bfa18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d853f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“pool-8-thread-11” prio=6 tid=0x284eb400 nid=0xab4 waiting on condition http://0x2a16f000…0x2a16fa98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d853f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“pool-8-thread-10” prio=6 tid=0x2858e800 nid=0x3e4 waiting on condition http://0x2a11f000…0x2a11fb18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d853f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“pool-8-thread-9” prio=6 tid=0x29068c00 nid=0x5fc waiting on condition http://0x2a0cf000…0x2a0cfb98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d853f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“pool-8-thread-8” prio=6 tid=0x29132c00 nid=0x63c waiting on condition http://0x2a07f000…0x2a07fc18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d853f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“pool-8-thread-7” prio=6 tid=0x28de6400 nid=0x410 waiting on condition http://0x2a02f000…0x2a02fc98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d853f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“pool-8-thread-6” prio=6 tid=0x290c1400 nid=0x99c waiting on condition http://0x29fdf000…0x29fdfd18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d853f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“pool-8-thread-5” prio=6 tid=0x28d66c00 nid=0x7f4 waiting on condition http://0x29f8f000…0x29f8fd98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d853f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“pool-8-thread-4” prio=6 tid=0x28d66400 nid=0xf24 waiting on condition http://0x29f3f000…0x29f3fa18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d853f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“pool-8-thread-3” prio=6 tid=0x28f2ac00 nid=0xdd0 waiting on condition http://0x29eef000…0x29eefa98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d853f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“pool-8-thread-2” prio=6 tid=0x28f2a800 nid=0xf18 waiting on condition http://0x29e9f000…0x29e9fb18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d853f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“pool-8-thread-1” prio=6 tid=0x28521800 nid=0xb1c waiting on condition http://0x29e4f000…0x29e4fb98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d853f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“client-17” daemon prio=6 tid=0x28521400 nid=0x2a0 waiting on condition http://0x29dff000…0x29dffc18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07b157a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

“client-16” daemon prio=6 tid=0x27dff800 nid=0xd8c waiting on condition http://0x29daf000…0x29dafc98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07b157a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

“client-15” daemon prio=6 tid=0x28465400 nid=0x2b8 waiting on condition http://0x29d5f000…0x29d5fd18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07b157a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

“client-14” daemon prio=6 tid=0x2844d800 nid=0xc38 waiting on condition http://0x29d0f000…0x29d0fd98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07b157a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

“client-13” daemon prio=6 tid=0x28685800 nid=0xefc waiting on condition http://0x29cbf000…0x29cbfa18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07b157a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

“client-12” daemon prio=6 tid=0x27e0e000 nid=0x26c waiting on condition http://0x29c6f000…0x29c6fb18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07b157a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

“client-11” daemon prio=6 tid=0x28ebd800 nid=0xc1c waiting on condition http://0x29bcf000…0x29bcfb98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07b157a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

“client-10” daemon prio=6 tid=0x290c3400 nid=0x874 waiting on condition http://0x29b7f000…0x29b7fc18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07b157a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

“client-9” daemon prio=6 tid=0x28c72c00 nid=0xd10 waiting on condition http://0x29b2f000…0x29b2fc98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07b157a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

“client-8” daemon prio=6 tid=0x28581400 nid=0x8fc waiting on condition http://0x29adf000…0x29adfd18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07b157a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

“client-7” daemon prio=6 tid=0x28c9ac00 nid=0x2f8 waiting on condition http://0x29a8f000…0x29a8fd98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07b157a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

“client-6” daemon prio=6 tid=0x286b7400 nid=0xc14 waiting on condition http://0x29a3f000…0x29a3fa18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07b157a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

“client-5” daemon prio=6 tid=0x28673c00 nid=0xb90 waiting on condition http://0x299ef000…0x299efa98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07b157a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

“client-4” daemon prio=6 tid=0x2878fc00 nid=0x404 waiting on condition http://0x2994f000…0x2994fb98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07b157a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

“client-3” daemon prio=6 tid=0x28e02c00 nid=0x6cc waiting on condition http://0x2985f000…0x2985fd98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07b157a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

“client-2” daemon prio=6 tid=0x28701400 nid=0xeb0 waiting on condition http://0x2980f000…0x2980fa18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07b157a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

“SocketAcceptorIoProcessor-0.1” prio=6 tid=0x28d47800 nid=0x838 runnable http://0x297bf000…0x297bfa98

java.lang.Thread.State: RUNNABLE

at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)

at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)

at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)

  • locked <0x07d6e328> (a sun.nio.ch.Util$1)

  • locked <0x07d6e318> (a java.util.Collections$UnmodifiableSet)

  • locked <0x07d697d8> (a sun.nio.ch.WindowsSelectorImpl)

at sun.nio.ch.SelectorImpl.select(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProce ssor.java:480)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“client-1” daemon prio=6 tid=0x28d47400 nid=0xc50 waiting on condition http://0x2976f000…0x2976fb18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07b157a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.lang.Thread.run(Unknown Source)

“SocketAcceptorIoProcessor-0.0” prio=6 tid=0x284d3c00 nid=0xf20 runnable http://0x2971f000…0x2971fb98

java.lang.Thread.State: RUNNABLE

at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)

at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)

at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)

  • locked <0x07d6e590> (a sun.nio.ch.Util$1)

  • locked <0x07d6e580> (a java.util.Collections$UnmodifiableSet)

  • locked <0x07d6d298> (a sun.nio.ch.WindowsSelectorImpl)

at sun.nio.ch.SelectorImpl.select(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProce ssor.java:480)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“SocketAcceptor-0” prio=6 tid=0x28edb400 nid=0xe4c runnable http://0x2962f000…0x2962fd18

java.lang.Thread.State: RUNNABLE

at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)

at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)

at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)

  • locked <0x07d88338> (a sun.nio.ch.Util$1)

  • locked <0x07d88328> (a java.util.Collections$UnmodifiableSet)

  • locked <0x07d880c0> (a sun.nio.ch.WindowsSelectorImpl)

at sun.nio.ch.SelectorImpl.select(Unknown Source)

at sun.nio.ch.SelectorImpl.select(Unknown Source)

at org.apache.mina.transport.socket.nio.SocketAcceptor$Worker.run(SocketAcceptor.j ava:220)

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“Socket Listener at port 5269” daemon prio=10 tid=0x28d65800 nid=0xa34 runnable http://0x295df000…0x295dfd98

java.lang.Thread.State: RUNNABLE

at java.net.PlainSocketImpl.socketAccept(Native Method)

at java.net.PlainSocketImpl.accept(Unknown Source)

  • locked <0x07b16240> (a java.net.SocksSocketImpl)

at java.net.ServerSocket.implAccept(Unknown Source)

at java.net.ServerSocket.accept(Unknown Source)

at org.jivesoftware.openfire.net.BlockingAcceptingMode.run(BlockingAcceptingMode.j ava:44)

at org.jivesoftware.openfire.net.SocketAcceptThread.run(SocketAcceptThread.java:89 )

“pool-openfire1” daemon prio=6 tid=0x28ebd400 nid=0xb00 waiting on condition http://0x2958f000…0x2958fa18

java.lang.Thread.State: TIMED_WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07bd6218> (a java.util.concurrent.SynchronousQueue$TransferStack)

at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)

at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)

at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)

at java.util.concurrent.SynchronousQueue.poll(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“Timer-0” daemon prio=6 tid=0x28f5ec00 nid=0xe3c in Object.wait() http://0x2953f000…0x2953fa98

java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.util.TimerThread.mainLoop(Unknown Source)

  • locked <0x07d7adb8> (a java.util.TaskQueue)

at java.util.TimerThread.run(Unknown Source)

“btpool0-0 - Acceptor0 SelectChannelConnector@0.0.0.0:9090” prio=6 tid=0x28782400 nid=0x160 runnable http://0x294ef000…0x294efb18

java.lang.Thread.State: RUNNABLE

at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)

at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)

at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)

  • locked <0x07d8c700> (a sun.nio.ch.Util$1)

  • locked <0x07d8c6f0> (a java.util.Collections$UnmodifiableSet)

  • locked <0x07d8bf78> (a sun.nio.ch.WindowsSelectorImpl)

at sun.nio.ch.SelectorImpl.select(Unknown Source)

at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:406)

at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:166)

at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java :124)

at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:515)

at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:450)

“pool-7-thread-1” prio=6 tid=0x28651000 nid=0xe00 waiting on condition http://0x2949f000…0x2949fb98

java.lang.Thread.State: TIMED_WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07d164b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNano s(Unknown Source)

at java.util.concurrent.DelayQueue.take(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“SocketSendingTracker” daemon prio=6 tid=0x2849c800 nid=0xc0c in Object.wait() http://0x2944f000…0x2944fc18

java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at org.jivesoftware.openfire.net.SocketSendingTracker$1.run(SocketSendingTracker.j ava:64)

  • locked <0x07d67318> (a org.jivesoftware.openfire.net.SocketSendingTracker$1)

“Flash Cross Domain” prio=6 tid=0x281a2800 nid=0xba0 runnable http://0x28baf000…0x28bafc98

java.lang.Thread.State: RUNNABLE

at java.net.PlainSocketImpl.socketAccept(Native Method)

at java.net.PlainSocketImpl.accept(Unknown Source)

  • locked <0x07b17318> (a java.net.SocksSocketImpl)

at java.net.ServerSocket.implAccept(Unknown Source)

at java.net.ServerSocket.accept(Unknown Source)

at org.jivesoftware.openfire.FlashCrossDomainHandler.startServer(FlashCrossDomainH andler.java:90)

at org.jivesoftware.openfire.FlashCrossDomainHandler.access$000(FlashCrossDomainHa ndler.java:24)

at org.jivesoftware.openfire.FlashCrossDomainHandler$1.run(FlashCrossDomainHandler .java:42)

at java.lang.Thread.run(Unknown Source)

“Update Manager” daemon prio=6 tid=0x2846c800 nid=0xb08 waiting on condition http://0x28b5f000…0x28b5fd18

java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.jivesoftware.openfire.update.UpdateManager$1.waitForNextCheck(UpdateManager .java:153)

at org.jivesoftware.openfire.update.UpdateManager$1.run(UpdateManager.java:106)

“pool-1-thread-1” prio=6 tid=0x28485800 nid=0x3f4 runnable http://0x28b0f000…0x28b0fd98

java.lang.Thread.State: RUNNABLE

at java.net.PlainSocketImpl.socketAccept(Native Method)

at java.net.PlainSocketImpl.accept(Unknown Source)

  • locked <0x07b1f0c8> (a java.net.SocksSocketImpl)

at java.net.ServerSocket.implAccept(Unknown Source)

at java.net.ServerSocket.accept(Unknown Source)

at org.jivesoftware.openfire.filetransfer.proxy.ProxyConnectionManager$1.run(Proxy ConnectionManager.java:97)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

“timer-openfire” daemon prio=6 tid=0x285e0c00 nid=0xbf8 in Object.wait() http://0x28abf000…0x28abfa98

java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.util.TimerThread.mainLoop(Unknown Source)

  • locked <0x07bd5ef0> (a java.util.TaskQueue)

at java.util.TimerThread.run(Unknown Source)

“Auditor” prio=6 tid=0x2848cc00 nid=0xa38 in Object.wait() http://0x28a6f000…0x28a6fb18

java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.util.TimerThread.mainLoop(Unknown Source)

  • locked <0x07bd6a68> (a java.util.TaskQueue)

at java.util.TimerThread.run(Unknown Source)

“ClusterManager events dispatcher” daemon prio=6 tid=0x28652400 nid=0xa14 waiting on condition http://0x28a1f000…0x28a1fb98

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07bd7068> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at org.jivesoftware.openfire.cluster.ClusterManager$1.run(ClusterManager.java:44)

“PubSub maintenance” prio=6 tid=0x281f5400 nid=0xaac in Object.wait() http://0x289cf000…0x289cfc18

java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.util.TimerThread.mainLoop(Unknown Source)

  • locked <0x07c3bae0> (a java.util.TaskQueue)

at java.util.TimerThread.run(Unknown Source)

“MUC cleanup” prio=6 tid=0x28391800 nid=0x250 in Object.wait() http://0x2897f000…0x2897fc98

java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.util.TimerThread.mainLoop(Unknown Source)

  • locked <0x07bb4f98> (a java.util.TaskQueue)

at java.util.TimerThread.run(Unknown Source)

"PEP avaiable sessions handler " daemon prio=6 tid=0x2861c400 nid=0xbb8 waiting on condition http://0x2892f000…0x2892fd18

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x07c3bdf8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(Unknown Source)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unk nown Source)

at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)

at org.jivesoftware.openfire.pep.IQPEPHandler$1.run(IQPEPHandler.java:123)

“Thread-0” daemon prio=6 tid=0x283a2c00 nid=0x694 waiting on condition http://0x288df000…0x288dfd98

java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.jivesoftware.database.ConnectionPool.run(ConnectionPool.java:374)

at java.lang.Thread.run(Unknown Source)

“Exe4JStartupThread” daemon prio=6 tid=0x28047400 nid=0xa64 waiting on condition http://0x2838f000…0x2838fa18

java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at com.exe4j.Controller$StartupThread.run(Unknown Source)

“Low Memory Detector” daemon prio=6 tid=0x28002800 nid=0xd78 runnable http://0x00000000…0x00000000

java.lang.Thread.State: RUNNABLE

“CompilerThread1” daemon prio=10 tid=0x28000c00 nid=0xed0 waiting on condition http://0x00000000…0x2829f408

java.lang.Thread.State: RUNNABLE

“CompilerThread0” daemon prio=10 tid=0x27e6ec00 nid=0x688 waiting on condition http://0x00000000…0x2824f488

java.lang.Thread.State: RUNNABLE

“Attach Listener” daemon prio=10 tid=0x27e6b000 nid=0x730 runnable http://0x00000000…0x00000000

java.lang.Thread.State: RUNNABLE

“Signal Dispatcher” daemon prio=10 tid=0x27e6a000 nid=0xfc0 runnable http://0x00000000…0x00000000

java.lang.Thread.State: RUNNABLE

“Finalizer” daemon prio=8 tid=0x27e5c400 nid=0xa1c in Object.wait() http://0x27f5f000…0x27f5fd98

java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.lang.ref.ReferenceQueue.remove(Unknown Source)

  • locked <0x07ad7c28> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(Unknown Source)

at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

“Reference Handler” daemon prio=10 tid=0x27e58800 nid=0xc58 in Object.wait() http://0x27f0f000…0x27f0fa18

java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at java.lang.Object.wait(Object.java:485)

at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)

  • locked <0x07ad8138> (a java.lang.ref.Reference$Lock)

“main” prio=6 tid=0x003c6c00 nid=0xf0 waiting on condition http://0x00000000…0x0007dc48

java.lang.Thread.State: RUNNABLE

“VM Thread” prio=10 tid=0x27e56000 nid=0x630 runnable

“GC task thread#0 (ParallelGC)” prio=6 tid=0x003cc800 nid=0x7ec runnable

“GC task thread#1 (ParallelGC)” prio=6 tid=0x003cd800 nid=0xba8 runnable

“VM Periodic Task Thread” prio=10 tid=0x28004000 nid=0xbf0 waiting on condition

We recently changed hosting environments, and installed Openfire on a new server. The problem still persists however.

Any ideas? Thank you.

Interestingly the issue looks exactly like this one: http://jira.codehaus.org/browse/JETTY-256

But that has been resolved in the java/jetty version you are using?

Can you take a few thread dumps, a few seconds apart and then diff the results so we can see the active threads

This is not a log message from the jetty source code?

It is not clear that this is a jetty bug, as it could simply be a mis-use of continuations?

The fact that it can be turned off suggests that it is not a hard bug within Jetty. More information is needed from multiple thread dumps to see where the looping is happening.

Hi Greg - thank you for following up on this thread. I will get that stack dump diff for you asap - hopefully tomorrow.

I just wanted to also mention that we reduced the maximum wait time on the binding client (JSJAC) down to 20 seconds (it was at 30) and it has helped the problem but not entirely. Instead of this problem occuring every 10 mins or so it is now occuring about once ot twice a day.

All the openfire binding system parameters are set to default. There are no specific errors or warnings generated before the problem begins.

I have attached various dumps at the time it is crashing.

When comparing the difference 95% of the dump is exactly the same but maybe you can see something I can’t.

Thanks for looking into this.

I am also seeing this, although a slightly different setup:

Windows WebServer 2008 redirecting with ISAPI filter to Openfire running on separate CentOS machine

I wonder if any solution was found to this issue?

Or if I can help by sending some diagnostic data - please tell me what to do!

Max.

Unfortunately we couldn’t resolve the problem but reduced it somewhat by speeding up the binding timeout so that the http threads expire quicker.

The server still jumps to 90%+ CPU everyday though!

Interestingly - what ISAPI Rewriter are you using? And how often are you experiencing the problem?

Daniel

I am using the Helicon ISAPI rewrite_3. Are you thinking the ISAPI bit has something to do with this? Could be a cache or timeout issue with the re-writer perhaps? - Can you clear your problem just by resetting your isapi filter, rather than resetting openfire? I guess that is not the most scientific of tests… hmm… Considering we have a separate server doing the re-directs, I will investigate the cpu load on the webserver to see if the isapi process is also going slow.

To be fair, I only noticed yesterday the masses of “JettyLog: continuation{}” entries in the debug log (as you described)… (pouring in every second that the debug log is enabled)

Last night I collected some stats, and according to “sysstat”, on the Linux server, the cpu is having jobs not clearing out even after 15 mins - I get a load-average report after 1 mins, 5 mins and 15 mins. They all show 1.73. I would expect something like that for the 1 mins column, but it really should be getting down to zero in 5 and 15 mins.

I am fairly sure our bind setting is currently for 20 mins, but I am going to try reducing that to 12 mins, so I can see if the cpu report tomorrow shows this savings in the 15 mins column. If that works, then I will try again at 4 mins !

I have not yet read the specs, but I am hoping that the timeout tick is reset even by a presence sent from the server to the web client. And in this case, provided I send a presence to my web clients once every 2 mins, then the timeout could even be set to 3 mins, and thus keep things as trim as possible.

Anyhow I will post some feedback tomorrow.

Well the cpu load at the isapi machine is basically nothing. It would appear on first glance that the isapi re-direct does not add any (significant) overhead.

Altering the timeout did not make any difference in our case, although as I have read the timeout is only relvant for closed or dormant connections, and I have tested and experienced that whilst connected (and receiving regular presence updates) the session would remain open.

So to conclude something… it might be that the load is just what might be expected considering the clients are effectively polling the server every second, and the debug log is just a red herring, where perhaps the JettyLog debug entries do not scale well.

Although this does not completely explain the processes that are not clearing in a timely fashion, but at our current levels of usage it is not an issue.

  • Daniel, I can very well see where this would lead to with some 100+ users though…

I will keep monitoring it over the next week, but I suspect we might just look to implement a custom openfire plugin and webcontrol, perhaps using JSON, to try and eliminate the isapi redirect and per-second polling altogether.

Max.

Has anyone found a work around or resolved this issue? I am currently being plagued by this problem running Openfire 3.6.0a. I’ve experienced it running the server on both CentOS 5.2 and Mac OS 10.5.5.

I’m using JSJac library to talk to Openfire with the following http-bind server settings:

xmpp.httpbind.client.requests.polling = 0

xmpp.httpbind.client.requests.wait = 60

Based on previous posts talking about proxying, I’ve tried two different ways of proxying the AJAX browser calls. One approach is a standard Apache proxy, the other is a Java servlet proxy, both show no difference in preventing the continuation loop.

The continuation loop generally occurs under a load of no more than 10 users anywhere from 1 - 6 hours of server uptime.

Any help would be greatly appreciated.

  • Aaron

Has anyone fixed this issue?

We also have same problem… sort of RetryContinuation loop.

Usually it happens with 10~15 concurrent users.

Version : 3.2.0

Client : JsJac Javascript library

Looking into this forum, I found an interesting post.

http://www.igniterealtime.org/community/thread/37669

I’m not sure that relates this issue.

Can someone try this?