Openfire 3.10.0 Beta - NIOConnection Thread Deadlock

In testing the latest 3.10.0 Beta from the Github repository, a thread deadlock issue has been encountered.

The issue apparently occurs when two users, each with the other on their Roster, disconnect from the server at the same time.

Below is the relevant thread dump output for the deadlock situation:


“C2S-Thread-2”:

at org.jivesoftware.openfire.nio.NIOConnection.isClosed(NIOConnection.java:249)

  • waiting to lock <0x0000000094e3e540> (a org.jivesoftware.openfire.nio.NIOConnection)

at org.jivesoftware.openfire.nio.NIOConnection.deliver(NIOConnection.java:257)

at org.jivesoftware.openfire.session.LocalClientSession.deliver(LocalClientSession .java:857)

at org.jivesoftware.openfire.session.LocalSession.process(LocalSession.java:289)

at org.jivesoftware.openfire.spi.RoutingTableImpl.routeToLocalDomain(RoutingTableI mpl.java:354)

at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.jav a:239)

at org.jivesoftware.openfire.roster.Roster.broadcastPresence(Roster.java:609)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.broadcastUpdate(Presenc eUpdateHandler.java:308)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:162)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:137)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:201)

at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:148)

at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:84)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:84)

at org.jivesoftware.openfire.SessionManager$ClientSessionListener.onConnectionClos e(SessionManager.java:1183)

at org.jivesoftware.openfire.nio.NIOConnection.notifyCloseListeners(NIOConnection. java:237)

at org.jivesoftware.openfire.nio.NIOConnection.close(NIOConnection.java:219)

  • locked <0x0000000094e47a98> (a org.jivesoftware.openfire.nio.NIOConnection)

at org.jivesoftware.openfire.nio.ConnectionHandler.sessionClosed(ConnectionHandler .java:115)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.sessionClosed( DefaultIoFilterChain.java:793)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionClosed(Def aultIoFilterChain.java:504)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$2(DefaultIoFilterC hain.java:500)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionClosed (DefaultIoFilterChain.java:923)

at org.apache.mina.core.filterchain.IoFilterAdapter.sessionClosed(IoFilterAdapter. java:88)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionClosed(Def aultIoFilterChain.java:504)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$2(DefaultIoFilterC hain.java:500)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionClosed (DefaultIoFilterChain.java:923)

at org.apache.mina.filter.codec.ProtocolCodecFilter.sessionClosed(ProtocolCodecFil ter.java:367)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionClosed(Def aultIoFilterChain.java:504)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$2(DefaultIoFilterC hain.java:500)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionClosed (DefaultIoFilterChain.java:923)

at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:109)

at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63)

at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(Ordere dThreadPoolExecutor.java:769)

at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(Order edThreadPoolExecutor.java:761)

at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThr eadPoolExecutor.java:703)

at java.lang.Thread.run(Thread.java:745)


“C2S-Thread-3”:

at org.jivesoftware.openfire.nio.NIOConnection.isClosed(NIOConnection.java:249)

  • waiting to lock <0x0000000094e47a98> (a org.jivesoftware.openfire.nio.NIOConnection)

at org.jivesoftware.openfire.nio.NIOConnection.deliver(NIOConnection.java:257)

at org.jivesoftware.openfire.session.LocalClientSession.deliver(LocalClientSession .java:857)

at org.jivesoftware.openfire.session.LocalSession.process(LocalSession.java:289)

at org.jivesoftware.openfire.spi.RoutingTableImpl.routeToLocalDomain(RoutingTableI mpl.java:354)

at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.jav a:239)

at org.jivesoftware.openfire.roster.Roster.broadcastPresence(Roster.java:609)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.broadcastUpdate(Presenc eUpdateHandler.java:308)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:162)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:137)

at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateH andler.java:201)

at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:148)

at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:84)

at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:84)

at org.jivesoftware.openfire.SessionManager$ClientSessionListener.onConnectionClos e(SessionManager.java:1183)

at org.jivesoftware.openfire.nio.NIOConnection.notifyCloseListeners(NIOConnection. java:237)

at org.jivesoftware.openfire.nio.NIOConnection.close(NIOConnection.java:219)

  • locked <0x0000000094e3e540> (a org.jivesoftware.openfire.nio.NIOConnection)

at org.jivesoftware.openfire.nio.ConnectionHandler.sessionClosed(ConnectionHandler .java:115)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.sessionClosed( DefaultIoFilterChain.java:793)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionClosed(Def aultIoFilterChain.java:504)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$2(DefaultIoFilterC hain.java:500)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionClosed (DefaultIoFilterChain.java:923)

at org.apache.mina.core.filterchain.IoFilterAdapter.sessionClosed(IoFilterAdapter. java:88)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionClosed(Def aultIoFilterChain.java:504)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$2(DefaultIoFilterC hain.java:500)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionClosed (DefaultIoFilterChain.java:923)

at org.apache.mina.filter.codec.ProtocolCodecFilter.sessionClosed(ProtocolCodecFil ter.java:367)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionClosed(Def aultIoFilterChain.java:504)

at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$2(DefaultIoFilterC hain.java:500)

at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionClosed (DefaultIoFilterChain.java:923)

at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:109)

at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63)

at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(Ordere dThreadPoolExecutor.java:769)

at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(Order edThreadPoolExecutor.java:761)

at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThr eadPoolExecutor.java:703)

at java.lang.Thread.run(Thread.java:745)


A possible solution is to refactor the NIOConnection.close method, to invoke notifyCloseListeners() after the synchronized block. This is also how the method was arranged in Openfire versions at least as recent as 3.7.

Below is a refactored version of the method:


public void close() {

synchronized(this) {

if (isClosed()) {

return;

}

try {

deliverRawText(flashClient ? "</flash:stream>" : "</stream:stream>", false);

} catch (Exception e) {

// Ignore

}

if (session != null) {

session.setStatus(Session.STATUS_CLOSED);

}

closed = true;

// commented, moved to after the synchronized block.

// notifyCloseListeners(); // clean up session, etc.

}

// moved from within the synchronized block.

notifyCloseListeners(); // clean up session, etc.

ioSession.close(false); // async via MINA

}

Thanks so much! I have filed this as OF-881 and made it a blocker prior to GA release. Could you submit a pull request on github for it?