This morning, I found the open_chat MUC on igniterealtime to be unresponsive. These were basically the same characteristics as I experienced in OF-983. I suspected another deadlock situation.
Turns out that there indeed was a deadlock, but it appears to be a different one than the one in OF-983. The timing of both occurrences is so close that there probably is a relation somewhere, but I've thought it sensible to track this one in a new issue.
This is the deadlock:
Found one Java-level deadlock:
=============================
"Server SR - 1141914984":
waiting to lock monitor 0x000000000ff51138 (object 0x00000006c3f71b98, a java.lang.String),
which is held by "socket_c2s-thread-1"
"socket_c2s-thread-1":
waiting for ownable synchronizer 0x00000006c4f88858, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "Server SR - 1141914984"
Java stack information for the threads listed above:
===================================================
"Server SR - 1141914984":
at org.jivesoftware.util.cache.DefaultLocalCacheStrategy.lookupLockForRelease(DefaultLocalCacheStrategy.java:169)
- waiting to lock <0x00000006c3f71b98> (a java.lang.String)
at org.jivesoftware.util.cache.DefaultLocalCacheStrategy.releaseLock(DefaultLocalCacheStrategy.java:148)
at org.jivesoftware.util.cache.DefaultLocalCacheStrategy.access$100(DefaultLocalCacheStrategy.java:41)
at org.jivesoftware.util.cache.DefaultLocalCacheStrategy$LocalLock.unlock(DefaultLocalCacheStrategy.java:201)
at org.jivesoftware.openfire.SessionManager.getIncomingServerSessions(SessionManager.java:875)
at org.jivesoftware.openfire.server.ServerDialback.validateRemoteDomain(ServerDialback.java:533)
at org.jivesoftware.openfire.session.LocalIncomingServerSession.validateSubsequentDomain(LocalIncomingServerSession.java:248)
at org.jivesoftware.openfire.net.ServerSocketReader.processUnknowPacket(ServerSocketReader.java:143)
at org.jivesoftware.openfire.net.SocketReader.process(SocketReader.java:235)
at org.jivesoftware.openfire.net.BlockingReadingMode.readStream(BlockingReadingMode.java:168)
at org.jivesoftware.openfire.net.BlockingReadingMode.run(BlockingReadingMode.java:76)
at org.jivesoftware.openfire.net.SocketReader.run(SocketReader.java:138)
at java.lang.Thread.run(Thread.java:745)
"socket_c2s-thread-1":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c4f88858> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at org.jivesoftware.util.cache.DefaultLocalCacheStrategy.acquireLock(DefaultLocalCacheStrategy.java:144)
at org.jivesoftware.util.cache.DefaultLocalCacheStrategy.access$000(DefaultLocalCacheStrategy.java:41)
at org.jivesoftware.util.cache.DefaultLocalCacheStrategy$LocalLock.lock(DefaultLocalCacheStrategy.java:196)
at org.jivesoftware.openfire.spi.RoutingTableImpl.addServerRoute(RoutingTableImpl.java:139)
at org.jivesoftware.openfire.spi.RoutingTableImpl.routeToRemoteDomain(RoutingTableImpl.java:502)
- locked <0x00000006c3f71b98> (a java.lang.String)
at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.java:265)
at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:135)
at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:84)
at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:88)
at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:66)
at org.jivesoftware.openfire.muc.spi.LocalMUCRole.send(LocalMUCRole.java:282)
at org.jivesoftware.openfire.muc.spi.LocalMUCRoom.broadcast(LocalMUCRoom.java:1185)
at org.jivesoftware.openfire.muc.cluster.BroadcastPresenceRequest$1.run(BroadcastPresenceRequest.java:74)
at org.jivesoftware.openfire.muc.cluster.MUCRoomTask.execute(MUCRoomTask.java:85)
at org.jivesoftware.openfire.muc.cluster.BroadcastPresenceRequest.run(BroadcastPresenceRequest.java:71)
at org.jivesoftware.openfire.muc.spi.LocalMUCRoom.broadcastPresence(LocalMUCRoom.java:1136)
at org.jivesoftware.openfire.muc.spi.LocalMUCRoom.joinRoom(LocalMUCRoom.java:685)
at org.jivesoftware.openfire.muc.spi.LocalMUCUser.process(LocalMUCUser.java:487)
at org.jivesoftware.openfire.muc.spi.LocalMUCUser.process(LocalMUCUser.java:199)
at org.jivesoftware.openfire.muc.spi.MultiUserChatServiceImpl.processPacket(MultiUserChatServiceImpl.java:332)
at org.jivesoftware.openfire.component.InternalComponentManager$RoutableComponents.process(InternalComponentManager.java:606)
at org.jivesoftware.openfire.spi.RoutingTableImpl.routeToComponent(RoutingTableImpl.java:419)
at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.java:261)
at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:170)
at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:84)
at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:88)
at org.jivesoftware.openfire.net.StanzaHandler.processPresence(StanzaHandler.java:364)
at org.jivesoftware.openfire.net.ClientStanzaHandler.processPresence(ClientStanzaHandler.java:106)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:281)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:203)
at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandler.java:181)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:690)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765)
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765)
at org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:407)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:236)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765)
at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:74)
at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:769)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:761)
at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:703)
at java.lang.Thread.run(Thread.java:745)
Found 1 deadlock.
This morning, I found the open_chat MUC on igniterealtime to be unresponsive. These were basically the same characteristics as I experienced in OF-983. I suspected another deadlock situation.
Turns out that there indeed was a deadlock, but it appears to be a different one than the one in OF-983. The timing of both occurrences is so close that there probably is a relation somewhere, but I've thought it sensible to track this one in a new issue.
This is the deadlock:
Found one Java-level deadlock: ============================= "Server SR - 1141914984": waiting to lock monitor 0x000000000ff51138 (object 0x00000006c3f71b98, a java.lang.String), which is held by "socket_c2s-thread-1" "socket_c2s-thread-1": waiting for ownable synchronizer 0x00000006c4f88858, (a java.util.concurrent.locks.ReentrantLock$NonfairSync), which is held by "Server SR - 1141914984" Java stack information for the threads listed above: =================================================== "Server SR - 1141914984": at org.jivesoftware.util.cache.DefaultLocalCacheStrategy.lookupLockForRelease(DefaultLocalCacheStrategy.java:169) - waiting to lock <0x00000006c3f71b98> (a java.lang.String) at org.jivesoftware.util.cache.DefaultLocalCacheStrategy.releaseLock(DefaultLocalCacheStrategy.java:148) at org.jivesoftware.util.cache.DefaultLocalCacheStrategy.access$100(DefaultLocalCacheStrategy.java:41) at org.jivesoftware.util.cache.DefaultLocalCacheStrategy$LocalLock.unlock(DefaultLocalCacheStrategy.java:201) at org.jivesoftware.openfire.SessionManager.getIncomingServerSessions(SessionManager.java:875) at org.jivesoftware.openfire.server.ServerDialback.validateRemoteDomain(ServerDialback.java:533) at org.jivesoftware.openfire.session.LocalIncomingServerSession.validateSubsequentDomain(LocalIncomingServerSession.java:248) at org.jivesoftware.openfire.net.ServerSocketReader.processUnknowPacket(ServerSocketReader.java:143) at org.jivesoftware.openfire.net.SocketReader.process(SocketReader.java:235) at org.jivesoftware.openfire.net.BlockingReadingMode.readStream(BlockingReadingMode.java:168) at org.jivesoftware.openfire.net.BlockingReadingMode.run(BlockingReadingMode.java:76) at org.jivesoftware.openfire.net.SocketReader.run(SocketReader.java:138) at java.lang.Thread.run(Thread.java:745) "socket_c2s-thread-1": at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c4f88858> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.jivesoftware.util.cache.DefaultLocalCacheStrategy.acquireLock(DefaultLocalCacheStrategy.java:144) at org.jivesoftware.util.cache.DefaultLocalCacheStrategy.access$000(DefaultLocalCacheStrategy.java:41) at org.jivesoftware.util.cache.DefaultLocalCacheStrategy$LocalLock.lock(DefaultLocalCacheStrategy.java:196) at org.jivesoftware.openfire.spi.RoutingTableImpl.addServerRoute(RoutingTableImpl.java:139) at org.jivesoftware.openfire.spi.RoutingTableImpl.routeToRemoteDomain(RoutingTableImpl.java:502) - locked <0x00000006c3f71b98> (a java.lang.String) at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.java:265) at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:135) at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:84) at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:88) at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:66) at org.jivesoftware.openfire.muc.spi.LocalMUCRole.send(LocalMUCRole.java:282) at org.jivesoftware.openfire.muc.spi.LocalMUCRoom.broadcast(LocalMUCRoom.java:1185) at org.jivesoftware.openfire.muc.cluster.BroadcastPresenceRequest$1.run(BroadcastPresenceRequest.java:74) at org.jivesoftware.openfire.muc.cluster.MUCRoomTask.execute(MUCRoomTask.java:85) at org.jivesoftware.openfire.muc.cluster.BroadcastPresenceRequest.run(BroadcastPresenceRequest.java:71) at org.jivesoftware.openfire.muc.spi.LocalMUCRoom.broadcastPresence(LocalMUCRoom.java:1136) at org.jivesoftware.openfire.muc.spi.LocalMUCRoom.joinRoom(LocalMUCRoom.java:685) at org.jivesoftware.openfire.muc.spi.LocalMUCUser.process(LocalMUCUser.java:487) at org.jivesoftware.openfire.muc.spi.LocalMUCUser.process(LocalMUCUser.java:199) at org.jivesoftware.openfire.muc.spi.MultiUserChatServiceImpl.processPacket(MultiUserChatServiceImpl.java:332) at org.jivesoftware.openfire.component.InternalComponentManager$RoutableComponents.process(InternalComponentManager.java:606) at org.jivesoftware.openfire.spi.RoutingTableImpl.routeToComponent(RoutingTableImpl.java:419) at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.java:261) at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:170) at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:84) at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:88) at org.jivesoftware.openfire.net.StanzaHandler.processPresence(StanzaHandler.java:364) at org.jivesoftware.openfire.net.ClientStanzaHandler.processPresence(ClientStanzaHandler.java:106) at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:281) at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:203) at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandler.java:181) at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:690) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765) at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765) at org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:407) at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:236) at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417) at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47) at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765) at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:74) at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63) at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTask(OrderedThreadPoolExecutor.java:769) at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.runTasks(OrderedThreadPoolExecutor.java:761) at org.apache.mina.filter.executor.OrderedThreadPoolExecutor$Worker.run(OrderedThreadPoolExecutor.java:703) at java.lang.Thread.run(Thread.java:745) Found 1 deadlock.