Cleanup routes from defunct cluster member servers
Description
When running in cluster mode we have noticed that shutting down one or more of the member servers triggers the following ISE due to invalid client session routes. These defunct sessions must be removed from the routing caches of the remaining member nodes.
java.lang.IllegalArgumentException: Requested node fd84a647-a0f1-4eba-903a-87c61a189340 not found in cluster at com.jivesoftware.util.cache.ClusteredCacheFactory.doClusterTask(ClusteredCacheFactory.java:307) at org.jivesoftware.util.cache.CacheFactory.doClusterTask(CacheFactory.java:574) at com.jivesoftware.util.cluster.ClusterPacketRouter.routePacket(ClusterPacketRouter.java:46) at org.jivesoftware.openfire.spi.RoutingTableImpl.routeToLocalDomain(RoutingTableImpl.java:353) at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.java:238) at org.jivesoftware.openfire.SessionManager.broadcastPresenceToOtherResources(SessionManager.java:648) at org.jivesoftware.openfire.roster.Roster.broadcastPresence(Roster.java:642) at org.jivesoftware.openfire.handler.PresenceUpdateHandler.broadcastUpdate(PresenceUpdateHandler.java:308) at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateHandler.java:162) at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateHandler.java:137) at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateHandler.java:201) at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:148) at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:84) at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:84) at org.jivesoftware.openfire.SessionPacketRouter.route(SessionPacketRouter.java:120) at org.jivesoftware.openfire.SessionPacketRouter.route(SessionPacketRouter.java:75) at org.jivesoftware.openfire.http.HttpSession.sendPendingPackets(HttpSession.java:650) at org.jivesoftware.openfire.http.HttpSessionManager$HttpPacketSender.run(HttpSessionManager.java:432) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
When running in cluster mode we have noticed that shutting down one or more of the member servers triggers the following ISE due to invalid client session routes. These defunct sessions must be removed from the routing caches of the remaining member nodes.
java.lang.IllegalArgumentException: Requested node fd84a647-a0f1-4eba-903a-87c61a189340 not found in cluster at com.jivesoftware.util.cache.ClusteredCacheFactory.doClusterTask(ClusteredCacheFactory.java:307) at org.jivesoftware.util.cache.CacheFactory.doClusterTask(CacheFactory.java:574) at com.jivesoftware.util.cluster.ClusterPacketRouter.routePacket(ClusterPacketRouter.java:46) at org.jivesoftware.openfire.spi.RoutingTableImpl.routeToLocalDomain(RoutingTableImpl.java:353) at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.java:238) at org.jivesoftware.openfire.SessionManager.broadcastPresenceToOtherResources(SessionManager.java:648) at org.jivesoftware.openfire.roster.Roster.broadcastPresence(Roster.java:642) at org.jivesoftware.openfire.handler.PresenceUpdateHandler.broadcastUpdate(PresenceUpdateHandler.java:308) at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateHandler.java:162) at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateHandler.java:137) at org.jivesoftware.openfire.handler.PresenceUpdateHandler.process(PresenceUpdateHandler.java:201) at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:148) at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:84) at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:84) at org.jivesoftware.openfire.SessionPacketRouter.route(SessionPacketRouter.java:120) at org.jivesoftware.openfire.SessionPacketRouter.route(SessionPacketRouter.java:75) at org.jivesoftware.openfire.http.HttpSession.sendPendingPackets(HttpSession.java:650) at org.jivesoftware.openfire.http.HttpSessionManager$HttpPacketSender.run(HttpSessionManager.java:432) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)