[CURATOR-325] Background retry falls into infinite loop of SessionExpiredException

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.9.1, 2.10.0
• Component/s: Client
• Labels: SessionExpiredException loop
• Environment: sun java jdk 1.7.0_55, curator 2.9.1, zookeeper 3.4.6
• Assignee: Unassigned
• Reporter: clive du
• Created: 20/May/16 05:27
• Updated: 06/Oct/17 14:22

Description

after long time gc pause,which longer than zookeeper session time,the zookeeper cluster invalidate the session id holding by the client and waiting the client to reconnect,but client consider the SessionExpiredException as retry exception and re-put to the background queue,so wo get the stacktrace infinitely.

12:50:54.337 [configuration-0-EventThread] DEBUG org.apache.curator.RetryLoop - Retrying operation
12:50:54.337 [configuration-0-EventThread] DEBUG org.apache.curator.RetryLoop - Retry-able exception received
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /dynamic/apps/258741001/DEV
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:304) ~[curator-framework-2.10.0.jar:na]
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:293) ~[curator-framework-2.10.0.jar:na]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:108) ~[curator-client-2.10.0.jar:na]
at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:290) [curator-framework-2.10.0.jar:na]
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:281) [curator-framework-2.10.0.jar:na]
at org.apache.curator.framework.imps.GetDataBuilderImpl$1.forPath(GetDataBuilderImpl.java:105) [curator-framework-2.10.0.jar:na]
at org.apache.curator.framework.imps.GetDataBuilderImpl$1.forPath(GetDataBuilderImpl.java:65) [curator-framework-2.10.0.jar:na]
at com.ctrip.flight.configuration.client.AbstractZookeeperClient.getData(AbstractZookeeperClient.java:68) [classes/:na]
at com.ctrip.flight.configuration.client.ZooKeeperConfigurationSource.getPublishNodeValue(ZooKeeperConfigurationSource.java:258) [classes/:na]
at com.ctrip.flight.configuration.client.ZooKeeperConfigurationSource.access$100(ZooKeeperConfigurationSource.java:45) [classes/:na]
at com.ctrip.flight.configuration.client.ZooKeeperConfigurationSource$1.nodeChanged(ZooKeeperConfigurationSource.java:105) [classes/:na]
at org.apache.curator.framework.recipes.cache.NodeCache$4.apply(NodeCache.java:310) [curator-recipes-2.10.0.jar:na]
at org.apache.curator.framework.recipes.cache.NodeCache$4.apply(NodeCache.java:304) [curator-recipes-2.10.0.jar:na]
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [curator-framework-2.10.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:310) [guava-19.0.jar:na]
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85) [curator-framework-2.10.0.jar:na]
at org.apache.curator.framework.recipes.cache.NodeCache.setNewData(NodeCache.java:302) [curator-recipes-2.10.0.jar:na]
at org.apache.curator.framework.recipes.cache.NodeCache.processBackgroundResult(NodeCache.java:269) [curator-recipes-2.10.0.jar:na]
at org.apache.curator.framework.recipes.cache.NodeCache.access$300(NodeCache.java:56) [curator-recipes-2.10.0.jar:na]
at org.apache.curator.framework.recipes.cache.NodeCache$3.processResult(NodeCache.java:122) [curator-recipes-2.10.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:749) [curator-framework-2.10.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:522) [curator-framework-2.10.0.jar:na]
at org.apache.curator.framework.imps.GetDataBuilderImpl$3.processResult(GetDataBuilderImpl.java:256) [curator-framework-2.10.0.jar:na]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:561) [zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) [zookeeper-3.4.6.jar:3.4.6-1569965]