[STORM-2873] Backpressure implentation delete ephemeral too frequently

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Labels: pull-request-available
• Assignee: Kishor Patil
• Reporter: Kishor Patil
• Created: 29/Dec/17 18:44
• Updated: 12/Feb/18 18:34
• Resolved: 01/Feb/18 14:37

Description

The backpressure implementation deletes the znode when not relevant but that hits zookeeper issue of too frequent deletion and creation or same path for ephemeral znode. Below is exception we get when zk hits this issue:

017-09-18 15:00:34.980 b.s.util WorkerBackpressureThread [WARN] Expecting exception of class: class org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoNodeException, but exception chain only contains: (#<NoAuthException org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /backpressure/TestTopology--170916-005358-117-1505523256/d11af335-6e03-48b6-801e-7369acfe9ffd-127.0.0.1-6721>)
2017-09-18 15:00:34.980 b.s.d.worker WorkerBackpressureThread [ERROR] workerBackpressure update failed when connecting to ZK ... will retry
java.lang.RuntimeException: org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /backpressure/TestTopology--170916-005358-117-1505523256/d11af335-6e03-48b6-801e-7369acfe9ffd-127.0.0.1-6721
	at backtype.storm.util$wrap_in_runtime.invoke(util.clj:52) ~[storm-core-0.10.2.y.jar:0.10.2.y]
	at backtype.storm.zookeeper$delete_node.doInvoke(zookeeper.clj:110) ~[storm-core-0.10.2.y.jar:0.10.2.y]
	at clojure.lang.RestFn.invoke(RestFn.java:464) ~[clojure-1.6.0.jar:?]
	at backtype.storm.zookeeper$delete_recursive.invoke(zookeeper.clj:189) ~[storm-core-0.10.2.y.jar:0.10.2.y]
	at backtype.storm.cluster_state.zookeeper_state_factory$_mkState$reify__4207.delete_node(zookeeper_state_factory.clj:117) ~[storm-core-0.10.2.y.jar:0.10.2.y]
	at sun.reflect.GeneratedMethodAccessor860.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_102]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.6.0.jar:?]
	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.6.0.jar:?]
	at org.apache.storm.pacemaker.pacemaker_state_factory$_mkState$reify__4254.delete_node(pacemaker_state_factory.clj:174) ~[storm-core-0.10.2.y.jar:0.10.2.y]
	at sun.reflect.GeneratedMethodAccessor859.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_102]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.6.0.jar:?]
	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.6.0.jar:?]
	at backtype.storm.cluster$mk_storm_cluster_state$reify__3873.worker_backpressure_BANG_(cluster.clj:421) ~[storm-core-0.10.2.y.jar:0.10.2.y]
	at sun.reflect.GeneratedMethodAccessor857.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_102]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.6.0.jar:?]
	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.6.0.jar:?]
	at backtype.storm.daemon.worker$mk_backpressure_handler$fn__7117.invoke(worker.clj:161) [storm-core-0.10.2.y.jar:0.10.2.y]
	at backtype.storm.disruptor$worker_backpressure_handler$reify__6432.onEvent(disruptor.clj:57) [storm-core-0.10.2.y.jar:0.10.2.y]
	at backtype.storm.utils.WorkerBackpressureThread.run(WorkerBackpressureThread.java:64) [storm-core-0.10.2.y.jar:0.10.2.y]
Caused by: org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /backpressure/TestTopology--170916-005358-117-1505523256/d11af335-6e03-48b6-801e-7369acfe9ffd-127.0.0.1-6721
	at org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:113) ~[storm-core-0.10.2.y.jar:0.10.2.y]
	at org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[storm-core-0.10.2.y.jar:0.10.2.y]
	at org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) ~[storm-core-0.10.2.y.jar:0.10.2.y]
	at org.apache.storm.shade.org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:239) ~[storm-core-0.10.2.y.jar:0.10.2.y]
	at org.apache.storm.shade.org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:234) ~[storm-core-0.10.2.y.jar:0.10.2.y]
	at org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[storm-core-0.10.2.y.jar:0.10.2.y]
	at org.apache.storm.shade.org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:230) ~[storm-core-0.10.2.y.jar:0.10.2.y]
	at org.apache.storm.shade.org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:215) ~[storm-core-0.10.2.y.jar:0.10.2.y]
	at org.apache.storm.shade.org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:42) ~[storm-core-0.10.2.y.jar:0.10.2.y]
	at backtype.storm.zookeeper$delete_node.doInvoke(zookeeper.clj:107) ~[storm-core-0.10.2.y.jar:0.10.2.y]
	... 23 more

Issue Links

• links to: GitHub Pull Request #2487
• links to: GitHub Pull Request #2529