Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2873

Backpressure implentation delete ephemeral too frequently

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: None

      Description

      The backpressure implementation deletes the znode when not relevant but that hits zookeeper issue of too frequent deletion and creation or same path for ephemeral znode. Below is exception we get when zk hits this issue:

      017-09-18 15:00:34.980 b.s.util WorkerBackpressureThread [WARN] Expecting exception of class: class org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoNodeException, but exception chain only contains: (#<NoAuthException org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /backpressure/TestTopology--170916-005358-117-1505523256/d11af335-6e03-48b6-801e-7369acfe9ffd-127.0.0.1-6721>)
      2017-09-18 15:00:34.980 b.s.d.worker WorkerBackpressureThread [ERROR] workerBackpressure update failed when connecting to ZK ... will retry
      java.lang.RuntimeException: org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /backpressure/TestTopology--170916-005358-117-1505523256/d11af335-6e03-48b6-801e-7369acfe9ffd-127.0.0.1-6721
      	at backtype.storm.util$wrap_in_runtime.invoke(util.clj:52) ~[storm-core-0.10.2.y.jar:0.10.2.y]
      	at backtype.storm.zookeeper$delete_node.doInvoke(zookeeper.clj:110) ~[storm-core-0.10.2.y.jar:0.10.2.y]
      	at clojure.lang.RestFn.invoke(RestFn.java:464) ~[clojure-1.6.0.jar:?]
      	at backtype.storm.zookeeper$delete_recursive.invoke(zookeeper.clj:189) ~[storm-core-0.10.2.y.jar:0.10.2.y]
      	at backtype.storm.cluster_state.zookeeper_state_factory$_mkState$reify__4207.delete_node(zookeeper_state_factory.clj:117) ~[storm-core-0.10.2.y.jar:0.10.2.y]
      	at sun.reflect.GeneratedMethodAccessor860.invoke(Unknown Source) ~[?:?]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_102]
      	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
      	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.6.0.jar:?]
      	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.6.0.jar:?]
      	at org.apache.storm.pacemaker.pacemaker_state_factory$_mkState$reify__4254.delete_node(pacemaker_state_factory.clj:174) ~[storm-core-0.10.2.y.jar:0.10.2.y]
      	at sun.reflect.GeneratedMethodAccessor859.invoke(Unknown Source) ~[?:?]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_102]
      	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
      	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.6.0.jar:?]
      	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.6.0.jar:?]
      	at backtype.storm.cluster$mk_storm_cluster_state$reify__3873.worker_backpressure_BANG_(cluster.clj:421) ~[storm-core-0.10.2.y.jar:0.10.2.y]
      	at sun.reflect.GeneratedMethodAccessor857.invoke(Unknown Source) ~[?:?]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_102]
      	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
      	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.6.0.jar:?]
      	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.6.0.jar:?]
      	at backtype.storm.daemon.worker$mk_backpressure_handler$fn__7117.invoke(worker.clj:161) [storm-core-0.10.2.y.jar:0.10.2.y]
      	at backtype.storm.disruptor$worker_backpressure_handler$reify__6432.onEvent(disruptor.clj:57) [storm-core-0.10.2.y.jar:0.10.2.y]
      	at backtype.storm.utils.WorkerBackpressureThread.run(WorkerBackpressureThread.java:64) [storm-core-0.10.2.y.jar:0.10.2.y]
      Caused by: org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /backpressure/TestTopology--170916-005358-117-1505523256/d11af335-6e03-48b6-801e-7369acfe9ffd-127.0.0.1-6721
      	at org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:113) ~[storm-core-0.10.2.y.jar:0.10.2.y]
      	at org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[storm-core-0.10.2.y.jar:0.10.2.y]
      	at org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) ~[storm-core-0.10.2.y.jar:0.10.2.y]
      	at org.apache.storm.shade.org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:239) ~[storm-core-0.10.2.y.jar:0.10.2.y]
      	at org.apache.storm.shade.org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:234) ~[storm-core-0.10.2.y.jar:0.10.2.y]
      	at org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[storm-core-0.10.2.y.jar:0.10.2.y]
      	at org.apache.storm.shade.org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:230) ~[storm-core-0.10.2.y.jar:0.10.2.y]
      	at org.apache.storm.shade.org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:215) ~[storm-core-0.10.2.y.jar:0.10.2.y]
      	at org.apache.storm.shade.org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:42) ~[storm-core-0.10.2.y.jar:0.10.2.y]
      	at backtype.storm.zookeeper$delete_node.doInvoke(zookeeper.clj:107) ~[storm-core-0.10.2.y.jar:0.10.2.y]
      	... 23 more
      

        Attachments

          Activity

            People

            • Assignee:
              kishorvpatil Kishor Patil
              Reporter:
              kishorvpatil Kishor Patil
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 2h 40m
                2h 40m