Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-3354

LeaderElector is not shut down properly

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0
    • 2.0.0
    • None

    Description

      Nimbus' LeaderElector is not shut down when Nimbus shuts down. This can cause test flakiness because the elector callback may be called after the Zookeeper client is closed.

      Additionally the LeaderListenerCallback may in some cases quit the leadership election if Nimbus isn't ready to become master, but I don't see any code to re-enter the election.
       

      java.lang.IllegalStateException: Client is not started
              at org.apache.storm.shade.org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:444) ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:139) ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:602) ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1191) ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1158) ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64) ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100) ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:1155) ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:605) ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:595) ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:360) ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:308) ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.zookeeper.ClientZookeeper.createNode(ClientZookeeper.java:98) ~[storm-client-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.nimbus.LeaderListenerCallback.setUpNimbusInfo(LeaderListenerCallback.java:154) ~[storm-server-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.nimbus.LeaderListenerCallback.leaderCallBack(LeaderListenerCallback.java:96) ~[storm-server-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.zookeeper.Zookeeper$1.isLeader(Zookeeper.java:123) ~[storm-server-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.recipes.leader.LeaderLatch$9.apply(LeaderLatch.java:665) ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.recipes.leader.LeaderLatch$9.apply(LeaderLatch.java:661) ~[shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) [shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85) [shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.recipes.leader.LeaderLatch.setLeadership(LeaderLatch.java:660) [shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.recipes.leader.LeaderLatch.checkLeadership(LeaderLatch.java:539) [shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.recipes.leader.LeaderLatch.access$700(LeaderLatch.java:65) [shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.recipes.leader.LeaderLatch$7.processResult(LeaderLatch.java:590) [shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:865) [shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:635) [shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:152) [shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.curator.framework.imps.GetChildrenBuilderImpl$2.processResult(GetChildrenBuilderImpl.java:187) [shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:590) [shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]
              at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) [shaded-deps-2.0.1-SNAPSHOT.jar:2.0.1-SNAPSHOT]

      Attachments

        Issue Links

          Activity

            People

              srdo Stig Rohde Døssing
              srdo Stig Rohde Døssing
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m