Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-3117

Deleting blobs for running topologies hoses Nimbus

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0
    • 2.0.0
    • None

    Description

      The following test pseudo-code causes issues:

      cluster.submitTopology(cluster.getTopologiesJarFile(), topoName, config, topology);
cluster.waitTopologyUp(topoName);
cluster.deleteAllBlobs();

      This causes nimbus to get stuck and restart:

       

      2018-06-20 15:48:14.273 o.a.s.d.n.Nimbus pool-27-thread-694 [INFO] Received topology submission for wc-topology-test (storm-0.10.2.y.251 JDK-1.8.0_131) 
2018-06-20 15:48:14.629 o.a.s.d.n.Nimbus pool-27-thread-694 [INFO] Activating wc-topology-test: wc-topology-test-1-1529509694
2018-06-20 15:48:14.724 o.a.s.d.n.Nimbus pool-27-thread-703 [INFO] TRANSITION: wc-topology-test-1-1529509694 KILL null true
2018-06-20 15:48:14.812 o.a.s.d.n.Nimbus pool-27-thread-704 [INFO] Deleted blob for key wc-topology-test-1-1529509694-stormconf.ser
2018-06-20 15:48:14.830 o.a.s.d.n.Nimbus pool-27-thread-704 [INFO] Deleted blob for key wc-topology-test-1-1529509694-stormcode.ser
2018-06-20 15:48:14.863 o.a.s.d.n.Nimbus pool-27-thread-704 [INFO] Deleted blob for key wc-topology-test-1-1529509694-stormjar.jar
2018-06-20 15:48:18.449 o.a.s.s.r.s.p.DefaultSchedulingPriorityStrategy timer [INFO] SIM Scheduling wc-topology-test-1-1529509694 with score of 0.3125
2018-06-20 15:48:18.492 o.a.s.s.Cluster timer [INFO] STATUS - wc-topology-test-1-1529509694 Running - Fully Scheduled by DefaultResourceAwareStrategy
2018-06-20 15:48:18.527 o.a.s.d.n.Nimbus timer [INFO] Setting new assignment for topology id wc-topology-test-1-1529509694:

2018-06-20 15:48:18.979 o.a.s.d.n.Nimbus pool-27-thread-722 [WARN] get blob meta exception.
org.apache.storm.utils.WrappedKeyNotFoundException: wc-topology-test-1-1529509694-stormjar.jar
        at org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:256) ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.blobstore.LocalFsBlobStore.getBlobMeta(LocalFsBlobStore.java:286) ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.daemon.nimbus.Nimbus.getBlobMeta(Nimbus.java:3483) [storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:4011) [storm-client-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3990) [storm-client-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:38) [shaded-deps-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [shaded-deps-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.security.auth.sasl.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:147) [storm-client-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:291) [shaded-deps-2.0.0.y.jar:2.0.0.y]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

2018-06-20 15:48:22.884 o.a.s.d.n.Nimbus timer [INFO] Renewing Creds For wc-topology-test-1-1529509694 with org.apache.storm.security.auth.kerberos.AutoTGT@4482469c owned by hadoopqa@DEV.YGRID.YAHOO.COM


2018-06-20 15:48:37.947 o.a.s.d.n.Nimbus timer [ERROR] Error while processing event
java.lang.RuntimeException: KeyNotFoundException(msg:wc-topology-test-1-1529509694-stormcode.ser)
        at org.apache.storm.daemon.nimbus.Nimbus.lambda$launchServer$48(Nimbus.java:2822) ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.StormTimer$1.run(StormTimer.java:111) ~[storm-client-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:227) [storm-client-2.0.0.y.jar:2.0.0.y]
Caused by: org.apache.storm.utils.WrappedKeyNotFoundException: wc-topology-test-1-1529509694-stormcode.ser
        at org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:256) ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.blobstore.LocalFsBlobStore.getBlobReplication(LocalFsBlobStore.java:420) ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.daemon.nimbus.Nimbus.getBlobReplicationCount(Nimbus.java:1517) ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.daemon.nimbus.Nimbus.getClusterInfoImpl(Nimbus.java:2675) ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.daemon.nimbus.Nimbus.sendClusterMetricsToExecutors(Nimbus.java:2686) ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.daemon.nimbus.Nimbus.lambda$launchServer$48(Nimbus.java:2819) ~[storm-server-2.0.0.y.jar:2.0.0.y]
        ... 2 more
2018-06-20 15:48:37.948 o.a.s.u.Utils timer [ERROR] Halting process: Error while processing event
java.lang.RuntimeException: Halting process: Error while processing event
        at org.apache.storm.utils.Utils.exitProcess(Utils.java:468) ~[storm-client-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.daemon.nimbus.Nimbus.lambda$new$17(Nimbus.java:488) ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:252) [storm-client-2.0.0.y.jar:2.0.0.y]
2018-06-20 15:48:37.950 o.a.s.d.n.Nimbus Thread-11 [INFO] Shutting down master
2018-06-20 15:48:37.950 o.a.s.u.Utils Thread-12 [INFO] Halting after 10 seconds

2018-06-20 15:48:46.672 o.a.s.d.n.Nimbus pool-27-thread-798 [WARN] get blob meta exception.
org.apache.storm.utils.WrappedKeyNotFoundException: wc-topology-test-1-1529509694-stormconf.ser
        at org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:256) ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.blobstore.LocalFsBlobStore.getBlobMeta(LocalFsBlobStore.java:286) ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.daemon.nimbus.Nimbus.getBlobMeta(Nimbus.java:3483) [storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:4011) [storm-client-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3990) [storm-client-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:38) [shaded-deps-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [shaded-deps-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.security.auth.sasl.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:147) [storm-client-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:291) [shaded-deps-2.0.0.y.jar:2.0.0.y]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
2018-06-20 15:48:47.950 o.a.s.u.Utils Thread-12 [WARN] Forcing Halt...

      Nimbus then continually restarts:

      2018-06-20 15:48:54.635 o.a.s.u.Utils main [ERROR] Received error in main thread.. terminating server...
java.lang.Error: java.lang.IllegalStateException: Could not find credentials for topology wc-topology-test-1-1529509694 at path /storms. Don't know how to fix this automatically. Please add needed ACLs, or delete the path.
        at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:603) ~[storm-client-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:582) ~[storm-client-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.utils.Utils$5.uncaughtException(Utils.java:931) [storm-client-2.0.0.y.jar:2.0.0.y]
        at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057) [?:1.8.0_131]
        at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052) [?:1.8.0_131]
        at java.lang.Thread.dispatchUncaughtException(Thread.java:1959) [?:1.8.0_131]
Caused by: java.lang.IllegalStateException: Could not find credentials for topology wc-topology-test-1-1529509694 at path /storms. Don't know how to fix this automatically. Please add needed ACLs, or delete the path.
        at org.apache.storm.zookeeper.AclEnforcement.getTopoAcl(AclEnforcement.java:194) ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.zookeeper.AclEnforcement.verifyParentWithTopoChildren(AclEnforcement.java:250) ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.zookeeper.AclEnforcement.verifyParentWithReadOnlyTopoChildren(AclEnforcement.java:258) ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.zookeeper.AclEnforcement.verifyAcls(AclEnforcement.java:136) ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.daemon.nimbus.Nimbus.launch(Nimbus.java:1155) ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.daemon.nimbus.Nimbus.main(Nimbus.java:1162) ~[storm-server-2.0.0.y.jar:2.0.0.y]

      Attachments

        Issue Links

          links to

          Web Link GitHub Pull Request #2732

          Activity

            People

              agresch Aaron Gresch
              agresch Aaron Gresch
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 50m
                  1h 50m