Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.0.0
-
None
Description
The following test pseudo-code causes issues:
cluster.submitTopology(cluster.getTopologiesJarFile(), topoName, config, topology); cluster.waitTopologyUp(topoName); cluster.deleteAllBlobs();
This causes nimbus to get stuck and restart:
2018-06-20 15:48:14.273 o.a.s.d.n.Nimbus pool-27-thread-694 [INFO] Received topology submission for wc-topology-test (storm-0.10.2.y.251 JDK-1.8.0_131) 2018-06-20 15:48:14.629 o.a.s.d.n.Nimbus pool-27-thread-694 [INFO] Activating wc-topology-test: wc-topology-test-1-1529509694 2018-06-20 15:48:14.724 o.a.s.d.n.Nimbus pool-27-thread-703 [INFO] TRANSITION: wc-topology-test-1-1529509694 KILL null true 2018-06-20 15:48:14.812 o.a.s.d.n.Nimbus pool-27-thread-704 [INFO] Deleted blob for key wc-topology-test-1-1529509694-stormconf.ser 2018-06-20 15:48:14.830 o.a.s.d.n.Nimbus pool-27-thread-704 [INFO] Deleted blob for key wc-topology-test-1-1529509694-stormcode.ser 2018-06-20 15:48:14.863 o.a.s.d.n.Nimbus pool-27-thread-704 [INFO] Deleted blob for key wc-topology-test-1-1529509694-stormjar.jar 2018-06-20 15:48:18.449 o.a.s.s.r.s.p.DefaultSchedulingPriorityStrategy timer [INFO] SIM Scheduling wc-topology-test-1-1529509694 with score of 0.3125 2018-06-20 15:48:18.492 o.a.s.s.Cluster timer [INFO] STATUS - wc-topology-test-1-1529509694 Running - Fully Scheduled by DefaultResourceAwareStrategy 2018-06-20 15:48:18.527 o.a.s.d.n.Nimbus timer [INFO] Setting new assignment for topology id wc-topology-test-1-1529509694: 2018-06-20 15:48:18.979 o.a.s.d.n.Nimbus pool-27-thread-722 [WARN] get blob meta exception. org.apache.storm.utils.WrappedKeyNotFoundException: wc-topology-test-1-1529509694-stormjar.jar at org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:256) ~[storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.blobstore.LocalFsBlobStore.getBlobMeta(LocalFsBlobStore.java:286) ~[storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.daemon.nimbus.Nimbus.getBlobMeta(Nimbus.java:3483) [storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:4011) [storm-client-2.0.0.y.jar:2.0.0.y] at org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3990) [storm-client-2.0.0.y.jar:2.0.0.y] at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:38) [shaded-deps-2.0.0.y.jar:2.0.0.y] at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [shaded-deps-2.0.0.y.jar:2.0.0.y] at org.apache.storm.security.auth.sasl.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:147) [storm-client-2.0.0.y.jar:2.0.0.y] at org.apache.storm.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:291) [shaded-deps-2.0.0.y.jar:2.0.0.y] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] 2018-06-20 15:48:22.884 o.a.s.d.n.Nimbus timer [INFO] Renewing Creds For wc-topology-test-1-1529509694 with org.apache.storm.security.auth.kerberos.AutoTGT@4482469c owned by hadoopqa@DEV.YGRID.YAHOO.COM 2018-06-20 15:48:37.947 o.a.s.d.n.Nimbus timer [ERROR] Error while processing event java.lang.RuntimeException: KeyNotFoundException(msg:wc-topology-test-1-1529509694-stormcode.ser) at org.apache.storm.daemon.nimbus.Nimbus.lambda$launchServer$48(Nimbus.java:2822) ~[storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.StormTimer$1.run(StormTimer.java:111) ~[storm-client-2.0.0.y.jar:2.0.0.y] at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:227) [storm-client-2.0.0.y.jar:2.0.0.y] Caused by: org.apache.storm.utils.WrappedKeyNotFoundException: wc-topology-test-1-1529509694-stormcode.ser at org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:256) ~[storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.blobstore.LocalFsBlobStore.getBlobReplication(LocalFsBlobStore.java:420) ~[storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.daemon.nimbus.Nimbus.getBlobReplicationCount(Nimbus.java:1517) ~[storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.daemon.nimbus.Nimbus.getClusterInfoImpl(Nimbus.java:2675) ~[storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.daemon.nimbus.Nimbus.sendClusterMetricsToExecutors(Nimbus.java:2686) ~[storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.daemon.nimbus.Nimbus.lambda$launchServer$48(Nimbus.java:2819) ~[storm-server-2.0.0.y.jar:2.0.0.y] ... 2 more 2018-06-20 15:48:37.948 o.a.s.u.Utils timer [ERROR] Halting process: Error while processing event java.lang.RuntimeException: Halting process: Error while processing event at org.apache.storm.utils.Utils.exitProcess(Utils.java:468) ~[storm-client-2.0.0.y.jar:2.0.0.y] at org.apache.storm.daemon.nimbus.Nimbus.lambda$new$17(Nimbus.java:488) ~[storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:252) [storm-client-2.0.0.y.jar:2.0.0.y] 2018-06-20 15:48:37.950 o.a.s.d.n.Nimbus Thread-11 [INFO] Shutting down master 2018-06-20 15:48:37.950 o.a.s.u.Utils Thread-12 [INFO] Halting after 10 seconds 2018-06-20 15:48:46.672 o.a.s.d.n.Nimbus pool-27-thread-798 [WARN] get blob meta exception. org.apache.storm.utils.WrappedKeyNotFoundException: wc-topology-test-1-1529509694-stormconf.ser at org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:256) ~[storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.blobstore.LocalFsBlobStore.getBlobMeta(LocalFsBlobStore.java:286) ~[storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.daemon.nimbus.Nimbus.getBlobMeta(Nimbus.java:3483) [storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:4011) [storm-client-2.0.0.y.jar:2.0.0.y] at org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3990) [storm-client-2.0.0.y.jar:2.0.0.y] at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:38) [shaded-deps-2.0.0.y.jar:2.0.0.y] at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [shaded-deps-2.0.0.y.jar:2.0.0.y] at org.apache.storm.security.auth.sasl.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:147) [storm-client-2.0.0.y.jar:2.0.0.y] at org.apache.storm.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:291) [shaded-deps-2.0.0.y.jar:2.0.0.y] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] 2018-06-20 15:48:47.950 o.a.s.u.Utils Thread-12 [WARN] Forcing Halt...
Nimbus then continually restarts:
2018-06-20 15:48:54.635 o.a.s.u.Utils main [ERROR] Received error in main thread.. terminating server... java.lang.Error: java.lang.IllegalStateException: Could not find credentials for topology wc-topology-test-1-1529509694 at path /storms. Don't know how to fix this automatically. Please add needed ACLs, or delete the path. at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:603) ~[storm-client-2.0.0.y.jar:2.0.0.y] at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:582) ~[storm-client-2.0.0.y.jar:2.0.0.y] at org.apache.storm.utils.Utils$5.uncaughtException(Utils.java:931) [storm-client-2.0.0.y.jar:2.0.0.y] at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057) [?:1.8.0_131] at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052) [?:1.8.0_131] at java.lang.Thread.dispatchUncaughtException(Thread.java:1959) [?:1.8.0_131] Caused by: java.lang.IllegalStateException: Could not find credentials for topology wc-topology-test-1-1529509694 at path /storms. Don't know how to fix this automatically. Please add needed ACLs, or delete the path. at org.apache.storm.zookeeper.AclEnforcement.getTopoAcl(AclEnforcement.java:194) ~[storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.zookeeper.AclEnforcement.verifyParentWithTopoChildren(AclEnforcement.java:250) ~[storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.zookeeper.AclEnforcement.verifyParentWithReadOnlyTopoChildren(AclEnforcement.java:258) ~[storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.zookeeper.AclEnforcement.verifyAcls(AclEnforcement.java:136) ~[storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.daemon.nimbus.Nimbus.launch(Nimbus.java:1155) ~[storm-server-2.0.0.y.jar:2.0.0.y] at org.apache.storm.daemon.nimbus.Nimbus.main(Nimbus.java:1162) ~[storm-server-2.0.0.y.jar:2.0.0.y]
Attachments
Issue Links
- links to