[STORM-3556] blob meta exception in Nimbus log

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.1.0
• Component/s: storm-server
• Assignee: Unassigned
• Reporter: AK47Sonic
• Created: 18/Dec/19 07:35
• Updated: 18/Dec/19 07:36

Description

Hi team,

When I kill topology on the UI, [WARN] get blob meta exception will be thrown in the Nimbus log. Is it a bug?

2019-12-18 02:14:16.782 o.a.s.d.n.Nimbus pool-29-thread-46 [INFO] Created download session ec557d66-96d9-4071-b99f-3122fa50f61c for word-count-9-1576653249-stormjar.jar
2019-12-18 02:14:16.794 o.a.s.d.n.Nimbus pool-29-thread-50 [INFO] Created download session 1bbae64e-89c6-4337-b87b-634dbb78f61c for word-count-9-1576653249-stormconf.ser
2019-12-18 02:14:16.807 o.a.s.d.n.Nimbus pool-29-thread-40 [INFO] Created download session a3f78bd3-673f-4780-a402-90708bd754f7 for word-count-9-1576653249-stormcode.ser
2019-12-18 02:14:17.472 o.a.s.d.n.Nimbus pool-29-thread-18 [INFO] Created download session a05848fa-e83e-4c13-b025-f14a468acf14 for word-count-9-1576653249-stormjar.jar
2019-12-18 02:14:17.483 o.a.s.d.n.Nimbus pool-29-thread-58 [INFO] Created download session 9119de95-cd34-48a4-b7ef-ae0f84de5689 for word-count-9-1576653249-stormcode.ser
2019-12-18 02:14:17.495 o.a.s.d.n.Nimbus pool-29-thread-29 [INFO] Created download session d8d06e2e-e82c-46a8-beee-ee8bf313e584 for word-count-9-1576653249-stormconf.ser
2019-12-18 02:14:17.538 o.a.s.d.n.Nimbus pool-29-thread-21 [INFO] Created download session f634e74b-a161-4295-bf17-bd631b3d6e6d for word-count-9-1576653249-stormjar.jar
2019-12-18 02:14:17.546 o.a.s.d.n.Nimbus pool-29-thread-18 [INFO] Created download session 29e1592a-136a-4ff6-a4de-9c4420d55849 for word-count-9-1576653249-stormcode.ser
2019-12-18 02:14:17.557 o.a.s.d.n.Nimbus pool-29-thread-48 [INFO] Created download session ca75e5b0-e855-4b5f-962f-f7accefdc260 for word-count-9-1576653249-stormconf.ser
2019-12-18 02:23:42.461 o.a.s.d.n.Nimbus pool-29-thread-36 [INFO] TRANSITION: word-count-9-1576653249 KILL 5 true
2019-12-18 02:23:42.463 o.a.s.d.n.Nimbus pool-29-thread-36 [INFO] Delaying event REMOVE for 5 secs for word-count-9-1576653249
2019-12-18 02:23:42.471 o.a.s.d.n.Nimbus pool-29-thread-36 [INFO] Adding topo to history log: word-count-9-1576653249
2019-12-18 02:23:47.464 o.a.s.d.n.Nimbus timer [INFO] TRANSITION: word-count-9-1576653249 REMOVE null false
2019-12-18 02:23:47.467 o.a.s.d.n.Nimbus timer [INFO] Killing topology: word-count-9-1576653249
2019-12-18 02:25:58.331 o.a.s.d.n.Nimbus timer [INFO] Cleaning up word-count-9-1576653249
2019-12-18 02:25:58.338 o.a.s.c.StormClusterStateImpl timer [INFO] Removing worker keys under /secretkeys/NIMBUS/word-count-9-1576653249
2019-12-18 02:25:58.339 o.a.s.c.StormClusterStateImpl timer [INFO] Removing worker keys under /secretkeys/DRPC/word-count-9-1576653249
2019-12-18 02:25:58.340 o.a.s.c.StormClusterStateImpl timer [INFO] Removing worker keys under /secretkeys/SUPERVISOR/word-count-9-1576653249
2019-12-18 02:25:58.341 o.a.s.d.n.Nimbus timer [INFO] Removing dependency jars from blobs - []
2019-12-18 02:26:04.774 o.a.s.d.n.Nimbus pool-29-thread-61 [WARN] get blob meta exception.
org.apache.storm.utils.WrappedKeyNotFoundException: word-count-9-1576653249-stormcode.ser
at org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:257) ~[storm-server-2.1.0.jar:2.1.0]
at org.apache.storm.blobstore.LocalFsBlobStore.getBlobMeta(LocalFsBlobStore.java:287) ~[storm-server-2.1.0.jar:2.1.0]
at org.apache.storm.daemon.nimbus.Nimbus.getBlobMeta(Nimbus.java:3651) [storm-server-2.1.0.jar:2.1.0]
at org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3951) [storm-client-2.1.0.jar:2.1.0]
at org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3930) [storm-client-2.1.0.jar:2.1.0]
at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:38) [storm-shaded-deps-2.1.0.jar:2.1.0]
at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [storm-shaded-deps-2.1.0.jar:2.1.0]
at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:174) [storm-client-2.1.0.jar:2.1.0]
at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) [storm-shaded-deps-2.1.0.jar:2.1.0]
at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) [storm-shaded-deps-2.1.0.jar:2.1.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
2019-12-18 02:26:04.784 o.a.s.d.n.Nimbus pool-29-thread-5 [WARN] get blob meta exception.
org.apache.storm.utils.WrappedKeyNotFoundException: word-count-9-1576653249-stormjar.jar
at org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:257) ~[storm-server-2.1.0.jar:2.1.0]
at org.apache.storm.blobstore.LocalFsBlobStore.getBlobMeta(LocalFsBlobStore.java:287) ~[storm-server-2.1.0.jar:2.1.0]
at org.apache.storm.daemon.nimbus.Nimbus.getBlobMeta(Nimbus.java:3651) [storm-server-2.1.0.jar:2.1.0]
at org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3951) [storm-client-2.1.0.jar:2.1.0]
at org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3930) [storm-client-2.1.0.jar:2.1.0]
at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:38) [storm-shaded-deps-2.1.0.jar:2.1.0]
at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [storm-shaded-deps-2.1.0.jar:2.1.0]
at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:174) [storm-client-2.1.0.jar:2.1.0]
at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) [storm-shaded-deps-2.1.0.jar:2.1.0]
at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) [storm-shaded-deps-2.1.0.jar:2.1.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]