[STORM-3733] AsyncLocalizer stuck looking for missing topology

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Assignee: Aaron Gresch
• Reporter: Aaron Gresch
• Created: 05/Jan/21 17:15
• Updated: 12/Jan/21 16:03
• Resolved: 12/Jan/21 16:03

Description

2020-11-09 20:18:12.325 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 2 [ERROR] Could not update blob, will retry again later
2020-11-09 20:18:43.744 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 2 [ERROR] Could not update blob, will retry again later
2020-11-09 20:19:14.726 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 2 [ERROR] Could not update blob, will retry again later
2020-11-09 20:19:46.148 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 2 [ERROR] Could not update blob, will retry again later
2020-11-09 20:20:16.560 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 0 [ERROR] Could not update blob, will retry again later
2020-11-09 20:20:47.990 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 0 [ERROR] Could not update blob, will retry again later
2020-11-09 20:21:19.403 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 0 [ERROR] Could not update blob, will retry again later
2020-11-09 20:21:50.818 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 0 [ERROR] Could not update blob, will retry again later
2020-11-09 20:22:21.257 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 1 [ERROR] Could not update blob, will retry again later
2020-11-09 20:22:52.668 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 1 [ERROR] Could not update blob, will retry again later
2020-11-09 20:23:24.082 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 1 [ERROR] Could not update blob, will retry again later
2020-11-09 20:23:55.512 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 1 [ERROR] Could not update blob, will retry again later
2020-11-09 20:24:25.919 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 2 [ERROR] Could not update blob, will retry again later
2020-11-09 20:24:57.343 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 2 [ERROR] Could not update blob, will retry again later
2020-11-09 20:25:28.773 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 2 [ERROR] Could not update blob, will retry again later

2020-11-09 20:16:09.659 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 1 [ERROR] Could not update blob, will retry again later
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Could not download...
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_262]
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_262]
        at org.apache.storm.localizer.AsyncLocalizer.updateBlobs(AsyncLocalizer.java:333) ~[storm-server-2.3.0.y.jar:2.3.0.y]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_262]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_262]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_262]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_262]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_262]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_262]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]
Caused by: java.lang.RuntimeException: Could not download...
        at org.apache.storm.localizer.AsyncLocalizer.lambda$downloadOrUpdate$10(AsyncLocalizer.java:297) ~[storm-server-2.3.0.y.jar:2.3.0.y]
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640) ~[?:1.8.0_262]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_262]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_262]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_262]
        ... 3 more
Caused by: org.apache.storm.utils.WrappedKeyNotFoundException: testTopology-743-1603821880-stormconf.ser
        at org.apache.storm.hdfs.blobstore.HdfsBlobStore.getStoredBlobMeta(HdfsBlobStore.java:192) ~[storm-hdfs-blobstore-2.3.0.y.jar:2.3.0.y]
        at org.apache.storm.hdfs.blobstore.HdfsBlobStore.getBlobMeta(HdfsBlobStore.java:221) ~[storm-hdfs-blobstore-2.3.0.y.jar:2.3.0.y]
        at org.apache.storm.hdfs.blobstore.HdfsClientBlobStore.getBlobMeta(HdfsClientBlobStore.java:73) ~[storm-hdfs-blobstore-2.3.0.y.jar:2.3.0.y]
        at org.apache.storm.localizer.LocallyCachedTopologyBlob.getRemoteVersion(LocallyCachedTopologyBlob.java:129) ~[storm-server-2.3.0.y.jar:2.3.0.y]
        at org.apache.storm.localizer.AsyncLocalizer.lambda$downloadOrUpdate$10(AsyncLocalizer.java:273) ~[storm-server-2.3.0.y.jar:2.3.0.y]
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640) ~[?:1.8.0_262]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_262]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_262]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_262]
        ... 3 more