Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-3733

AsyncLocalizer stuck looking for missing topology

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.3.0
    • Component/s: None
    • Labels:
      None

      Description

      2020-11-09 20:18:12.325 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 2 [ERROR] Could not update blob, will retry again later
      2020-11-09 20:18:43.744 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 2 [ERROR] Could not update blob, will retry again later
      2020-11-09 20:19:14.726 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 2 [ERROR] Could not update blob, will retry again later
      2020-11-09 20:19:46.148 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 2 [ERROR] Could not update blob, will retry again later
      2020-11-09 20:20:16.560 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 0 [ERROR] Could not update blob, will retry again later
      2020-11-09 20:20:47.990 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 0 [ERROR] Could not update blob, will retry again later
      2020-11-09 20:21:19.403 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 0 [ERROR] Could not update blob, will retry again later
      2020-11-09 20:21:50.818 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 0 [ERROR] Could not update blob, will retry again later
      2020-11-09 20:22:21.257 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 1 [ERROR] Could not update blob, will retry again later
      2020-11-09 20:22:52.668 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 1 [ERROR] Could not update blob, will retry again later
      2020-11-09 20:23:24.082 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 1 [ERROR] Could not update blob, will retry again later
      2020-11-09 20:23:55.512 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 1 [ERROR] Could not update blob, will retry again later
      2020-11-09 20:24:25.919 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 2 [ERROR] Could not update blob, will retry again later
      2020-11-09 20:24:57.343 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 2 [ERROR] Could not update blob, will retry again later
      2020-11-09 20:25:28.773 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 2 [ERROR] Could not update blob, will retry again later
      
      2020-11-09 20:16:09.659 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 1 [ERROR] Could not update blob, will retry again later
      java.util.concurrent.ExecutionException: java.lang.RuntimeException: Could not download...
              at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_262]
              at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_262]
              at org.apache.storm.localizer.AsyncLocalizer.updateBlobs(AsyncLocalizer.java:333) ~[storm-server-2.3.0.y.jar:2.3.0.y]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_262]
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_262]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_262]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_262]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_262]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_262]
              at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]
      Caused by: java.lang.RuntimeException: Could not download...
              at org.apache.storm.localizer.AsyncLocalizer.lambda$downloadOrUpdate$10(AsyncLocalizer.java:297) ~[storm-server-2.3.0.y.jar:2.3.0.y]
              at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640) ~[?:1.8.0_262]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_262]
              at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_262]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_262]
              ... 3 more
      Caused by: org.apache.storm.utils.WrappedKeyNotFoundException: testTopology-743-1603821880-stormconf.ser
              at org.apache.storm.hdfs.blobstore.HdfsBlobStore.getStoredBlobMeta(HdfsBlobStore.java:192) ~[storm-hdfs-blobstore-2.3.0.y.jar:2.3.0.y]
              at org.apache.storm.hdfs.blobstore.HdfsBlobStore.getBlobMeta(HdfsBlobStore.java:221) ~[storm-hdfs-blobstore-2.3.0.y.jar:2.3.0.y]
              at org.apache.storm.hdfs.blobstore.HdfsClientBlobStore.getBlobMeta(HdfsClientBlobStore.java:73) ~[storm-hdfs-blobstore-2.3.0.y.jar:2.3.0.y]
              at org.apache.storm.localizer.LocallyCachedTopologyBlob.getRemoteVersion(LocallyCachedTopologyBlob.java:129) ~[storm-server-2.3.0.y.jar:2.3.0.y]
              at org.apache.storm.localizer.AsyncLocalizer.lambda$downloadOrUpdate$10(AsyncLocalizer.java:273) ~[storm-server-2.3.0.y.jar:2.3.0.y]
              at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640) ~[?:1.8.0_262]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_262]
              at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_262]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_262]
              ... 3 more
      

        Attachments

          Activity

            People

            • Assignee:
              agresch Aaron Gresch
              Reporter:
              agresch Aaron Gresch
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 50m
                50m