[STORM-3701] Race Condition between cleanup thread and download tasks

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 2.1.0, 2.2.0
• Fix Version/s: 2.3.0
• Component/s: storm-server
• Assignee: Rui Li
• Reporter: Rui Li
• Created: 16/Sep/20 15:43
• Updated: 22/Sep/20 03:45
• Resolved: 21/Sep/20 18:39

Description

We captured a issue on our supervisor:

2020-06-09 23:30:08.723 o.a.s.l.LocalizedResource AsyncLocalizer Task Executor - 0 [INFO] completelyRemoveUnusedUser directu for directory /home/y/var/storm/supervisor/usercache/directu
 2020-06-09 23:30:08.724 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - 0 [WARN] Caught Exception While Downloading (rethrowing)...
 java.io.FileNotFoundException: File '/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser' does not exist
         at org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297) ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
         at org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851) ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
         at org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:316) ~[storm-client-2.3.0.y.jar:2.3.0.y]
         at org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:477) ~[storm-client-2.3.0.y.jar:2.3.0.y]
         at org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:311) ~[storm-client-2.3.0.y.jar:2.3.0.y]
         at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.get(AsyncLocalizer.java:698) [storm-server-2.3.0.y.jar:2.3.0.y]
         at org.apache.storm.localizer.AsyncLocalizer$DownloadBlobs.get(AsyncLocalizer.java:683) [storm-server-2.3.0.y.jar:2.3.0.y]
         at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) [?:1.8.0_242]
         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_242]
         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_242]
         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_242]
         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_242]
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
         at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
 2020-06-09 23:30:08.725 o.a.s.d.s.Slot SLOT_6782 [ERROR] java.io.FileNotFoundException: File '/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser' does not exist
 2020-06-09 23:30:08.725 o.a.s.l.AsyncLocalizer SLOT_6782 [INFO] Port and assignment info: PortAndAssignmentImpl\{dg_itp-605-1591745383 on 6782}
 2020-06-09 23:30:08.726 o.a.s.l.AsyncLocalizer SLOT_6782 [WARN] Local base blobs have not been downloaded yet.
 java.io.FileNotFoundException: File '/home/y/var/storm/supervisor/stormdist/dg_itp-605-1591745383/stormconf.ser' does not exist
         at org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297) ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
         at org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851) ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
         at org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:316) ~[storm-client-2.3.0.y.jar:2.3.0.y]
         at org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:477) ~[storm-client-2.3.0.y.jar:2.3.0.y]
         at org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:311) ~[storm-client-2.3.0.y.jar:2.3.0.y]
         at org.apache.storm.localizer.AsyncLocalizer.getLocalResources(AsyncLocalizer.java:362) ~[storm-server-2.3.0.y.jar:2.3.0.y]
         at org.apache.storm.localizer.AsyncLocalizer.releaseSlotFor(AsyncLocalizer.java:472) [storm-server-2.3.0.y.jar:2.3.0.y]
         at org.apache.storm.daemon.supervisor.Slot.handleWaitingForBlobLocalization(Slot.java:549) [storm-server-2.3.0.y.jar:2.3.0.y]
         at org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:298) [storm-server-2.3.0.y.jar:2.3.0.y]
         at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:1039) [storm-server-2.3.0.y.jar:2.3.0.y]

The root issue is the delay at https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/localizer/AsyncLocalizer.java#L641 which will cause the safeTopologyIds information out-of-date.