[STORM-3501] Local Cluster worker restarts

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0, 2.1.0
• Fix Version/s: 2.1.0
• Component/s: storm-server
• Labels: pull-request-available
• Environment: Linux
• Assignee: Diogo Monteiro
• Reporter: Diogo Monteiro
• Created: 28/Aug/19 13:34
• Updated: 31/Aug/19 18:46
• Resolved: 31/Aug/19 17:21

Description

I was trying to launch a topology that I'm developing (in 2.0.0) and noticed that the worker was getting restarted each ~30 seconds. 
I placed a breakpoint in the kill method of LocalContainer (https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/daemon/supervisor/LocalContainer.java#L66) to try and understand why the worker was getting restarted. 
 
The call stack was:
_kill:66, LocalContainer (org.apache.storm.daemon.supervisor)
killContainerFor:269, Slot (org.apache.storm.daemon.supervisor)
handleRunning:724, Slot (org.apache.storm.daemon.supervisor)
stateMachineStep:218, Slot (org.apache.storm.daemon.supervisor)
run:931, Slot (org.apache.storm.daemon.supervisor) _
 
With this I can understand that the worker is killed because a blob has changed (https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/daemon/supervisor/Slot.java#L724). In fact, there's a changing blob in the dynamicState at that point.
 
I checked the AsyncLocalizer which downloads, caches blobs locally, and notifies the Slot state machine of a changing blob.
 
I noticed this:

• https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/localizer/AsyncLocalizer.java#L339
• https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/localizer/AsyncLocalizer.java#L265
• https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/localizer/LocallyCachedTopologyBlob.java#L142
• https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/localizer/LocallyCachedTopologyBlob.java#L192
   

Which tell me that (correct me if I'm wrong):

• Supervisor tries to update blobs each 30 seconds.
• The topology jar blob requires extraction of the resources directory (either from a jar or directly in a classpath URL). It does so in fetchUnzipToTemp and it's existence is checked in isFullyDownloaded.
• The Slot is notified of a changing blob if:
• the remote version is different from the local version (the code has changed).
• OR the blob is not fully downloaded (the jar exists, and the extracted resources directory exists).

 
Well, I did not have a resources folder under the root of the classpath, and that's why the worker was being restarted each ~30 seconds, as the Slot was being notified of a changing blob everytime updateBlobs ran. 
I created a resources folder (with dummy files) under the root of the classpath and the problem is now solved.
 
However, if I understand correctly, the resources folder is only required for multilang. Our topologies do not use multilang and this do not happen in Storm 1.1.3 for instance.

 

Happy to submit MR.

Issue Links

• links to: GitHub Pull Request #3120