[STORM-1946] ShellBolt.java - On busy system BoltHeartbeatTimerTask fires before setHeartbeat() is executed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0, 1.0.2, 1.1.0
• Component/s: storm-core
• Labels: patch
• Flags: Patch
• Epic Link: Release Apache Storm 1.0.2
• Assignee: Slava Andreyev
• Reporter: Slava Andreyev
• Created: 06/Jul/16 15:05
• Updated: 08/Jul/16 14:00
• Resolved: 08/Jul/16 07:24

Description

When storm stars a large number of ShellBolt-s that consume a lot of CPU time to initialize, it creates a lot of contention between processes for CPU resource. That leads to BoltHeartbeatTimerTask being fired up after 1 second delay before setHeartbeat() assigns initial value to lastHeartbeatTimestamp variable.
As a result when BoltHeartbeatTimeTask fires up for the first time, getLastHeartbeat() returns value of 0. This in turn leads bolt to die with "subprocess heartbeat timeout" message.
The fix is to place setHeartBeat() before BoltHeartbeatTimerTask is created. The patch for this is attached.

Issue Links

• links to: GitHub Pull Request #1542