[STORM-682] Supervisor local worker state corrupted and failing to start.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0, 0.9.4
• Component/s: storm-core
• Assignee: Parth Brahmbhatt
• Reporter: Parth Brahmbhatt
• Created: 20/Feb/15 18:56
• Updated: 05/Oct/15 01:03
• Resolved: 18/Mar/15 15:07

Description

If supervisor's cleanup of a worker fails to delete some heartbeat files the local state of the supervisors get corrupted.The only way to recover the supervisor from this state is to delete the local state folder where supervisor stores all worker information.This fix can get very cumbersome if it happens on multiple worker nodes.

The root cause of the issue is the order in which worker heartbeat versioned store files are created vs the deletion order of those files. LocalState.put first creates a data file X and then marks a success by creating a file X.version. During get it first checks for all *.version files , tries to find the largest value of X and then issues a read against X. See the below pseudo code

start_supervisor() {
workerIds = `ls local-state/workers`
for each workerId in workerIds
     versions =  `ls local-state/workers/workerId/heartbeats/*.version`
     latest_version = max(versions)
     read  local-state/workers/workerId/heartbeats/latest_version [Note there is no .version extension] 
}

During cleanup it first tries to delete file X and then X.version. If X gets deleted but X.version fails to delete the supervisor fails to start with FileNotFoundException in the code above.

We propose to change the deletion order so the .version files get deleted before the data file and catch any IOException when reading worker heartbeats to avoid supervisor failure.