[SAMZA-1158] Adding monitor to clean up stale local stores of tasks.

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.13.0
• Fix Version/s: 0.13.0
• Assignee: Shanthoosh Venkataraman
• Reporter: Shanthoosh Venkataraman
• Created: 20/Mar/17 19:11
• Updated: 17/Apr/17 21:54
• Resolved: 17/Apr/17 21:54

Description

Add samza-rest manager that periodically cleans up the stale local stores of dead jobs/tasks.

It performs the store deletion in two phases. Initially it deletes the offset file in the local task stores if the following condition is true. ((jobIsNotRunning || preferedHost != nmHost) && offsetFilelastModifiedTime is greater than deleteRetention).

During the subsequent run, it deletes the local task stores if it does not contain offset file.

Refer to the design doc of SAMZA-656 (https://issues.apache.org/jira/secure/attachment/12828083/DESIGN-SAMZA-656.pdf) for more details.