[SAMZA-1482] Restart or fail Samza jobs in YARN when input topic partition changes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Assignee: Yi Pan
• Reporter: Yi Pan
• Created: 07/Nov/17 23:50
• Updated: 20/Nov/17 19:50
• Resolved: 20/Nov/17 19:50

Description

Currently, after a Samza job is started, it works only on a set of fixed input topic partitions at the start-up time. When input topic partitions are expanded, we often lose the messages sent in the new partitions, until we restart the job.

SAMZA-882 added a input stream partition count monitor inside the JobCoordinator. This ticket is targeted to use this monitor metrics and trigger the following actions in YARN:

1. for stateless jobs, shutdown the JobCoordinator w/ UNDEFINED status code s.t. YARN will restart the whole job
2. for stateful jobs, shutdown the JobCoordinator w/ FAILED status code s.t. YARN will stop the whole job

Issue Links

• links to: GitHub Pull Request #351
• links to: GitHub Pull Request #363