[SAMZA-881] Re-think the Samza Job Coordinator

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Jagadish
• Reporter: Jagadish
• Created: 29/Feb/16 19:30
• Updated: 23/Apr/16 00:05

Description

Currently, the only way to run Samza containers in distributed mode is using Yarn. However, there has been interest to run Samza on top of other resource managers with the recent explosion in the # of such systems. Users have also requested us to run Samza as a library, and to run Samza on Docker containers managed by Kubernetes.

We must re-think the JobCoordinator functionality as follows:
1. ID assignment: Provide an ID to each SamzaContainer.
2. JobModel agreement: Ensure containers agree on a JobModel.
3. Re-start the SamzaContainer when the job model changes.

This will arguably require some leader election (depending on how users choose to run Samza).

Issue Links

• relates to: SAMZA-680 Invert the JobCoordinator and AM logic
• relates to: SAMZA-375 Investigate Mesos Job Support
• relates to: SAMZA-516 Support standalone Samza jobs
• relates to: SAMZA-41 Support static partition assignment in LocalJobFactory
• relates to: SAMZA-871 Implement heart-beat mechanism between JobCoordinator and all running containers

Sub-Tasks

• Invert the JobCoordinator and AM logic: Resolved
• Refactor the UI state variables: Resolved
• Refactor all unit tests to work with the new JobCoordinator APIs: Open
• Documentation updates for refactored JC: Resolved