[SAMZA-555] Write a ZK-based samza-standalone JobRunner

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.0
• Component/s: container
• Assignee: Chris Riccomini
• Reporter: Chris Riccomini
• Created: 11/Feb/15 17:03
• Updated: 18/Jan/17 23:10

Description

SAMZA-516 documents the design of a standalone Samza job, which uses ZK for liveness monitoring and partition assignment rather than YARN. The first step to implementing this design is to add a samza-standalone submodule in Samza, which implements a JobRunner that uses ZK to elect a leader for the job (JobCoordinator), and then start a container. When failures occur, the leader should shift partitions within ZK to move partitions off of the dead container. Likewise, when a new container is added, the leader should shift partitions to the new container. The SamzaContainer should remain immutable, and simply be bounced when partitions are added or removed from it.

Issue Links

• relates to: SAMZA-1064 Standalone Samza with Zookeeper for Coordination