[SAMZA-516] Support standalone Samza jobs

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.0
• Component/s: container
• Labels: design project
• Assignee: Chris Riccomini
• Reporter: Chris Riccomini
• Created: 22/Jan/15 00:54
• Updated: 17/Jan/17 21:25

Description

Samza currently supports two modes of operation out of the box: local and YARN. With local mode, a single Java process starts the JobCoordinator, creates a single container, and executes it locally. All partitions are procesed within this container. With YARN, a YARN grid is required to execute the Samza job. In addition, SAMZA-375 introduces a patch to run Samza in Mesos.

There have been several requests lately to be able to run Samza jobs without any resource manager (YARN, Mesos, etc), but still run it in a distributed fashion.

The goal of this ticket is to design and implement a samza-standalone module, which will:

1. Support executing a single Samza job in one or more containers.
2. Support failover, in cases where a machine is lost.

Issue Links

• is related to: SAMZA-1063 Samza Standalone Project
• is related to: SAMZA-881 Re-think the Samza Job Coordinator
• relates to: SAMZA-41 Support static partition assignment in LocalJobFactory

Sub-Tasks

• Write a ZK-based samza-standalone JobRunner: Open
• Add pausing and assignment delays to standalone JobRunner: Open
• Reuse local state in SamzaContainer on clean shutdown: Resolved
• Write a samza-standalone UI proxy: Open
• Expose logs in samza-standalone JobRunner: Open