Description
Samza currently supports two modes of operation out of the box: local and YARN. With local mode, a single Java process starts the JobCoordinator, creates a single container, and executes it locally. All partitions are procesed within this container. With YARN, a YARN grid is required to execute the Samza job. In addition, SAMZA-375 introduces a patch to run Samza in Mesos.
There have been several requests lately to be able to run Samza jobs without any resource manager (YARN, Mesos, etc), but still run it in a distributed fashion.
The goal of this ticket is to design and implement a samza-standalone module, which will:
- Support executing a single Samza job in one or more containers.
- Support failover, in cases where a machine is lost.
Attachments
Attachments
Issue Links
- is related to
-
SAMZA-1063 Samza Standalone Project
-
- In Progress
-
-
SAMZA-881 Re-think the Samza Job Coordinator
-
- Open
-
- relates to
-
SAMZA-41 Support static partition assignment in LocalJobFactory
-
- Resolved
-
1.
|
Write a ZK-based samza-standalone JobRunner |
|
Open | Chris Riccomini |
2.
|
Add pausing and assignment delays to standalone JobRunner |
|
Open | Unassigned |
3.
|
Reuse local state in SamzaContainer on clean shutdown |
|
Resolved | Navina Ramesh |
4.
|
Write a samza-standalone UI proxy |
|
Open | Unassigned |
5.
|
Expose logs in samza-standalone JobRunner |
|
Open | Unassigned |