Samza currently supports two modes of operation out of the box: local and YARN. With local mode, a single Java process starts the JobCoordinator, creates a single container, and executes it locally. All partitions are procesed within this container. With YARN, a YARN grid is required to execute the Samza job. In addition, SAMZA-375 introduces a patch to run Samza in Mesos.
There have been several requests lately to be able to run Samza jobs without any resource manager (YARN, Mesos, etc), but still run it in a distributed fashion.
The goal of this ticket is to design and implement a samza-standalone module, which will:
- Support executing a single Samza job in one or more containers.
- Support failover, in cases where a machine is lost.
|Write a ZK-based samza-standalone JobRunner||Open|
|Add pausing and assignment delays to standalone JobRunner||Open||Unassigned|
|Write a samza-standalone UI proxy||Open||Unassigned|
|Expose logs in samza-standalone JobRunner||Open||Unassigned|