Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-881

Re-think the Samza Job Coordinator

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently, the only way to run Samza containers in distributed mode is using Yarn. However, there has been interest to run Samza on top of other resource managers with the recent explosion in the # of such systems. Users have also requested us to run Samza as a library, and to run Samza on Docker containers managed by Kubernetes.

      We must re-think the JobCoordinator functionality as follows:
      1. ID assignment: Provide an ID to each SamzaContainer.
      2. JobModel agreement: Ensure containers agree on a JobModel.
      3. Re-start the SamzaContainer when the job model changes.

      This will arguably require some leader election (depending on how users choose to run Samza).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jagadish1989@gmail.com Jagadish
                Reporter:
                jagadish1989@gmail.com Jagadish
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: