Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-865

REST API for starting and stopping Samza jobs.

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: None
    • Labels:

      Description

      The shell scripts we currently use to start and stop jobs are mostly specific to YARN and are not well documented APIs. This puts some burden on developers, which may develop a Samza solution with a YARN alternative or try to integrate Samza with other services like dashboards.

      This ticket is to discuss the addition of a REST API to start, stop and get status for jobs. Such an API could be extended for other use cases like modifying checkpoints, getting stack traces, getting logs, etc.

      1. DESIGN-SAMZA-865-3.pdf
        87 kB
        Jake Maes
      2. DESIGN-SAMZA-865-3.md
        13 kB
        Jake Maes
      3. DESIGN-SAMZA-865-2.pdf
        86 kB
        Jake Maes
      4. DESIGN-SAMZA-865-2.md
        13 kB
        Jake Maes
      5. DESIGN-SAMZA-865-1.pdf
        75 kB
        Jake Maes
      6. DESIGN-SAMZA-865-1.md
        10 kB
        Jake Maes

        Activity

        Hide
        jmakes Jake Maes added a comment -

        Great feedback, thanks!

        Yi Pan (Data Infrastructure)
        1. You're right, the design doc wasn't clear on this. In fact, the REST service is mostly geared toward the "Samza-as-a-service" model, which is more likely to focus on a hosted service rather than standalone. I clarified that the real use case is to have a common API regardless of the underlying cluster implementation, so as to enable generic tools, dashboards, etc.
        2. Great suggestions. Added them.
        3. That's right. I suppose I mentioned it in case any "rebels" decided to use an older version. But if that happens, they may also encounter other problems due to the YARN version. It has been removed.
        4. I was worried these would make it too verbose, but it's not too bad. See version 2 of the design doc.
        5. Yes. Actually the doc was conflating the "central hosts" and RM hosts. In the event that they're separate hosts, the Installation Finder can be adapted to discover jobs wherever they are stored.
        6. Good point. The "Packaging" wiki only calls out that the configuration is NOT part of the job package. I'll update the design doc to describe the default assumption and that the InstallationFinder can be used to adapt to any deviation.

        Xinyu Liu, thanks, I totally missed that!

        Show
        jmakes Jake Maes added a comment - Great feedback, thanks! Yi Pan (Data Infrastructure) 1. You're right, the design doc wasn't clear on this. In fact, the REST service is mostly geared toward the "Samza-as-a-service" model, which is more likely to focus on a hosted service rather than standalone. I clarified that the real use case is to have a common API regardless of the underlying cluster implementation, so as to enable generic tools, dashboards, etc. 2. Great suggestions. Added them. 3. That's right. I suppose I mentioned it in case any "rebels" decided to use an older version. But if that happens, they may also encounter other problems due to the YARN version. It has been removed. 4. I was worried these would make it too verbose, but it's not too bad. See version 2 of the design doc. 5. Yes. Actually the doc was conflating the "central hosts" and RM hosts. In the event that they're separate hosts, the Installation Finder can be adapted to discover jobs wherever they are stored. 6. Good point. The "Packaging" wiki only calls out that the configuration is NOT part of the job package. I'll update the design doc to describe the default assumption and that the InstallationFinder can be used to adapt to any deviation. Xinyu Liu , thanks, I totally missed that!
        Hide
        xinyu Xinyu Liu added a comment -

        +1 on the REST API design. The API is very useful for admin and remote access. One comment, I saw the response from the API has the "status" and "statusDetail". Could you please provide more details about them, like all the possible status or statusDetail a job might have? Thanks!

        Show
        xinyu Xinyu Liu added a comment - +1 on the REST API design. The API is very useful for admin and remote access. One comment, I saw the response from the API has the "status" and "statusDetail". Could you please provide more details about them, like all the possible status or statusDetail a job might have? Thanks!
        Hide
        nickpan47 Yi Pan (Data Infrastructure) added a comment -

        Hi, Jake Maes, sorry for the late review. I took a look at the design doc and it looks good overall. I have a few suggestions/questions listed below:

        1. For deployment tooling, what do you vision as the use case of “common interface to start, stop, and list jobs … in a mixed environment”? It is a bit hard to imagine how the RESTful APIs will work for standalone jobs where there is no central team manage them.
        2. In Future Use Cases, the following could be good examples as well:
          1. Dynamic log level changes
          2. Dynamic configuration changes
        3. In the “Leverage YARN Resource Manager REST API” section: It requires a minimum YARN version of 2.5.1 ==> we should remove this since we are already on 2.6.1 and above
        4. In the API design, it would be good to mention the error codes returned as well, in addition to success responses
        5. A quick question: wouldn’t the abstraction of InstallationFinder also removes the strict assumption that job packages are installed on a central host? It can be imaged that there could be a “central” repository to register all installed Samza job packages, not on a single host, then some script to download and launch those jobs in Mesos or Kubernetes.
        6. It would also be good to mention how we expect the configuration be packaged and deployed. This part is a little fuzzy in the design doc. The current assumption is that the deployment of a job also include deployment of the configuration from a “central repository” together w/ the job.

        Thanks a lot!

        Show
        nickpan47 Yi Pan (Data Infrastructure) added a comment - Hi, Jake Maes , sorry for the late review. I took a look at the design doc and it looks good overall. I have a few suggestions/questions listed below: For deployment tooling, what do you vision as the use case of “common interface to start, stop, and list jobs … in a mixed environment”? It is a bit hard to imagine how the RESTful APIs will work for standalone jobs where there is no central team manage them. In Future Use Cases, the following could be good examples as well: Dynamic log level changes Dynamic configuration changes In the “Leverage YARN Resource Manager REST API” section: It requires a minimum YARN version of 2.5.1 ==> we should remove this since we are already on 2.6.1 and above In the API design, it would be good to mention the error codes returned as well, in addition to success responses A quick question: wouldn’t the abstraction of InstallationFinder also removes the strict assumption that job packages are installed on a central host? It can be imaged that there could be a “central” repository to register all installed Samza job packages, not on a single host, then some script to download and launch those jobs in Mesos or Kubernetes. It would also be good to mention how we expect the configuration be packaged and deployed. This part is a little fuzzy in the design doc. The current assumption is that the deployment of a job also include deployment of the configuration from a “central repository” together w/ the job. Thanks a lot!
        Hide
        jmakes Jake Maes added a comment -

        Attaching initial design doc.

        The proposed API for start, stop, and list is probably of most interest to folks.

        Anyhow, feedback is appreciated!

        Show
        jmakes Jake Maes added a comment - Attaching initial design doc. The proposed API for start, stop, and list is probably of most interest to folks. Anyhow, feedback is appreciated!

          People

          • Assignee:
            jmakes Jake Maes
            Reporter:
            jmakes Jake Maes
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development