Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Currently, a fixed number of containers is assigned to a job as an input configuration parameter. However, with this design jobs can fail due to lack of enough resources (such as memory), or they can become a bottleneck in a workflow containing many jobs. While auto-scaling is much broader term, the goal of this project will be to enable a Samza job to automatically scale its containers such that there is improved job performance.

      1. DESIGN-SAMZA-719.pdf
        844 kB
        Shadi A. Noghabi
      2. DESIGN-SAMZA-719-v2.pdf
        592 kB
        Shadi A. Noghabi
      3. DESIGN-SAMZA-719-v2.md
        48 kB
        Shadi A. Noghabi

        Issue Links

          Activity

          Hide
          snoghabi Shadi A. Noghabi added a comment - - edited

          As part of this task SAMZA-704 is creating a tool to write to the coordinator stream. This tool can be used to write the new number of containers to the coordinator stream.

          Additionally, SAMZA-724 reads new configs from the coordinator stream and reacts accordingly. With this tool the new number of containers can be read and the job can scale up/down accordingly.

          Show
          snoghabi Shadi A. Noghabi added a comment - - edited As part of this task SAMZA-704 is creating a tool to write to the coordinator stream. This tool can be used to write the new number of containers to the coordinator stream. Additionally, SAMZA-724 reads new configs from the coordinator stream and reacts accordingly. With this tool the new number of containers can be read and the job can scale up/down accordingly.
          Hide
          snoghabi Shadi A. Noghabi added a comment -

          Design document added.

          Show
          snoghabi Shadi A. Noghabi added a comment - Design document added.
          Hide
          snoghabi Shadi A. Noghabi added a comment -

          The RB can be found in https://reviews.apache.org/r/36471/. This is still a work in progress RB.

          Show
          snoghabi Shadi A. Noghabi added a comment - The RB can be found in https://reviews.apache.org/r/36471/ . This is still a work in progress RB.

            People

            • Assignee:
              snoghabi Shadi A. Noghabi
              Reporter:
              snoghabi Shadi A. Noghabi
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development