Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Currently, a fixed number of containers is assigned to a job as an input configuration parameter. However, with this design jobs can fail due to lack of enough resources (such as memory), or they can become a bottleneck in a workflow containing many jobs. While auto-scaling is much broader term, the goal of this project will be to enable a Samza job to automatically scale its containers such that there is improved job performance.

      1. DESIGN-SAMZA-719-v2.md
        48 kB
        Shadi A. Noghabi
      2. DESIGN-SAMZA-719-v2.pdf
        592 kB
        Shadi A. Noghabi
      3. DESIGN-SAMZA-719.pdf
        844 kB
        Shadi A. Noghabi

        Issue Links

          Activity

          Hide
          xinyu Xinyu Liu added a comment -

          realdinghao: Richard T. B. Ma and his students at National Univ of Singapore is continuing this work. It will be really helpful if you can give us more details about your use cases, e.g. your data processing flow and the requirements you have for for auto-scaling. We don't have a timeline for this feature to be available in Samza releases yet, but with Shadi A. Noghabi's existing work, I expect this may happen in the next few months. Thanks.

          Show
          xinyu Xinyu Liu added a comment - realdinghao : Richard T. B. Ma and his students at National Univ of Singapore is continuing this work. It will be really helpful if you can give us more details about your use cases, e.g. your data processing flow and the requirements you have for for auto-scaling. We don't have a timeline for this feature to be available in Samza releases yet, but with Shadi A. Noghabi 's existing work, I expect this may happen in the next few months. Thanks.
          Hide
          realdinghao realdinghao added a comment -

          I think this feature is very important. I wonder when this feature would be included in the main version of samza

          Show
          realdinghao realdinghao added a comment - I think this feature is very important. I wonder when this feature would be included in the main version of samza
          Hide
          snoghabi Shadi A. Noghabi added a comment -

          The RB can be found in https://reviews.apache.org/r/36471/. This is still a work in progress RB.

          Show
          snoghabi Shadi A. Noghabi added a comment - The RB can be found in https://reviews.apache.org/r/36471/ . This is still a work in progress RB.
          Hide
          snoghabi Shadi A. Noghabi added a comment -

          Design document added.

          Show
          snoghabi Shadi A. Noghabi added a comment - Design document added.
          Hide
          snoghabi Shadi A. Noghabi added a comment - - edited

          As part of this task SAMZA-704 is creating a tool to write to the coordinator stream. This tool can be used to write the new number of containers to the coordinator stream.

          Additionally, SAMZA-724 reads new configs from the coordinator stream and reacts accordingly. With this tool the new number of containers can be read and the job can scale up/down accordingly.

          Show
          snoghabi Shadi A. Noghabi added a comment - - edited As part of this task SAMZA-704 is creating a tool to write to the coordinator stream. This tool can be used to write the new number of containers to the coordinator stream. Additionally, SAMZA-724 reads new configs from the coordinator stream and reacts accordingly. With this tool the new number of containers can be read and the job can scale up/down accordingly.

            People

            • Assignee:
              snoghabi Shadi A. Noghabi
              Reporter:
              snoghabi Shadi A. Noghabi
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:

                Development