Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-719 Auto-scaling containers in Samza
  3. SAMZA-724

Create a tool to read configuration changes after the the bootstrap of the job, and react to them accordingly

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: None
    • Labels:
      None

      Description

      After a job is submitted, it might need some configuration change, specifically it might need more containers. In SAMZA-704 a tool is being added to write to the coordinator stream. This tool can be used to write new configurations to the coordinator stream. However, another tool (ConfigManager) is needed to read the config changes and react to them, which is the goal of this task. This tool should be brought up after the job is submitted and read any config changes added to the coordinator stream, and react to each accordingly.

      Each configuration will need a different action, and the should be added one-by one.

      1. config-manager-separate-module-v2.patch
        36 kB
        Shadi A. Noghabi
      2. config-manager-separate-module-v4.patch
        37 kB
        Shadi A. Noghabi
      3. config-manager-separate-module-v5.patch
        38 kB
        Shadi A. Noghabi

        Activity

        Hide
        snoghabi Shadi A. Noghabi added a comment -

        Currently the main configuration targeted is changing the number of containers of the job, and the server URL where the job model is stored. The RB for this part can be found in https://reviews.apache.org/r/36006/

        Show
        snoghabi Shadi A. Noghabi added a comment - Currently the main configuration targeted is changing the number of containers of the job, and the server URL where the job model is stored. The RB for this part can be found in https://reviews.apache.org/r/36006/
        Hide
        snoghabi Shadi A. Noghabi added a comment - - edited

        Patch added
        Yi Pan (Data Infrastructure) Navina Ramesh Could you please review it again?

        Thanks!

        Show
        snoghabi Shadi A. Noghabi added a comment - - edited Patch added Yi Pan (Data Infrastructure) Navina Ramesh Could you please review it again? Thanks!
        Hide
        snoghabi Shadi A. Noghabi added a comment -

        Patch updated with the reviews applied

        Show
        snoghabi Shadi A. Noghabi added a comment - Patch updated with the reviews applied
        Hide
        navina Navina Ramesh added a comment -

        Verified and Committed. Thanks!

        Show
        navina Navina Ramesh added a comment - Verified and Committed. Thanks!

          People

          • Assignee:
            snoghabi Shadi A. Noghabi
            Reporter:
            snoghabi Shadi A. Noghabi
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development