Details

    • Type: Bug
    • Status: Accepted
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: agent, c++ api, master
    • Labels:

      Description

      As mentioned in MESOS-938, one way to support task replacement and scaling could be to split the responsibility into several smaller primitives for 1) reducing complexity 2) Make it easier to comprehend and 3) easier and incremental in implementation.

      resizeTask() would be the primitive to either
      1) Scale a running task's resources down
      2) Scale a running task's resources up by using extra auxiliary offers.

        Issue Links

          Activity

          Hide
          tstclair Timothy St. Clair added a comment -

          I'm game for *this, unless you have something already.
          Also I think Ian Downes may be interested, b/c I don't think the capabilities have been added to the cgroups code on master.

          Show
          tstclair Timothy St. Clair added a comment - I'm game for *this, unless you have something already. Also I think Ian Downes may be interested, b/c I don't think the capabilities have been added to the cgroups code on master.
          Hide
          nnielsen Niklas Quarfot Nielsen added a comment -

          Sounds great! Go for it. Containerizers have to support Containerizer::update() - it's a part of the usual flow of:

          1) runTask comes in.
          2) If executor is not running, start it in a container. (Only with the resources given to start the executor)
          3) Meanwhile, add task as pending on executor.
          4) Executor registers with the slave and receive the task (Updates container resources to contain executor + task)

          Or did I misunderstand your concern?

          Show
          nnielsen Niklas Quarfot Nielsen added a comment - Sounds great! Go for it. Containerizers have to support Containerizer::update() - it's a part of the usual flow of: 1) runTask comes in. 2) If executor is not running, start it in a container. (Only with the resources given to start the executor) 3) Meanwhile, add task as pending on executor. 4) Executor registers with the slave and receive the task (Updates container resources to contain executor + task) Or did I misunderstand your concern?
          Hide
          nnielsen Niklas Quarfot Nielsen added a comment -

          Btw - you should also be looking for a shepherd

          Show
          nnielsen Niklas Quarfot Nielsen added a comment - Btw - you should also be looking for a shepherd
          Hide
          idownes Ian Downes added a comment -

          Containerizers should support update(Resources) which can be called at any time, either in the scenario Niklas Quarfot Nielsen describes (though I believe we do add the first task's resources into the executor's when launching?) or for frameworks that add/remove (multiple) tasks to a running executor.

          Current master cgroups code does support dynamic updating. It's brutal for memory in that the new limit is applied, regardless of the current usage, which could lead to immediate OOMs. We also envision that in the future some resources, i.e., filesystem space once enforced, could be non-decreasing on update.

          Show
          idownes Ian Downes added a comment - Containerizers should support update(Resources) which can be called at any time, either in the scenario Niklas Quarfot Nielsen describes (though I believe we do add the first task's resources into the executor's when launching?) or for frameworks that add/remove (multiple) tasks to a running executor. Current master cgroups code does support dynamic updating. It's brutal for memory in that the new limit is applied, regardless of the current usage, which could lead to immediate OOMs. We also envision that in the future some resources, i.e., filesystem space once enforced, could be non-decreasing on update.
          Hide
          idownes Ian Downes added a comment -

          I'd like to offer to shepherd this work, with the assistance of Benjamin Hindman. Can we start discussing how this could be implemented and where changes will be made?

          Show
          idownes Ian Downes added a comment - I'd like to offer to shepherd this work, with the assistance of Benjamin Hindman . Can we start discussing how this could be implemented and where changes will be made?
          Hide
          Yifan Yifan Gu added a comment - - edited

          Hey guys, I am interested in this too, and just put a proof of concept on the review board. Hope that will help us to push this feature.

          So currently, I only plan to support resizing a single task.

          When scale up the task, the master claimed the additional resources first, and wait for the resize reply, if it succeeds, the master simply forwards the reply to the framework.
          if it fails, the master recovers the resources of both the task and the allocator.

          When scale down the task, the master wait for the resize reply before alternate the resources of the task and the allocator.

          The link is:
          https://reviews.apache.org/r/22526/

          Thanks!

          Show
          Yifan Yifan Gu added a comment - - edited Hey guys, I am interested in this too, and just put a proof of concept on the review board. Hope that will help us to push this feature. So currently, I only plan to support resizing a single task. When scale up the task, the master claimed the additional resources first, and wait for the resize reply, if it succeeds, the master simply forwards the reply to the framework. if it fails, the master recovers the resources of both the task and the allocator. When scale down the task, the master wait for the resize reply before alternate the resources of the task and the allocator. The link is: https://reviews.apache.org/r/22526/ Thanks!
          Hide
          rkannan82 Kannan Rajah added a comment -

          This JIRA has not been updated in a year. This feature will benefit Apache Myriad which launches place holder tasks to represent the YARN containers. These container sizes will keep varying and so we need a way to resize the place holder tasks to avoid wasting resources. Currently, we are thinking of a workaround in Myriad itself. But I would like to know if there is a plan on when the feature will be implemented.

          Show
          rkannan82 Kannan Rajah added a comment - This JIRA has not been updated in a year. This feature will benefit Apache Myriad which launches place holder tasks to represent the YARN containers. These container sizes will keep varying and so we need a way to resize the place holder tasks to avoid wasting resources. Currently, we are thinking of a workaround in Myriad itself. But I would like to know if there is a plan on when the feature will be implemented.
          Hide
          qianzhang Qian Zhang added a comment -

          I am drafting a design doc for task resizing, will post it soon once the draft version is done.

          Show
          qianzhang Qian Zhang added a comment - I am drafting a design doc for task resizing, will post it soon once the draft version is done.
          Hide
          d.s.karthick Karthick Duraisamy Soundararaj added a comment - - edited

          Hey Qian Zhang, I am glad that there is some attention to this ticket. As you might have seen already, the attached patch does not work on the trunk/latest master branch. Although mesos has evolved and its design is different that what it was at the time when the attached patch was developed, parts of this could be reused. I would be happy to work with you on this patch or pass the master compatible version of this patch that I already have if it could help . Do you have a rough design draft that you could share with me?

          Show
          d.s.karthick Karthick Duraisamy Soundararaj added a comment - - edited Hey Qian Zhang , I am glad that there is some attention to this ticket. As you might have seen already, the attached patch does not work on the trunk/latest master branch. Although mesos has evolved and its design is different that what it was at the time when the attached patch was developed, parts of this could be reused. I would be happy to work with you on this patch or pass the master compatible version of this patch that I already have if it could help . Do you have a rough design draft that you could share with me?
          Hide
          adam-mesos Adam B added a comment -

          There were some issues/concerns that came about during the previous attempt, and you'll need a Shepherd to ensure that you don't fall into the same traps, and to help guide your design so that your fix can be accepted quickly.

          Ian Downes, are you (still) willing to shepherd this? (or maybe Niklas Quarfot Nielsen or Timothy St. Clair)

          Show
          adam-mesos Adam B added a comment - There were some issues/concerns that came about during the previous attempt, and you'll need a Shepherd to ensure that you don't fall into the same traps, and to help guide your design so that your fix can be accepted quickly. Ian Downes , are you (still) willing to shepherd this? (or maybe Niklas Quarfot Nielsen or Timothy St. Clair )
          Hide
          d.s.karthick Karthick Duraisamy Soundararaj added a comment - - edited

          Adam B Thanks for checking in. A Shepard would be of great help for guidance and feedback.

          Show
          d.s.karthick Karthick Duraisamy Soundararaj added a comment - - edited Adam B Thanks for checking in. A Shepard would be of great help for guidance and feedback.
          Hide
          qianzhang Qian Zhang added a comment -

          Karthick Duraisamy Soundararaj, here is the draft design doc I just finished, any comments are welcome
          https://docs.google.com/document/d/15rVmS2AXLzTDSEugAVDxWuHFUentp82KhL2yzxBCsi8/edit?usp=sharing

          Regarding the shepherd, I discussed with Niklas Quarfot Nielsen before, he would like to be the shepherd for this ticket, thanks Niklas Quarfot Nielsen

          Show
          qianzhang Qian Zhang added a comment - Karthick Duraisamy Soundararaj , here is the draft design doc I just finished, any comments are welcome https://docs.google.com/document/d/15rVmS2AXLzTDSEugAVDxWuHFUentp82KhL2yzxBCsi8/edit?usp=sharing Regarding the shepherd, I discussed with Niklas Quarfot Nielsen before, he would like to be the shepherd for this ticket, thanks Niklas Quarfot Nielsen
          Hide
          d.s.karthick Karthick Duraisamy Soundararaj added a comment -

          Qian Zhang Thanks. I have added my comments.

          Show
          d.s.karthick Karthick Duraisamy Soundararaj added a comment - Qian Zhang Thanks. I have added my comments.
          Hide
          d.s.karthick Karthick Duraisamy Soundararaj added a comment -

          Qian Zhang Are you actively working on this?

          Show
          d.s.karthick Karthick Duraisamy Soundararaj added a comment - Qian Zhang Are you actively working on this?
          Hide
          qianzhang Qian Zhang added a comment - - edited

          Yeah, but there are some design decisions need to be finalized after I discuss with shepherd, however, it seem Niklas Quarfot Nielsen does not have enough time on shepherding this ticket ...

          Show
          qianzhang Qian Zhang added a comment - - edited Yeah, but there are some design decisions need to be finalized after I discuss with shepherd, however, it seem Niklas Quarfot Nielsen does not have enough time on shepherding this ticket ...

            People

            • Assignee:
              qianzhang Qian Zhang
              Reporter:
              nnielsen Niklas Quarfot Nielsen
              Shepherd:
              Niklas Quarfot Nielsen
            • Votes:
              5 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

              • Created:
                Updated:

                Development