Details

    • Epic Name:
      Optimistic Offers

      Description

      Background
      The current implementation of resource offers only enable a single framework scheduler to make scheduling decisions for some available resources at a time. In some circumstances, this is good, i.e., when we don't want other framework schedulers to have access to some resources. However, in other circumstances, there are advantages to letting multiple framework schedulers attempt to make scheduling decisions for the same allocation of resources in parallel.

      If you think about this from a "concurrency control" perspective, the current implementation of resource offers is pessimistic, the resources contained within an offer are locked until the framework scheduler that they were offered to launches tasks with them or declines them. In addition to making pessimistic offers we'd like to give out optimistic offers, where the same resources are offered to multiple framework schedulers at the same time, and framework schedulers "compete" for those resources on a first-come-first-serve basis (i.e., the first to launch a task "wins"). We've always reserved the right to rescind resource offers using the 'rescind' primitive in the API, and a framework scheduler should be prepared to launch a task and have those tasks go lost because another framework already started to use those resources.

      Feature
      We plan to take a step towards optimistic offers, by introducing primitives that allow resources to be offered to multiple frameworks at once. At first, we will use these primitives to optimistically allocate resources that are reserved for a particular framework/role but have not been allocated by that framework/role.

      The work with optimistic offers will closely resemble the existing oversubscription feature. Optimistically offered resources are likely to be considered "revocable resources" (the concept that using resources not reserved for you means you might get those resources revoked). In effect, we can may create something like a "spot" market for unused resources, driving up utilization by letting frameworks that are willing to use revocable resources run tasks.

      Future Work
      This ticket tracks the introduction of some aspects of optimistic offers.

      Taken to the limit, one could imagine always making optimistic resource offers. This bears a striking resemblance with the Google Omega model (an isomorphism even). However, being able to configure what resources should be allocated optimistically and what resources should be allocated pessimistically gives even more control to a datacenter/cluster operator that might want to, for example, never let multiple frameworks (roles) compete for some set of resources.

      1. optimisitic-offers.pdf
        59 kB
        Benjamin Hindman

        Issue Links

          Issues in Epic

          There are no issues in this epic.

            Activity

            Hide
            benjaminhindman Benjamin Hindman added a comment -

            I've attached some slides that I've used in the past to describe optimistic offers and their relationship with the Google Omega model.

            Show
            benjaminhindman Benjamin Hindman added a comment - I've attached some slides that I've used in the past to describe optimistic offers and their relationship with the Google Omega model.
            Hide
            tstclair Timothy St. Clair added a comment - - edited

            This is awesome!
            I'm curious how this the framework protocol is going to change around this.

            Show
            tstclair Timothy St. Clair added a comment - - edited This is awesome! I'm curious how this the framework protocol is going to change around this.
            Hide
            nnielsen Niklas Quarfot Nielsen added a comment -

            Huge +1 - would love to help out.

            Show
            nnielsen Niklas Quarfot Nielsen added a comment - Huge +1 - would love to help out.
            Hide
            itamar_yowza Itamar Ostricher added a comment -

            I'd love to see this feature implemented!

            I've written a framework that actually takes advantage of such optimistic behavior, by "holding on" to resources, anticipating it will have more work to schedule on them.
            As it is now, I need to decline the stored resources after some timeout, to avoid starving other frameworks, and there's a tradeoff between the ability to launch new tasks as they come in, and the starvation due to stored resources.

            With optimistic offers, the timeout is not needed anymore, and the tradeoff is resolved optimally! I already have code in place to remove stored resources if they are rescinded

            Show
            itamar_yowza Itamar Ostricher added a comment - I'd love to see this feature implemented! I've written a framework that actually takes advantage of such optimistic behavior, by "holding on" to resources, anticipating it will have more work to schedule on them. As it is now, I need to decline the stored resources after some timeout, to avoid starving other frameworks, and there's a tradeoff between the ability to launch new tasks as they come in, and the starvation due to stored resources. With optimistic offers, the timeout is not needed anymore, and the tradeoff is resolved optimally! I already have code in place to remove stored resources if they are rescinded
            Hide
            adam-mesos Adam B added a comment -

            Itamar Ostricher, please note that you can let Mesos automatically rescind offers after a timeout by specifying the `--offer_timeout` flag on the master.

            Show
            adam-mesos Adam B added a comment - Itamar Ostricher , please note that you can let Mesos automatically rescind offers after a timeout by specifying the `--offer_timeout` flag on the master.
            Hide
            itamar_yowza Itamar Ostricher added a comment -

            Cool. Good to know.

            Show
            itamar_yowza Itamar Ostricher added a comment - Cool. Good to know.
            Hide
            liqlin Liqiang Lin added a comment -

            Is there any design doc I can get?

            If it's optimistic offer, who will resolve the resource conflict in accepting offer? Allocator or Mesos master? I think sending run task message should be after resolving resource conflict. If it's allocator's responsibility to resolve resource conflict, will a new interface in allocator be introduced? If yes, how this new interface interact with existing recoverResources() function?

            Please append more design about how to resolve resource conflict in accepting offer.
            Thanks

            Show
            liqlin Liqiang Lin added a comment - Is there any design doc I can get? If it's optimistic offer, who will resolve the resource conflict in accepting offer? Allocator or Mesos master? I think sending run task message should be after resolving resource conflict. If it's allocator's responsibility to resolve resource conflict, will a new interface in allocator be introduced? If yes, how this new interface interact with existing recoverResources() function? Please append more design about how to resolve resource conflict in accepting offer. Thanks
            Hide
            reachbach Bharath Ravi Kumar added a comment -

            Is there a particular milestone that this feature is being targeted for (tentatively)? Would be useful to know that. Thanks.

            Show
            reachbach Bharath Ravi Kumar added a comment - Is there a particular milestone that this feature is being targeted for (tentatively)? Would be useful to know that. Thanks.
            Hide
            kaysoky Joseph Wu added a comment -

            We plan to release the MVP before the end of this year, so tentatively sometime between v0.26.0 and v0.28.0.

            Show
            kaysoky Joseph Wu added a comment - We plan to release the MVP before the end of this year, so tentatively sometime between v0.26.0 and v0.28.0.
            Hide
            klaus1982 Klaus Ma added a comment - - edited

            I update this JIRA to align with its description. The feature we were working on is Oversubscription for Reservation which is moved to MESOS-4967.

            Thanks
            Klaus

            Show
            klaus1982 Klaus Ma added a comment - - edited I update this JIRA to align with its description. The feature we were working on is Oversubscription for Reservation which is moved to MESOS-4967 . Thanks Klaus

              People

              • Assignee:
                hartem Artem Harutyunyan
                Reporter:
                benjaminhindman Benjamin Hindman
              • Votes:
                8 Vote for this issue
                Watchers:
                57 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Development