Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-736

Add a multi-resource fair sharing metric

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.0.4-alpha
    • 2.1.0-beta
    • scheduler
    • None
    • Reviewed

    Description

      Currently, at a regular interval, the fair scheduler computes a fair memory share for each queue and application inside it. This fair share is not used for scheduling decisions, but is displayed in the web UI, exposed as a metric, and used for preemption decisions.

      With DRF and multi-resource scheduling, assigning a memory share as the fair share metric to every queue no longer makes sense. It's not obvious what the replacement should be, but probably something like fractional fairness within a queue, or distance from an ideal cluster state.

      Attachments

        1. YARN-736.patch
          41 kB
          Sandy Ryza
        2. YARN-736-1.patch
          47 kB
          Sandy Ryza
        3. YARN-736-2.patch
          41 kB
          Sandy Ryza
        4. YARN-736-3.patch
          43 kB
          Sandy Ryza
        5. YARN-736-4.patch
          43 kB
          Sandy Ryza

        Issue Links

          depends upon

          New Feature - A new feature of the product, which has yet to be developed. YARN-326 Add multi-resource scheduling to the fair scheduler

          • Major - Major loss of function.
          • Closed

          Activity

            People

              sandyr Sandy Ryza
              sandyr Sandy Ryza
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: