Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-736

Add a multi-resource fair sharing metric

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.4-alpha
    • Fix Version/s: 2.1.0-beta
    • Component/s: scheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, at a regular interval, the fair scheduler computes a fair memory share for each queue and application inside it. This fair share is not used for scheduling decisions, but is displayed in the web UI, exposed as a metric, and used for preemption decisions.

      With DRF and multi-resource scheduling, assigning a memory share as the fair share metric to every queue no longer makes sense. It's not obvious what the replacement should be, but probably something like fractional fairness within a queue, or distance from an ideal cluster state.

        Attachments

        1. YARN-736-4.patch
          43 kB
          Sandy Ryza
        2. YARN-736-3.patch
          43 kB
          Sandy Ryza
        3. YARN-736-2.patch
          41 kB
          Sandy Ryza
        4. YARN-736-1.patch
          47 kB
          Sandy Ryza
        5. YARN-736.patch
          41 kB
          Sandy Ryza

          Issue Links

            Activity

              People

              • Assignee:
                sandyr Sandy Ryza
                Reporter:
                sandyr Sandy Ryza
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: