Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9698 [Umbrella] Tools to help migration from Fair Scheduler to Capacity Scheduler
  3. YARN-10168

FS-CS Converter: tool doesn't handle min/max resource conversion correctly

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      Trying to understand logics of convert min and max resource from FS to CS, and found some issues:

      1)

      In FSQueueConverter#emitMaximumCapacity

      Existing logic in FS is to either specify a maximum percentage for queues against cluster resources. Or, specify an absolute valued maximum resource.

      In the existing FS2CS converter, when a percentage-based maximum resource is specified, the converter takes a global resource from fs2cs CLI, and applies percentages to that. It is not correct since the percentage-based value will get lost, and in the future when cluster resources go up and down, the maximum resource cannot be changed.

      2)

      The logic to deal with min/weight resource is also questionable:

      The existing fs2cs tool, it takes precedence of percentage over absoluteResource, and could set both to a queue config. See FSQueueConverter.Capacity#toString

      However, in CS, comparing to FS, the weights/min resource is quite different:

      CS use the same queue.capacity to specify both percentage-based or absolute-resource-based configs (Similar to how FS deal with maximum Resource).
      The capacity defines guaranteed resource, which also impact fairshare of the queue. (The more guaranteed resource a queue has, the larger "pie" the queue can get if there's any additional available resource).
      In FS, minResource defined the guaranteed resource, and weight defined how much the pie can grow to.

      So to me, in FS, we should pick and choose either weight or minResource to generate CS.

      3)

      In FS, mix-use of absolute-resource configs (like min/maxResource), and percentage-based (like weight) is allowed. But in CS, it is not allowed. The reason is discussed on YARN-5881, and find [a]Should we support specifying a mix of percentage ...

      The existing fs2cs doesn't handle the issue, which could set mixed absolute resource and percentage-based resources.

        Attachments

        1. YARN-10168-001.patch
          19 kB
          Peter Bacsko
        2. YARN-10168-002.patch
          23 kB
          Peter Bacsko
        3. YARN-10168-003.patch
          26 kB
          Peter Bacsko
        4. YARN-10168-004.patch
          26 kB
          Peter Bacsko
        5. YARN-10168-005.patch
          26 kB
          Peter Bacsko

          Activity

            People

            • Assignee:
              pbacsko Peter Bacsko
              Reporter:
              leftnoteasy Wangda Tan
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: