Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10496 [Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler
  3. YARN-10573

Enhance placement rule conversion in fs2cs in weight mode and enable it by default

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.4.0
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      If we're using weight mode, we have much more freedom when it comes to placement rules.

      We need to do three things in this ticket:
      #1
      In YARN-10525, weight conversion became the default in fs2cs. This also means that we can support nested rules properly and also queues can be created under root

      Therefore, a lot of warnings and validations inside QueuePlacementConverter are not necessary and only relevant if the user chose percentage-based conversion in the command line.

      #2
      Remove unnecessary stuff in the RuleHandler which are already supported (plus the test code which affected by this):

      • SPECIFIED_NOT_FIRST
      • USER_MAX_APPS_DEFAULT
      • USER_MAX_RUNNING_APPS

      #3
      Currently, users have to use "-m" or "--convert-placement-rules" switch to convert the placement rules from FS.
      Initially, we converted to the old mapping rule format, which has serious limitations, so we disabled the automatic conversion.
      With the new JSON-based format and placement engine, this conversion should happen automatically.

        Attachments

        1. YARN-10573-003.patch
          30 kB
          Peter Bacsko
        2. YARN-10573-002.patch
          24 kB
          Peter Bacsko
        3. YARN-10573-001.patch
          24 kB
          Peter Bacsko

          Activity

            People

            • Assignee:
              pbacsko Peter Bacsko
              Reporter:
              pbacsko Peter Bacsko
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: