Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9698 [Umbrella] Tools to help migration from Fair Scheduler to Capacity Scheduler
  3. YARN-10197

FS-CS converter: fix emitted ordering policy string and max-am-resource percent value

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      There are three problems that have to be addressed in the converter:

      1) For yarn.scheduler.capacity.<path>.ordering-policy, the emitted value should be "fifo", not "FIFO". The reason we generate "FIFO" is because FifoPolicy.NAME consists of uppercase letters.

      2) maximum-am-resource-percent calculation is faulty too. For example, in FS, you can globally disable it with <queueMaxAMShareDefault>-1.0</queueMaxAMShareDefault>. However this case is not handled properly in FSConfigToCSConfigConverter.emitDefaultMaxAMShare(). -1.0 means that max AM check is disabled, therefore we have to generate the value "1.0" to allow as many AMs as possible. In FSQueueConverter.emitMaxAMShare(), we should also check if the current value differs from the global setting and only then output "1.0" for a given queue.

      3) The multi-leaf queue check is no longer necessary and it doesn't work anyway. The CS instance catches this kind of error during the verification step.

        Attachments

        1. YARN-10197-003.patch
          25 kB
          Peter Bacsko
        2. YARN-10197-002.patch
          8 kB
          Peter Bacsko
        3. YARN-10197-001.patch
          8 kB
          Peter Bacsko

          Activity

            People

            • Assignee:
              pbacsko Peter Bacsko
              Reporter:
              pbacsko Peter Bacsko

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment