Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9698 [Umbrella] Tools to help migration from Fair Scheduler to Capacity Scheduler
  3. YARN-9899

Migration tool that help to generate CS config based on FS config [Phase 2]

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.0
    • None
    • Reviewed

    Description

      YARN-9699 laid down the groundworks of a converter from FS to CS config.

      During the development of the converter, we came up with the following things to fix. 

      1. If we don't specify a mandatory option, we have this stacktrace for example:

       

      org.apache.commons.cli.MissingOptionException: Missing required option: o
       at org.apache.commons.cli.Parser.checkRequiredOptions(Parser.java:299)
       at org.apache.commons.cli.Parser.parse(Parser.java:231)
       at org.apache.commons.cli.Parser.parse(Parser.java:85)
       at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:100)
       at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1572)

       

      We should provide a more concise and meaningful error message (without stacktrace on the CLI, but we should log the exception with stacktrace to the RM log).

      An explanation of the missing option is also required.

      2. We may think about how to handle exceptions from commons CLI: MissingArgumentException vs. MissingOptionException

      3. We need to provide a -h / --help option for the CLI that prints all the possible options / arguments.

      4. Last but not least: We should move the CLI command to a more reasonable place:

      As YARN-9699 implemented it, the command can be invoked like: 

      /opt/hadoop/bin/yarn resourcemanager -convert-fs-configuration -y /opt/hadoop/etc/hadoop/yarn-site.xml -f /opt/hadoop/etc/hadoop/fair-scheduler.xml -r ~systest/sample-rules-config.properties -o /tmp/fs-cs-output
      

      This is problematic, as if YARN RM is already running, we need to stop it in order to start the RM again with the conversion switch.

      5. Add unit test coverage for QueuePlacementConverter

      6. Close some feature gaps.

       

      Attachments

        1. YARN-9899-001.patch
          39 kB
          Peter Bacsko
        2. YARN-9899-002.patch
          38 kB
          Peter Bacsko
        3. YARN-9899-003.patch
          37 kB
          Peter Bacsko
        4. YARN-9899-004.patch
          44 kB
          Peter Bacsko
        5. YARN-9899-005.patch
          44 kB
          Peter Bacsko
        6. YARN-9899-006.patch
          44 kB
          Peter Bacsko
        7. YARN-9899-007.patch
          44 kB
          Peter Bacsko

        Issue Links

          Activity

            People

              pbacsko Peter Bacsko
              snemeth Szilard Nemeth
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: