[YARN-9899] Migration tool that help to generate CS config based on FS config [Phase 2]

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0
• Labels: fs2cs
• Hadoop Flags: Reviewed
• Assignee: Peter Bacsko
• Reporter: Szilard Nemeth
• Created: 14/Oct/19 10:38
• Updated: 30/Jan/20 13:44
• Resolved: 26/Nov/19 20:25

Description

YARN-9699 laid down the groundworks of a converter from FS to CS config.

During the development of the converter, we came up with the following things to fix. 

1. If we don't specify a mandatory option, we have this stacktrace for example:

 

org.apache.commons.cli.MissingOptionException: Missing required option: o
 at org.apache.commons.cli.Parser.checkRequiredOptions(Parser.java:299)
 at org.apache.commons.cli.Parser.parse(Parser.java:231)
 at org.apache.commons.cli.Parser.parse(Parser.java:85)
 at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:100)
 at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1572)

 

We should provide a more concise and meaningful error message (without stacktrace on the CLI, but we should log the exception with stacktrace to the RM log).

An explanation of the missing option is also required.

2. We may think about how to handle exceptions from commons CLI: MissingArgumentException vs. MissingOptionException

3. We need to provide a -h / --help option for the CLI that prints all the possible options / arguments.

4. Last but not least: We should move the CLI command to a more reasonable place:

As YARN-9699 implemented it, the command can be invoked like: 

/opt/hadoop/bin/yarn resourcemanager -convert-fs-configuration -y /opt/hadoop/etc/hadoop/yarn-site.xml -f /opt/hadoop/etc/hadoop/fair-scheduler.xml -r ~systest/sample-rules-config.properties -o /tmp/fs-cs-output

This is problematic, as if YARN RM is already running, we need to stop it in order to start the RM again with the conversion switch.

5. Add unit test coverage for QueuePlacementConverter

6. Close some feature gaps.

 

Issue Links

• is a clone of: YARN-9699 Migration tool that help to generate CS config based on FS config [Phase 1]