Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.0
    • None
    • Reviewed

    Description

      Add a "d" / "-dry-run" switch to the tool. The purpose of this would be to inform the user whether a conversion is possible and if it is, are there any warnings.

      Attachments

        1. YARN-10067-001.patch
          36 kB
          Peter Bacsko
        2. YARN-10067-002.patch
          36 kB
          Peter Bacsko
        3. YARN-10067-003.patch
          47 kB
          Peter Bacsko
        4. YARN-10067-004.patch
          47 kB
          Peter Bacsko
        5. YARN-10067-005.patch
          49 kB
          Szilard Nemeth
        6. YARN-10067-006.patch
          50 kB
          Szilard Nemeth
        7. YARN-10067-007.patch
          50 kB
          Peter Bacsko
        8. YARN-10067-007.patch
          50 kB
          Peter Bacsko

        Activity

          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 42s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 20m 45s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 40s trunk passed
          +1 mvnsite 0m 46s trunk passed
          +1 shadedclient 14m 13s branch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 13s trunk passed
          +1 javadoc 0m 30s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 41s the patch passed
          +1 javac 0m 41s the patch passed
          -0 checkstyle 0m 30s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 33 new + 3 unchanged - 0 fixed = 36 total (was 3)
          +1 mvnsite 0m 47s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 14m 31s patch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 24s the patch passed
          +1 javadoc 0m 27s the patch passed
                Other Tests
          +1 unit 85m 36s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          144m 31s



          Subsystem Report/Notes
          Docker Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3
          JIRA Issue YARN-10067
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12989860/YARN-10067-001.patch
          Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 3992c1f32f57 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / b19d87c
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_232
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/25326/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/25326/testReport/
          Max. process+thread count 822 (vs. ulimit of 5500)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/25326/console
          Powered by Apache Yetus 0.8.0 http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 42s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.       trunk Compile Tests +1 mvninstall 20m 45s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 40s trunk passed +1 mvnsite 0m 46s trunk passed +1 shadedclient 14m 13s branch has no errors when building and testing our client artifacts. +1 findbugs 1m 13s trunk passed +1 javadoc 0m 30s trunk passed       Patch Compile Tests +1 mvninstall 0m 46s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed -0 checkstyle 0m 30s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 33 new + 3 unchanged - 0 fixed = 36 total (was 3) +1 mvnsite 0m 47s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 14m 31s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 24s the patch passed +1 javadoc 0m 27s the patch passed       Other Tests +1 unit 85m 36s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 144m 31s Subsystem Report/Notes Docker Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 JIRA Issue YARN-10067 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12989860/YARN-10067-001.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 3992c1f32f57 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / b19d87c maven version: Apache Maven 3.3.9 Default Java 1.8.0_232 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/25326/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/25326/testReport/ Max. process+thread count 822 (vs. ulimit of 5500) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/25326/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 18m 1s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 36s trunk passed
          +1 mvnsite 0m 47s trunk passed
          +1 shadedclient 13m 15s branch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 9s trunk passed
          +1 javadoc 0m 33s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 41s the patch passed
          +1 compile 0m 36s the patch passed
          +1 javac 0m 36s the patch passed
          -0 checkstyle 0m 27s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 3 unchanged - 0 fixed = 5 total (was 3)
          +1 mvnsite 0m 39s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 12m 16s patch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 15s the patch passed
          +1 javadoc 0m 31s the patch passed
                Other Tests
          +1 unit 82m 15s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          134m 44s



          Subsystem Report/Notes
          Docker Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3
          JIRA Issue YARN-10067
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12989875/YARN-10067-002.patch
          Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 992b908563b1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / b19d87c
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_232
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/25327/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/25327/testReport/
          Max. process+thread count 887 (vs. ulimit of 5500)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/25327/console
          Powered by Apache Yetus 0.8.0 http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.       trunk Compile Tests +1 mvninstall 18m 1s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 0m 47s trunk passed +1 shadedclient 13m 15s branch has no errors when building and testing our client artifacts. +1 findbugs 1m 9s trunk passed +1 javadoc 0m 33s trunk passed       Patch Compile Tests +1 mvninstall 0m 41s the patch passed +1 compile 0m 36s the patch passed +1 javac 0m 36s the patch passed -0 checkstyle 0m 27s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 3 unchanged - 0 fixed = 5 total (was 3) +1 mvnsite 0m 39s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 12m 16s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 15s the patch passed +1 javadoc 0m 31s the patch passed       Other Tests +1 unit 82m 15s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 134m 44s Subsystem Report/Notes Docker Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 JIRA Issue YARN-10067 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12989875/YARN-10067-002.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 992b908563b1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / b19d87c maven version: Apache Maven 3.3.9 Default Java 1.8.0_232 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/25327/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/25327/testReport/ Max. process+thread count 887 (vs. ulimit of 5500) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/25327/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 18m 19s trunk passed
          +1 compile 0m 57s trunk passed
          +1 checkstyle 0m 40s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 shadedclient 13m 26s branch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 13s trunk passed
          +1 javadoc 0m 31s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 41s the patch passed
          +1 compile 0m 38s the patch passed
          +1 javac 0m 38s the patch passed
          -0 checkstyle 0m 27s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 3 unchanged - 0 fixed = 5 total (was 3)
          +1 mvnsite 0m 43s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 12m 26s patch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 14s the patch passed
          +1 javadoc 0m 30s the patch passed
                Other Tests
          +1 unit 82m 19s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          135m 38s



          Subsystem Report/Notes
          Docker Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3
          JIRA Issue YARN-10067
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12990002/YARN-10067-003.patch
          Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 2995f87b46ff 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4a76ab7
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_232
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/25333/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/25333/testReport/
          Max. process+thread count 851 (vs. ulimit of 5500)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/25333/console
          Powered by Apache Yetus 0.8.0 http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.       trunk Compile Tests +1 mvninstall 18m 19s trunk passed +1 compile 0m 57s trunk passed +1 checkstyle 0m 40s trunk passed +1 mvnsite 0m 50s trunk passed +1 shadedclient 13m 26s branch has no errors when building and testing our client artifacts. +1 findbugs 1m 13s trunk passed +1 javadoc 0m 31s trunk passed       Patch Compile Tests +1 mvninstall 0m 41s the patch passed +1 compile 0m 38s the patch passed +1 javac 0m 38s the patch passed -0 checkstyle 0m 27s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 3 unchanged - 0 fixed = 5 total (was 3) +1 mvnsite 0m 43s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 12m 26s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 14s the patch passed +1 javadoc 0m 30s the patch passed       Other Tests +1 unit 82m 19s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 135m 38s Subsystem Report/Notes Docker Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 JIRA Issue YARN-10067 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12990002/YARN-10067-003.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 2995f87b46ff 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 4a76ab7 maven version: Apache Maven 3.3.9 Default Java 1.8.0_232 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/25333/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/25333/testReport/ Max. process+thread count 851 (vs. ulimit of 5500) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/25333/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.
          pbacsko Peter Bacsko added a comment -

          snemeth could you pls check the changes?

          pbacsko Peter Bacsko added a comment - snemeth could you pls check the changes?
          snemeth Szilard Nemeth added a comment -

          Hi pbacsko!
          Thanks for your patch. I have some comments:

          1. In FSConfigToCSConfigArgumentHandler#parseAndConvert: Please extract the code from

          dryRun = cliParser.hasOption(CliOption.DRY_RUN.shortSwitch);
          

          until

           converter.convert(params);
          

          to a separate method, for better readability.

          2. In FSConfigToCSConfigArgumentHandler#parseAndConvert: The exception handling logic is quite verbose as of now.
          You could extract a code block that is occurring 3 times of the exception handling:

          if (dryRun) {
          	dryRunResultHolder.addDryRunError(msg);
          	} else {
          		logAndStdErr(e, msg);
          		return -1;
          	}
          

          3. I think FSConfigToCSConfigArgumentHandler#printDryRunResults is a good method, in terms of contents. I would rather move the whole printing logic into DryRunResultHolder instead, so printing its own results can be the responsibility of that class.

          4. Nit: FSConfigToCSConfigConverter#dryRun: You can omit "= false" from the declaration, since as per Java standards, booleans are initialized to false by default.

          5. I can see in many places that the boolean dryRun + the DryRunResultsHolder are passed in tandem. For example, in FSConfigToCSConfigConverter, in FSConfigToCSConfigRuleHandler and in FSConfigToCSConfigArgumentHandler.
          Can you create a class to hold these two together?
          For example, I can image something named like "RuntimeParameters" and there you could hide the details like dry run, as well as any other future runtime options.
          Methods like FSConfigToCSConfigRuleHandler#handle and FSQueueConverter#convertQueueHierarchy could simply pass (delegate) the exceptionMessage to an instance of this RuntimeParameters class and the instance could decide on what to do with the error message:
          Either throw an UnsupportedPropertyException or to record it as a dry run error. This way, the dry-run feature is better abstracted, in my opinion.

          6. Why don't you use FSConfigToCSConfigConverterParams#isDryRun anywhere? Is this intentional?

          7. In TestFSQueueConverter, you have very similar code calls to create the FSQueueConverter objects. I would suggest to extract a method that creates a builder object with those common calls, e.g.

            FSQueueConverterBuilder.create()
                  .withRuleHandler(ruleHandler)
                  .withCapacitySchedulerConfig(csConfig)
                  .withPreemptionEnabled(false)
                  .withSizeBasedWeight(false)
                  .withAutoCreateChildQueues(true)
                  .withClusterResource(CLUSTER_RESOURCE)
                  .withQueueMaxAMShareDefault(0.16f)
                  .withQueueMaxAppsDefault(15)
                  .withDryRun(false)
          

          and then tweak the builder to meet the testcase needs. This way, you can have a default builder object with a few additional calls to it to prepare the converter object.

          snemeth Szilard Nemeth added a comment - Hi pbacsko ! Thanks for your patch. I have some comments: 1. In FSConfigToCSConfigArgumentHandler#parseAndConvert: Please extract the code from dryRun = cliParser.hasOption(CliOption.DRY_RUN.shortSwitch); until converter.convert(params); to a separate method, for better readability. 2. In FSConfigToCSConfigArgumentHandler#parseAndConvert: The exception handling logic is quite verbose as of now. You could extract a code block that is occurring 3 times of the exception handling: if (dryRun) { dryRunResultHolder.addDryRunError(msg); } else { logAndStdErr(e, msg); return -1; } 3. I think FSConfigToCSConfigArgumentHandler#printDryRunResults is a good method, in terms of contents. I would rather move the whole printing logic into DryRunResultHolder instead, so printing its own results can be the responsibility of that class. 4. Nit: FSConfigToCSConfigConverter#dryRun: You can omit "= false" from the declaration, since as per Java standards, booleans are initialized to false by default. 5. I can see in many places that the boolean dryRun + the DryRunResultsHolder are passed in tandem. For example, in FSConfigToCSConfigConverter, in FSConfigToCSConfigRuleHandler and in FSConfigToCSConfigArgumentHandler. Can you create a class to hold these two together? For example, I can image something named like "RuntimeParameters" and there you could hide the details like dry run, as well as any other future runtime options. Methods like FSConfigToCSConfigRuleHandler#handle and FSQueueConverter#convertQueueHierarchy could simply pass (delegate) the exceptionMessage to an instance of this RuntimeParameters class and the instance could decide on what to do with the error message: Either throw an UnsupportedPropertyException or to record it as a dry run error. This way, the dry-run feature is better abstracted, in my opinion. 6. Why don't you use FSConfigToCSConfigConverterParams#isDryRun anywhere? Is this intentional? 7. In TestFSQueueConverter, you have very similar code calls to create the FSQueueConverter objects. I would suggest to extract a method that creates a builder object with those common calls, e.g. FSQueueConverterBuilder.create() .withRuleHandler(ruleHandler) .withCapacitySchedulerConfig(csConfig) .withPreemptionEnabled( false ) .withSizeBasedWeight( false ) .withAutoCreateChildQueues( true ) .withClusterResource(CLUSTER_RESOURCE) .withQueueMaxAMShareDefault(0.16f) .withQueueMaxAppsDefault(15) .withDryRun( false ) and then tweak the builder to meet the testcase needs. This way, you can have a default builder object with a few additional calls to it to prepare the converter object.
          pbacsko Peter Bacsko added a comment -

          Thanks snemeth.

          I addressed all comments except #5 which to me seems like a bit of an over-engineering (but if you show me an example, I can be convinced).

          pbacsko Peter Bacsko added a comment - Thanks snemeth . I addressed all comments except #5 which to me seems like a bit of an over-engineering (but if you show me an example, I can be convinced).
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 50s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 21m 31s trunk passed
          +1 compile 0m 41s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 0m 43s trunk passed
          +1 shadedclient 14m 4s branch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 12s trunk passed
          +1 javadoc 0m 30s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 41s the patch passed
          +1 compile 0m 37s the patch passed
          +1 javac 0m 37s the patch passed
          +1 checkstyle 0m 27s the patch passed
          +1 mvnsite 0m 39s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 13m 43s patch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 17s the patch passed
          +1 javadoc 0m 27s the patch passed
                Other Tests
          +1 unit 85m 34s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 26s The patch does not generate ASF License warnings.
          143m 51s



          Subsystem Report/Notes
          Docker Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3
          JIRA Issue YARN-10067
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12990190/YARN-10067-004.patch
          Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 33eeac236b8e 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 17aa8f6
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_232
          findbugs v3.1.0-RC1
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/25348/testReport/
          Max. process+thread count 839 (vs. ulimit of 5500)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/25348/console
          Powered by Apache Yetus 0.8.0 http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 50s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.       trunk Compile Tests +1 mvninstall 21m 31s trunk passed +1 compile 0m 41s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 43s trunk passed +1 shadedclient 14m 4s branch has no errors when building and testing our client artifacts. +1 findbugs 1m 12s trunk passed +1 javadoc 0m 30s trunk passed       Patch Compile Tests +1 mvninstall 0m 41s the patch passed +1 compile 0m 37s the patch passed +1 javac 0m 37s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 0m 39s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 13m 43s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 17s the patch passed +1 javadoc 0m 27s the patch passed       Other Tests +1 unit 85m 34s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 143m 51s Subsystem Report/Notes Docker Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 JIRA Issue YARN-10067 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12990190/YARN-10067-004.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 33eeac236b8e 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 17aa8f6 maven version: Apache Maven 3.3.9 Default Java 1.8.0_232 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/25348/testReport/ Max. process+thread count 839 (vs. ulimit of 5500) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/25348/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.
          snemeth Szilard Nemeth added a comment -

          Hi pbacsko!

          Thanks for fixing the concerns I raised.
          Patch looks good now.
          Regarding what I've mentioned in my previous comment with bullet point 5, I uploaded a patch based on your latest patch, so you can check what I was meant. You can probably adjust the namings here and there but I think you will get the basic idea: No scattered dryRun checks everywhere in the code, all of them are handled in a single point, ConversionOptions. Also, if we ever want to have more conversion options like dry run, we can add it more easily.
          Please share your thoughts.

          Thanks.

          snemeth Szilard Nemeth added a comment - Hi pbacsko ! Thanks for fixing the concerns I raised. Patch looks good now. Regarding what I've mentioned in my previous comment with bullet point 5, I uploaded a patch based on your latest patch, so you can check what I was meant. You can probably adjust the namings here and there but I think you will get the basic idea: No scattered dryRun checks everywhere in the code, all of them are handled in a single point, ConversionOptions. Also, if we ever want to have more conversion options like dry run, we can add it more easily. Please share your thoughts. Thanks.
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 42s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 19m 29s trunk passed
          +1 compile 0m 40s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 0m 44s trunk passed
          +1 shadedclient 14m 14s branch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 34s trunk passed
          +1 javadoc 0m 39s trunk passed
                Patch Compile Tests
          -1 mvninstall 0m 48s hadoop-yarn-server-resourcemanager in the patch failed.
          -1 compile 0m 43s hadoop-yarn-server-resourcemanager in the patch failed.
          -1 javac 0m 43s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 checkstyle 0m 31s the patch passed
          -1 mvnsite 0m 43s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 shadedclient 3m 53s patch has errors when building and testing our client artifacts.
          -1 findbugs 0m 23s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 javadoc 0m 26s the patch passed
                Other Tests
          -1 unit 0m 39s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          47m 6s



          Subsystem Report/Notes
          Docker Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3
          JIRA Issue YARN-10067
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12990451/YARN-10067-005.patch
          Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 7e10af8f6159 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 93233a7
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_232
          findbugs v3.1.0-RC1
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/25361/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          compile https://builds.apache.org/job/PreCommit-YARN-Build/25361/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/25361/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/25361/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/25361/artifact/out/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/25361/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/25361/testReport/
          Max. process+thread count 307 (vs. ulimit of 5500)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/25361/console
          Powered by Apache Yetus 0.8.0 http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 42s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.       trunk Compile Tests +1 mvninstall 19m 29s trunk passed +1 compile 0m 40s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 44s trunk passed +1 shadedclient 14m 14s branch has no errors when building and testing our client artifacts. +1 findbugs 1m 34s trunk passed +1 javadoc 0m 39s trunk passed       Patch Compile Tests -1 mvninstall 0m 48s hadoop-yarn-server-resourcemanager in the patch failed. -1 compile 0m 43s hadoop-yarn-server-resourcemanager in the patch failed. -1 javac 0m 43s hadoop-yarn-server-resourcemanager in the patch failed. +1 checkstyle 0m 31s the patch passed -1 mvnsite 0m 43s hadoop-yarn-server-resourcemanager in the patch failed. +1 whitespace 0m 0s The patch has no whitespace issues. -1 shadedclient 3m 53s patch has errors when building and testing our client artifacts. -1 findbugs 0m 23s hadoop-yarn-server-resourcemanager in the patch failed. +1 javadoc 0m 26s the patch passed       Other Tests -1 unit 0m 39s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 47m 6s Subsystem Report/Notes Docker Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 JIRA Issue YARN-10067 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12990451/YARN-10067-005.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 7e10af8f6159 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 93233a7 maven version: Apache Maven 3.3.9 Default Java 1.8.0_232 findbugs v3.1.0-RC1 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/25361/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/25361/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/25361/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/25361/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/25361/artifact/out/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/25361/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/25361/testReport/ Max. process+thread count 307 (vs. ulimit of 5500) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/25361/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.
          snemeth Szilard Nemeth added a comment -

          Sorry pbacsko, the patch I uploaded did not compile, messed up something while generating the patch. Looking into this now.

          snemeth Szilard Nemeth added a comment - Sorry pbacsko , the patch I uploaded did not compile, messed up something while generating the patch. Looking into this now.
          snemeth Szilard Nemeth added a comment -

          Uploaded patch006 to fix the compilation errors.

          snemeth Szilard Nemeth added a comment - Uploaded patch006 to fix the compilation errors.
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 42s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 19m 38s trunk passed
          +1 compile 0m 42s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 0m 44s trunk passed
          +1 shadedclient 14m 7s branch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 12s trunk passed
          +1 javadoc 0m 30s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 44s the patch passed
          +1 compile 0m 37s the patch passed
          +1 javac 0m 37s the patch passed
          +1 checkstyle 0m 28s the patch passed
          +1 mvnsite 0m 39s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 14m 23s patch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 20s the patch passed
          +1 javadoc 0m 26s the patch passed
                Other Tests
          -1 unit 86m 21s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          143m 30s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.fair.converter.TestFSQueueConverter
            hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy
            hadoop.yarn.server.resourcemanager.scheduler.fair.converter.TestFSConfigToCSConfigRuleHandler



          Subsystem Report/Notes
          Docker Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3
          JIRA Issue YARN-10067
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12990500/YARN-10067-006.patch
          Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 2ec0b68f1423 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9da294a
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_232
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-YARN-Build/25362/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/25362/testReport/
          Max. process+thread count 809 (vs. ulimit of 5500)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/25362/console
          Powered by Apache Yetus 0.8.0 http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 42s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.       trunk Compile Tests +1 mvninstall 19m 38s trunk passed +1 compile 0m 42s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 0m 44s trunk passed +1 shadedclient 14m 7s branch has no errors when building and testing our client artifacts. +1 findbugs 1m 12s trunk passed +1 javadoc 0m 30s trunk passed       Patch Compile Tests +1 mvninstall 0m 44s the patch passed +1 compile 0m 37s the patch passed +1 javac 0m 37s the patch passed +1 checkstyle 0m 28s the patch passed +1 mvnsite 0m 39s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 14m 23s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 20s the patch passed +1 javadoc 0m 26s the patch passed       Other Tests -1 unit 86m 21s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 143m 30s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.fair.converter.TestFSQueueConverter   hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy   hadoop.yarn.server.resourcemanager.scheduler.fair.converter.TestFSConfigToCSConfigRuleHandler Subsystem Report/Notes Docker Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 JIRA Issue YARN-10067 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12990500/YARN-10067-006.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 2ec0b68f1423 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 9da294a maven version: Apache Maven 3.3.9 Default Java 1.8.0_232 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/25362/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/25362/testReport/ Max. process+thread count 809 (vs. ulimit of 5500) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/25362/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.
          pbacsko Peter Bacsko added a comment -

          Thanks snemeth to me the enhancements look good.

          Uploaded patch v7 to fix unit tests.

          pbacsko Peter Bacsko added a comment - Thanks snemeth  to me the enhancements look good. Uploaded patch v7 to fix unit tests.
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 42s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 19m 13s trunk passed
          +1 compile 0m 42s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 0m 45s trunk passed
          +1 shadedclient 14m 3s branch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 12s trunk passed
          +1 javadoc 0m 30s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 42s the patch passed
          +1 compile 0m 36s the patch passed
          +1 javac 0m 36s the patch passed
          +1 checkstyle 0m 27s the patch passed
          +1 mvnsite 0m 40s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 13m 47s patch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 19s the patch passed
          +1 javadoc 0m 26s the patch passed
                Other Tests
          +1 unit 87m 18s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 26s The patch does not generate ASF License warnings.
          143m 14s



          Subsystem Report/Notes
          Docker Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3
          JIRA Issue YARN-10067
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12990530/YARN-10067-007.patch
          Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 4e62c6597cc8 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 49df838
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_232
          findbugs v3.1.0-RC1
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/25366/testReport/
          Max. process+thread count 829 (vs. ulimit of 5500)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/25366/console
          Powered by Apache Yetus 0.8.0 http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 42s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.       trunk Compile Tests +1 mvninstall 19m 13s trunk passed +1 compile 0m 42s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 45s trunk passed +1 shadedclient 14m 3s branch has no errors when building and testing our client artifacts. +1 findbugs 1m 12s trunk passed +1 javadoc 0m 30s trunk passed       Patch Compile Tests +1 mvninstall 0m 42s the patch passed +1 compile 0m 36s the patch passed +1 javac 0m 36s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 0m 40s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 13m 47s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 19s the patch passed +1 javadoc 0m 26s the patch passed       Other Tests +1 unit 87m 18s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 143m 14s Subsystem Report/Notes Docker Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 JIRA Issue YARN-10067 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12990530/YARN-10067-007.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 4e62c6597cc8 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 49df838 maven version: Apache Maven 3.3.9 Default Java 1.8.0_232 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/25366/testReport/ Max. process+thread count 829 (vs. ulimit of 5500) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/25366/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 49s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 22m 19s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 shadedclient 14m 59s branch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 26s trunk passed
          +1 javadoc 0m 31s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 50s the patch passed
          +1 compile 0m 39s the patch passed
          +1 javac 0m 39s the patch passed
          +1 checkstyle 0m 28s the patch passed
          +1 mvnsite 0m 42s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 13m 45s patch has no errors when building and testing our client artifacts.
          +1 findbugs 1m 21s the patch passed
          +1 javadoc 0m 29s the patch passed
                Other Tests
          -1 unit 87m 44s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          148m 32s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing



          Subsystem Report/Notes
          Docker Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3
          JIRA Issue YARN-10067
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12990530/YARN-10067-007.patch
          Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 63cddb87ca81 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / c4fb43c
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_232
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-YARN-Build/25368/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/25368/testReport/
          Max. process+thread count 818 (vs. ulimit of 5500)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/25368/console
          Powered by Apache Yetus 0.8.0 http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 49s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.       trunk Compile Tests +1 mvninstall 22m 19s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 0m 53s trunk passed +1 shadedclient 14m 59s branch has no errors when building and testing our client artifacts. +1 findbugs 1m 26s trunk passed +1 javadoc 0m 31s trunk passed       Patch Compile Tests +1 mvninstall 0m 50s the patch passed +1 compile 0m 39s the patch passed +1 javac 0m 39s the patch passed +1 checkstyle 0m 28s the patch passed +1 mvnsite 0m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 13m 45s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 21s the patch passed +1 javadoc 0m 29s the patch passed       Other Tests -1 unit 87m 44s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 148m 32s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing Subsystem Report/Notes Docker Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 JIRA Issue YARN-10067 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12990530/YARN-10067-007.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 63cddb87ca81 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / c4fb43c maven version: Apache Maven 3.3.9 Default Java 1.8.0_232 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/25368/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/25368/testReport/ Max. process+thread count 818 (vs. ulimit of 5500) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/25368/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.
          snemeth Szilard Nemeth added a comment -

          Thanks pbacsko!
          Latest patch looks good to me, committed to trunk.
          Resolving the jira as I don't think we need any backports.

          snemeth Szilard Nemeth added a comment - Thanks pbacsko ! Latest patch looks good to me, committed to trunk. Resolving the jira as I don't think we need any backports.
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17854 (See https://builds.apache.org/job/Hadoop-trunk-Commit/17854/)
          YARN-10067. Add dry-run feature to FS-CS converter tool. Contributed by (snemeth: rev 24e6a9e43a210cdecaa8e87926eef09c869988f9)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigArgumentHandler.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigRuleHandler.java
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverterBuilder.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverter.java
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/DryRunResultHolder.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverterMain.java
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/ConversionOptions.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigRuleHandler.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17854 (See https://builds.apache.org/job/Hadoop-trunk-Commit/17854/ ) YARN-10067 . Add dry-run feature to FS-CS converter tool. Contributed by (snemeth: rev 24e6a9e43a210cdecaa8e87926eef09c869988f9) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigArgumentHandler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigRuleHandler.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverterBuilder.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverter.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/DryRunResultHolder.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverterMain.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/ConversionOptions.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigRuleHandler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java

          People

            pbacsko Peter Bacsko
            pbacsko Peter Bacsko
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: