Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6428

Queue AM limit is not honored in CS always

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Steps to reproduce
      ------------------------
      Setup cluster with 40 GB and 40 vcores with 4 Node managers with 10 GB each.
      Configure 100% to default queue as capacity and max am limit as 10 %
      Minimum scheduler memory and vcore as 512,1
      Expected
      AM limit 4096 and 4 vores

      Actual
      AM limit 4096+512 and 4+1 vcore

      1. YARN-6428.0001.patch
        8 kB
        Bibin A Chundatt
      2. YARN-6428.0002.patch
        9 kB
        Bibin A Chundatt
      3. YARN-6428.0003.patch
        4 kB
        Bibin A Chundatt
      4. YARN-6428-branch-2.8.0003.patch
        4 kB
        Bibin A Chundatt

        Activity

        Hide
        bibinchundatt Bibin A Chundatt added a comment - - edited

        ResourceCalculator#multiplyAndNormalizeUp using double to multiply and ceil for am limit factor.

        roundUp(
                    (int)Math.ceil(r.getMemorySize() * by)
        

        0.1 will be 0.100000000012312444 so multiple and ceil and round by will give give 4097 instead of 4096.

        Show
        bibinchundatt Bibin A Chundatt added a comment - - edited ResourceCalculator#multiplyAndNormalizeUp using double to multiply and ceil for am limit factor. roundUp( ( int ) Math .ceil(r.getMemorySize() * by) 0.1 will be 0.100000000012312444 so multiple and ceil and round by will give give 4097 instead of 4096.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Attaching patch for the same

        Show
        bibinchundatt Bibin A Chundatt added a comment - Attaching patch for the same
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 26s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 13m 8s trunk passed
        +1 compile 11m 28s trunk passed
        +1 checkstyle 0m 53s trunk passed
        +1 mvnsite 1m 16s trunk passed
        +1 mvneclipse 0m 40s trunk passed
        +1 findbugs 2m 10s trunk passed
        +1 javadoc 1m 1s trunk passed
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 0m 57s the patch passed
        +1 compile 8m 26s the patch passed
        +1 javac 8m 26s the patch passed
        +1 checkstyle 0m 53s the patch passed
        +1 mvnsite 1m 15s the patch passed
        +1 mvneclipse 0m 39s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 28s the patch passed
        +1 javadoc 1m 1s the patch passed
        +1 unit 2m 28s hadoop-yarn-common in the patch passed.
        -1 unit 38m 50s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 32s The patch does not generate ASF License warnings.
        97m 23s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.TestResourceTrackerService



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6428
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861662/YARN-6428.0001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 929170560ff1 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 845529b
        Default Java 1.8.0_121
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15484/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15484/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15484/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 13m 8s trunk passed +1 compile 11m 28s trunk passed +1 checkstyle 0m 53s trunk passed +1 mvnsite 1m 16s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 2m 10s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 57s the patch passed +1 compile 8m 26s the patch passed +1 javac 8m 26s the patch passed +1 checkstyle 0m 53s the patch passed +1 mvnsite 1m 15s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 28s the patch passed +1 javadoc 1m 1s the patch passed +1 unit 2m 28s hadoop-yarn-common in the patch passed. -1 unit 38m 50s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 97m 23s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestResourceTrackerService Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6428 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861662/YARN-6428.0001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 929170560ff1 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 845529b Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/15484/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15484/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/15484/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Sunil G
        Could you please review patch attached

        Show
        bibinchundatt Bibin A Chundatt added a comment - Sunil G Could you please review patch attached
        Hide
        varun_saxena Varun Saxena added a comment -

        Thanks Bibin A Chundatt for the patch.
        The said issue due to conversion does exist.

        So it does make sense to have a float version of the method in ResourceCalculator to avoid such issues due to conversion.
        We can alternatively strip off the extraneous decimal digits too but that would not be as straightforward.
        And in this case, as ResourceCalculator is annotated as Private, we can add an abstract method to it as well.

        However, it seems the values passed to ResourceCalculator#multiplyAndNormalizeUp are all in float. But as double has larger range and more precision, it might be better, in general as a data type, when we do calculations in scheduler.
        So can't we alternatively read these configurations as double too? Thoughts?

        Show
        varun_saxena Varun Saxena added a comment - Thanks Bibin A Chundatt for the patch. The said issue due to conversion does exist. So it does make sense to have a float version of the method in ResourceCalculator to avoid such issues due to conversion. We can alternatively strip off the extraneous decimal digits too but that would not be as straightforward. And in this case, as ResourceCalculator is annotated as Private, we can add an abstract method to it as well. However, it seems the values passed to ResourceCalculator#multiplyAndNormalizeUp are all in float. But as double has larger range and more precision, it might be better, in general as a data type, when we do calculations in scheduler. So can't we alternatively read these configurations as double too? Thoughts?
        Hide
        bibinchundatt Bibin A Chundatt added a comment - - edited

        Varun Saxena
        Thank you for looking into the issue. The second approach will be a bigger change than
        current. for example: QueueCapacities currently uses float .
        In current implementation we have option to used both interface.

        Show
        bibinchundatt Bibin A Chundatt added a comment - - edited Varun Saxena Thank you for looking into the issue. The second approach will be a bigger change than current. for example: QueueCapacities currently uses float . In current implementation we have option to used both interface.
        Hide
        bibinchundatt Bibin A Chundatt added a comment - - edited

        DefaultResourceCalculator any idea why 0.5 is added don't find the same in DominantResourceCalculator

          @Override
          public Resource multiplyAndNormalizeUp(Resource r, double by,
              Resource stepFactor) {
            return Resources.createResource(
                roundUp((long) (r.getMemorySize() * by + 0.5),
                    stepFactor.getMemorySize()));
          }
        

        We should use Math.ceil here too.

        Show
        bibinchundatt Bibin A Chundatt added a comment - - edited DefaultResourceCalculator any idea why 0.5 is added don't find the same in DominantResourceCalculator @Override public Resource multiplyAndNormalizeUp(Resource r, double by, Resource stepFactor) { return Resources.createResource( roundUp(( long ) (r.getMemorySize() * by + 0.5), stepFactor.getMemorySize())); } We should use Math.ceil here too.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        cc/ Wangda Tan thoughts??

        Show
        bibinchundatt Bibin A Chundatt added a comment - cc/ Wangda Tan thoughts??
        Hide
        varun_saxena Varun Saxena added a comment -

        You are right about changing it to double will be a far reaching change i.e. it will impact quite a few areas. All our queue capacities for instance are stored as float.
        To be honest there are inconsistencies in usage of double and float all across.
        So adding a float version of method will be the easiest fix.
        However, let Wangda comment on this.

        Show
        varun_saxena Varun Saxena added a comment - You are right about changing it to double will be a far reaching change i.e. it will impact quite a few areas. All our queue capacities for instance are stored as float. To be honest there are inconsistencies in usage of double and float all across. So adding a float version of method will be the easiest fix. However, let Wangda comment on this.
        Hide
        bibinchundatt Bibin A Chundatt added a comment - - edited

        Sunil G / Wangda Tan
        Could you please share your thoughts.?

        Show
        bibinchundatt Bibin A Chundatt added a comment - - edited Sunil G / Wangda Tan Could you please share your thoughts.?
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Varun SaxenaNaganarasimha Garla/Sunil G
        Any review comments i need to update.

        Show
        bibinchundatt Bibin A Chundatt added a comment - Varun Saxena Naganarasimha Garla / Sunil G Any review comments i need to update.
        Hide
        bibinchundatt Bibin A Chundatt added a comment - - edited

        During offline discussion with Naganarasimha Garla for

          @Override
          public Resource multiplyAndNormalizeUp(Resource r, double by,
              Resource stepFactor) {
            return Resources.createResource(
                roundUp(
                    (int)Math.ceil(r.getMemorySize() * by), stepFactor.getMemorySize()),
                roundUp(
                    (int)Math.ceil(r.getVirtualCores() * by),
                    stepFactor.getVirtualCores())
                );
          }
        

        In the above base code instead off (int)Math.ceil(r.getMemorySize() * by) it should be (long)Math.ceil(r.getMemorySize() * by) . For DefaultResourceCalculator long is used but DominantResourceCalculator int is used.Seems to be a miss during type datatype change

        Show
        bibinchundatt Bibin A Chundatt added a comment - - edited During offline discussion with Naganarasimha Garla for @Override public Resource multiplyAndNormalizeUp(Resource r, double by, Resource stepFactor) { return Resources.createResource( roundUp( ( int ) Math .ceil(r.getMemorySize() * by), stepFactor.getMemorySize()), roundUp( ( int ) Math .ceil(r.getVirtualCores() * by), stepFactor.getVirtualCores()) ); } In the above base code instead off (int)Math.ceil(r.getMemorySize() * by) it should be (long)Math.ceil(r.getMemorySize() * by) . For DefaultResourceCalculator long is used but DominantResourceCalculator int is used.Seems to be a miss during type datatype change
        Hide
        Naganarasimha Naganarasimha G R added a comment -

        Hi Bibin A Chundatt,
        Also we can have the same code rounding up and down in both calculators (in DefaultResourceCalculator we are adopting r.getMemorySize * by +.5 but Math.ceil in Dominant).
        Similar int and long needs to be handled for existing multiplyAndNormalizeDown.
        Other than that overall patch looks fine to me !

        Show
        Naganarasimha Naganarasimha G R added a comment - Hi Bibin A Chundatt , Also we can have the same code rounding up and down in both calculators (in DefaultResourceCalculator we are adopting r.getMemorySize * by +.5 but Math.ceil in Dominant). Similar int and long needs to be handled for existing multiplyAndNormalizeDown . Other than that overall patch looks fine to me !
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Similar int and long needs to be handled for existing multiplyAndNormalizeDown.

        Handled.

        Minor change in patch i have done. Instead of adding additional interface for float existing one i have changed to float.I don't see any impact of this change.

        Show
        bibinchundatt Bibin A Chundatt added a comment - Similar int and long needs to be handled for existing multiplyAndNormalizeDown. Handled. Minor change in patch i have done. Instead of adding additional interface for float existing one i have changed to float.I don't see any impact of this change.
        Hide
        sunilg Sunil G added a comment -

        Bibin A Chundatt and Naganarasimha Garla

        I think adding more api's for a datatype change is simpler here, however i feel we can try limit the apis in this class. As resource profile is coming in, it may become slightly confusing/complex. Could we continue use double itself but truncate. Some thing like Math.floor(value * 1000) / 1000;. This ll give 3 digits precision. cc/Wangda Tan

        Show
        sunilg Sunil G added a comment - Bibin A Chundatt and Naganarasimha Garla I think adding more api's for a datatype change is simpler here, however i feel we can try limit the apis in this class. As resource profile is coming in, it may become slightly confusing/complex. Could we continue use double itself but truncate. Some thing like Math.floor(value * 1000) / 1000; . This ll give 3 digits precision. cc/ Wangda Tan
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 16s Maven dependency ordering for branch
        +1 mvninstall 13m 35s trunk passed
        +1 compile 8m 50s trunk passed
        +1 checkstyle 0m 54s trunk passed
        +1 mvnsite 1m 17s trunk passed
        -1 findbugs 1m 2s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 1 extant Findbugs warnings.
        +1 javadoc 1m 2s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 2s the patch passed
        +1 compile 6m 2s the patch passed
        +1 javac 6m 2s the patch passed
        -0 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 226 unchanged - 3 fixed = 227 total (was 229)
        +1 mvnsite 1m 19s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 33s the patch passed
        +1 javadoc 1m 7s the patch passed
        +1 unit 2m 36s hadoop-yarn-common in the patch passed.
        -1 unit 40m 55s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 32s The patch does not generate ASF License warnings.
        93m 9s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
          hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6428
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871660/YARN-6428.0002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2c96bb7414bd 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 536f057
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16131/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16131/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16131/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16131/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16131/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 13m 35s trunk passed +1 compile 8m 50s trunk passed +1 checkstyle 0m 54s trunk passed +1 mvnsite 1m 17s trunk passed -1 findbugs 1m 2s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 2s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 2s the patch passed +1 compile 6m 2s the patch passed +1 javac 6m 2s the patch passed -0 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 226 unchanged - 3 fixed = 227 total (was 229) +1 mvnsite 1m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 33s the patch passed +1 javadoc 1m 7s the patch passed +1 unit 2m 36s hadoop-yarn-common in the patch passed. -1 unit 40m 55s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 93m 9s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6428 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12871660/YARN-6428.0002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2c96bb7414bd 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 536f057 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16131/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16131/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16131/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16131/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16131/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Math.floor(value * 1000) / 1000

        Could reduce the range of values which can be used by 1000.

        Also any issue your are finding in changing the existing datatype to float?

        Show
        bibinchundatt Bibin A Chundatt added a comment - Math.floor(value * 1000) / 1000 Could reduce the range of values which can be used by 1000. Also any issue your are finding in changing the existing datatype to float?
        Hide
        Naganarasimha Naganarasimha G R added a comment -

        Thanks Bibin A Chundatt
        Earlier too in our offline discussion i had preferred for opting modification of the existing methods rather than adding a new interface if it suffices for all the callers.
        Sunil G,
        I think its fine to modify the existing interface which is done in 2nd patch rather than having a workaround, whats your opinion ?
        If no issues i plan to go ahead and commit the 2nd patch.

        Show
        Naganarasimha Naganarasimha G R added a comment - Thanks Bibin A Chundatt Earlier too in our offline discussion i had preferred for opting modification of the existing methods rather than adding a new interface if it suffices for all the callers. Sunil G , I think its fine to modify the existing interface which is done in 2nd patch rather than having a workaround, whats your opinion ? If no issues i plan to go ahead and commit the 2nd patch.
        Hide
        sunilg Sunil G added a comment -

        I guess adding new apis may more complicate here. cc Wangda Tan who was also part of offline discussion.

        Show
        sunilg Sunil G added a comment - I guess adding new apis may more complicate here. cc Wangda Tan who was also part of offline discussion.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Sunil G/Wangda Tan
        Can we go ahead with the update?? Any comments??

        Show
        bibinchundatt Bibin A Chundatt added a comment - Sunil G / Wangda Tan Can we go ahead with the update?? Any comments??
        Hide
        Naganarasimha Naganarasimha G R added a comment -

        Sunil G & Tan, Wangda,
        Current API is mostly used by passing float parameter except for the TempQueuePerPartition.normalizedGuarantee, where in here too we could have used float but its just that its been set to double. So for all current usages i feel the existing patch is good considering that TempQueuePerPartition.normalizedGuarantee is corrected to float. Thoughts ?

        Show
        Naganarasimha Naganarasimha G R added a comment - Sunil G & Tan, Wangda , Current API is mostly used by passing float parameter except for the TempQueuePerPartition.normalizedGuarantee , where in here too we could have used float but its just that its been set to double. So for all current usages i feel the existing patch is good considering that TempQueuePerPartition.normalizedGuarantee is corrected to float. Thoughts ?
        Hide
        sunilg Sunil G added a comment -

        I had an offline discussion with Wangda Tan. I think there might be a compatibility break though its from Resources class which is marked as Unstable. Since multiplyAndNormalizeUp public and used within, it is a breakage when double is changed to float in multiplyAndNormalizeUp. Adding a new float type API will add more complexities to Resources class going forward. So using Math.floor(value * 10^N) / 10^N where N could be 6 will help to resolve the problem for now. Thoughts?

        Show
        sunilg Sunil G added a comment - I had an offline discussion with Wangda Tan . I think there might be a compatibility break though its from Resources class which is marked as Unstable. Since multiplyAndNormalizeUp public and used within, it is a breakage when double is changed to float in multiplyAndNormalizeUp . Adding a new float type API will add more complexities to Resources class going forward. So using Math.floor(value * 10^N) / 10^N where N could be 6 will help to resolve the problem for now. Thoughts?
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Sunil G

        I think there might be a compatibility break though its from Resources class

        LimitedPrivate only applicable for specified projects.. So i don't think its an issue.

        @InterfaceAudience.LimitedPrivate({"YARN", "MapReduce"})
        @Unstable
        

        So using Math.floor(value * 10^N) / 10^N where N could be 6 will help to resolve the problem for now.

        The issue with solution mentioned is we are limiting the max value of resource that can be configured for attributes to MAX VALUE / 10^N i wouldn't prefer for these reasons.

        IF you feel all the above can be ignored then i can go ahead and change.

        Show
        bibinchundatt Bibin A Chundatt added a comment - Sunil G I think there might be a compatibility break though its from Resources class LimitedPrivate only applicable for specified projects.. So i don't think its an issue. @InterfaceAudience.LimitedPrivate({ "YARN" , "MapReduce" }) @Unstable So using Math.floor(value * 10^N) / 10^N where N could be 6 will help to resolve the problem for now. The issue with solution mentioned is we are limiting the max value of resource that can be configured for attributes to MAX VALUE / 10^N i wouldn't prefer for these reasons. IF you feel all the above can be ignored then i can go ahead and change.
        Hide
        sunilg Sunil G added a comment -

        LimitedPrivate only applicable for specified projects.. So i don't think its an issue.

        I guess it might be still a problem. In scheduler and rm, we have various plugin interfaces. Also a recent change of one api in this class broke tez compilation. Hence I do not think its a good idea to change from double to float and break an existing api.

        {{Math.floor(value * 10^N) / 10^N }} will be doing conversion to float. A precision of 6 should be good enough rt. Do you think there are some case where precision 6 will be a problem ?

        Show
        sunilg Sunil G added a comment - LimitedPrivate only applicable for specified projects.. So i don't think its an issue. I guess it might be still a problem. In scheduler and rm, we have various plugin interfaces. Also a recent change of one api in this class broke tez compilation. Hence I do not think its a good idea to change from double to float and break an existing api. {{Math.floor(value * 10^N) / 10^N }} will be doing conversion to float. A precision of 6 should be good enough rt. Do you think there are some case where precision 6 will be a problem ?
        Hide
        Naganarasimha Naganarasimha G R added a comment -

        As discussed offline with Sunil G, What Bibin A Chundatt wanted to say is : In ResourceCalculator. multiplyAndNormalizeUp implementations, as we multiply with the resource.mem (long value & CPU usually will not have a problem ) first with 10^6 so the value of memory cannot exceed by Long.MAX VALUE / 10^N as it would hit the limit and become negative value.

        Show
        Naganarasimha Naganarasimha G R added a comment - As discussed offline with Sunil G , What Bibin A Chundatt wanted to say is : In ResourceCalculator. multiplyAndNormalizeUp implementations, as we multiply with the resource.mem (long value & CPU usually will not have a problem ) first with 10^6 so the value of memory cannot exceed by Long.MAX VALUE / 10^N as it would hit the limit and become negative value.
        Hide
        sunilg Sunil G added a comment -

        Thanks Naganarasimha Garla and Bibin A Chundatt for the clarifications.

        If we use a very big value in long, then it could cause an overflow once we multiple with 10^6. We moved from int to long to support -ve round of cases, so I am also not much lenient in putting a loop hole there. Instead could we use BigDecimal to set precision point. I could see that we are using BigInteger in few places, so it may be fine. I will wait for Wangda Tan here.
        My alternate proposal is something like this

        		BigDecimal bd = new BigDecimal(by).setScale(2, RoundingMode.HALF_EVEN);
        		double by = bd.doubleValue();
        

        I will wait for more comments too at this point.

        Show
        sunilg Sunil G added a comment - Thanks Naganarasimha Garla and Bibin A Chundatt for the clarifications. If we use a very big value in long, then it could cause an overflow once we multiple with 10^6. We moved from int to long to support -ve round of cases, so I am also not much lenient in putting a loop hole there. Instead could we use BigDecimal to set precision point. I could see that we are using BigInteger in few places, so it may be fine. I will wait for Wangda Tan here. My alternate proposal is something like this BigDecimal bd = new BigDecimal(by).setScale(2, RoundingMode.HALF_EVEN); double by = bd.doubleValue(); I will wait for more comments too at this point.
        Hide
        Naganarasimha Naganarasimha G R added a comment -

        Sunil Govind, so IIUC you are trying to propose to round of the multiplication factor to 2 decimal points ? agree thats its one option but is it not overkill than just adding one more overloaded method ? But anyway i am ok with both options

        Show
        Naganarasimha Naganarasimha G R added a comment - Sunil Govind , so IIUC you are trying to propose to round of the multiplication factor to 2 decimal points ? agree thats its one option but is it not overkill than just adding one more overloaded method ? But anyway i am ok with both options
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Sunil G/Naganarasimha Garla
        Another solution if the interface cant be changed is to just convert to float as following.Since value anyways converted to long the conversion to float also will be fine.

          @Override
          public Resource multiplyAndNormalizeUp(Resource r, double by,
              Resource stepFactor) {
            return Resources.createResource(
                roundUp(
                    (long) Math.ceil((float) (r.getMemorySize() * by)),
                    stepFactor.getMemorySize()),
                roundUp(
                    (int) Math.ceil((float) (r.getVirtualCores() * by)),
                    stepFactor.getVirtualCores())
                );
          }
        
        Show
        bibinchundatt Bibin A Chundatt added a comment - Sunil G / Naganarasimha Garla Another solution if the interface cant be changed is to just convert to float as following.Since value anyways converted to long the conversion to float also will be fine. @Override public Resource multiplyAndNormalizeUp(Resource r, double by, Resource stepFactor) { return Resources.createResource( roundUp( ( long ) Math .ceil(( float ) (r.getMemorySize() * by)), stepFactor.getMemorySize()), roundUp( ( int ) Math .ceil(( float ) (r.getVirtualCores() * by)), stepFactor.getVirtualCores()) ); }
        Hide
        sunilg Sunil G added a comment -

        Thanks Naganarasimha Garla and Bibin A Chundatt

        I think latest suggestion from Bibin seems more easier and straight forward. 1 for this approach. I am not seeing much of a pblm, a jenkins run is needed

        Show
        sunilg Sunil G added a comment - Thanks Naganarasimha Garla and Bibin A Chundatt I think latest suggestion from Bibin seems more easier and straight forward. 1 for this approach. I am not seeing much of a pblm, a jenkins run is needed
        Hide
        Naganarasimha Naganarasimha G R added a comment -

        1 from my side for the approach and it requires a patch upload even before jenkins run ...

        Show
        Naganarasimha Naganarasimha G R added a comment - 1 from my side for the approach and it requires a patch upload even before jenkins run ...
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Attaching patch as per the last conclusion

        Show
        bibinchundatt Bibin A Chundatt added a comment - Attaching patch as per the last conclusion
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 41s Maven dependency ordering for branch
        +1 mvninstall 13m 32s trunk passed
        +1 compile 9m 0s trunk passed
        +1 checkstyle 0m 53s trunk passed
        +1 mvnsite 1m 17s trunk passed
        +1 findbugs 2m 5s trunk passed
        +1 javadoc 1m 0s trunk passed
              Patch Compile Tests
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 1s the patch passed
        +1 compile 5m 7s the patch passed
        +1 javac 5m 7s the patch passed
        +1 checkstyle 0m 51s the patch passed
        +1 mvnsite 1m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 21s the patch passed
        +1 javadoc 0m 59s the patch passed
              Other Tests
        +1 unit 2m 23s hadoop-yarn-common in the patch passed.
        -1 unit 42m 26s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 34s The patch does not generate ASF License warnings.
        93m 25s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6428
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876036/YARN-6428.0003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 23faa58af6e9 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 82cb2a6
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16323/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16323/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16323/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 41s Maven dependency ordering for branch +1 mvninstall 13m 32s trunk passed +1 compile 9m 0s trunk passed +1 checkstyle 0m 53s trunk passed +1 mvnsite 1m 17s trunk passed +1 findbugs 2m 5s trunk passed +1 javadoc 1m 0s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 1s the patch passed +1 compile 5m 7s the patch passed +1 javac 5m 7s the patch passed +1 checkstyle 0m 51s the patch passed +1 mvnsite 1m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 21s the patch passed +1 javadoc 0m 59s the patch passed       Other Tests +1 unit 2m 23s hadoop-yarn-common in the patch passed. -1 unit 42m 26s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 93m 25s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6428 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876036/YARN-6428.0003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 23faa58af6e9 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 82cb2a6 Default Java 1.8.0_131 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/16323/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16323/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16323/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Testcase failures are not related to patch attached

        Show
        bibinchundatt Bibin A Chundatt added a comment - Testcase failures are not related to patch attached
        Hide
        Naganarasimha Naganarasimha G R added a comment -

        Thanks Bibin A Chundatt, Latest patch LGTM, will commit it if no more comments from others on it.

        Show
        Naganarasimha Naganarasimha G R added a comment - Thanks Bibin A Chundatt , Latest patch LGTM, will commit it if no more comments from others on it.
        Hide
        sunilg Sunil G added a comment -

        Thanks Bibin A Chundatt and Naganarasimha Garla

        +1 from my end.

        Show
        sunilg Sunil G added a comment - Thanks Bibin A Chundatt and Naganarasimha Garla +1 from my end.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 docker 0m 15s Docker failed to build yetus/hadoop:311d924.



        Subsystem Report/Notes
        JIRA Issue YARN-6428
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876184/YARN-6428.branch-2.8.2.0003.patch
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16331/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 15s Docker failed to build yetus/hadoop:311d924. Subsystem Report/Notes JIRA Issue YARN-6428 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876184/YARN-6428.branch-2.8.2.0003.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/16331/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Attaching branch-2.8 patch . Previous patch naming pattern was wrong

        Show
        bibinchundatt Bibin A Chundatt added a comment - Attaching branch-2.8 patch . Previous patch naming pattern was wrong
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              branch-2.8 Compile Tests
        0 mvndep 0m 21s Maven dependency ordering for branch
        +1 mvninstall 6m 43s branch-2.8 passed
        +1 compile 1m 56s branch-2.8 passed with JDK v1.8.0_131
        +1 compile 2m 18s branch-2.8 passed with JDK v1.7.0_131
        +1 checkstyle 0m 32s branch-2.8 passed
        +1 mvnsite 1m 10s branch-2.8 passed
        +1 findbugs 2m 19s branch-2.8 passed
        +1 javadoc 0m 49s branch-2.8 passed with JDK v1.8.0_131
        +1 javadoc 0m 56s branch-2.8 passed with JDK v1.7.0_131
              Patch Compile Tests
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 0m 59s the patch passed
        +1 compile 1m 50s the patch passed with JDK v1.8.0_131
        +1 javac 1m 50s the patch passed
        +1 compile 2m 15s the patch passed with JDK v1.7.0_131
        +1 javac 2m 15s the patch passed
        +1 checkstyle 0m 33s the patch passed
        +1 mvnsite 1m 9s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 44s the patch passed
        +1 javadoc 0m 46s the patch passed with JDK v1.8.0_131
        +1 javadoc 0m 55s the patch passed with JDK v1.7.0_131
              Other Tests
        +1 unit 2m 28s hadoop-yarn-common in the patch passed with JDK v1.7.0_131.
        -1 unit 75m 50s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        192m 13s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
        JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMTokens



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:d946387
        JIRA Issue YARN-6428
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876193/YARN-6428-branch-2.8.0003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 624b6844703c 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.8 / 5ad710b
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16332/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16332/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16332/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       branch-2.8 Compile Tests 0 mvndep 0m 21s Maven dependency ordering for branch +1 mvninstall 6m 43s branch-2.8 passed +1 compile 1m 56s branch-2.8 passed with JDK v1.8.0_131 +1 compile 2m 18s branch-2.8 passed with JDK v1.7.0_131 +1 checkstyle 0m 32s branch-2.8 passed +1 mvnsite 1m 10s branch-2.8 passed +1 findbugs 2m 19s branch-2.8 passed +1 javadoc 0m 49s branch-2.8 passed with JDK v1.8.0_131 +1 javadoc 0m 56s branch-2.8 passed with JDK v1.7.0_131       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 0m 59s the patch passed +1 compile 1m 50s the patch passed with JDK v1.8.0_131 +1 javac 1m 50s the patch passed +1 compile 2m 15s the patch passed with JDK v1.7.0_131 +1 javac 2m 15s the patch passed +1 checkstyle 0m 33s the patch passed +1 mvnsite 1m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 44s the patch passed +1 javadoc 0m 46s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 55s the patch passed with JDK v1.7.0_131       Other Tests +1 unit 2m 28s hadoop-yarn-common in the patch passed with JDK v1.7.0_131. -1 unit 75m 50s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 192m 13s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:d946387 JIRA Issue YARN-6428 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876193/YARN-6428-branch-2.8.0003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 624b6844703c 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / 5ad710b Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/16332/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16332/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16332/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              branch-2.8 Compile Tests
        0 mvndep 0m 21s Maven dependency ordering for branch
        +1 mvninstall 9m 17s branch-2.8 passed
        +1 compile 2m 11s branch-2.8 passed with JDK v1.8.0_131
        +1 compile 2m 24s branch-2.8 passed with JDK v1.7.0_131
        +1 checkstyle 0m 35s branch-2.8 passed
        +1 mvnsite 1m 13s branch-2.8 passed
        +1 findbugs 2m 27s branch-2.8 passed
        +1 javadoc 0m 51s branch-2.8 passed with JDK v1.8.0_131
        +1 javadoc 0m 58s branch-2.8 passed with JDK v1.7.0_131
              Patch Compile Tests
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 1m 0s the patch passed
        +1 compile 2m 7s the patch passed with JDK v1.8.0_131
        +1 javac 2m 7s the patch passed
        +1 compile 2m 19s the patch passed with JDK v1.7.0_131
        +1 javac 2m 19s the patch passed
        +1 checkstyle 0m 32s the patch passed
        +1 mvnsite 1m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 50s the patch passed
        +1 javadoc 0m 46s the patch passed with JDK v1.8.0_131
        +1 javadoc 0m 58s the patch passed with JDK v1.7.0_131
              Other Tests
        +1 unit 2m 25s hadoop-yarn-common in the patch passed with JDK v1.7.0_131.
        -1 unit 75m 33s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        195m 46s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerLazyPreemption
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
        JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMTokens



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:d946387
        JIRA Issue YARN-6428
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876193/YARN-6428-branch-2.8.0003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 1e029bc2e7a6 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.8 / 5ad710b
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16333/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16333/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16333/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       branch-2.8 Compile Tests 0 mvndep 0m 21s Maven dependency ordering for branch +1 mvninstall 9m 17s branch-2.8 passed +1 compile 2m 11s branch-2.8 passed with JDK v1.8.0_131 +1 compile 2m 24s branch-2.8 passed with JDK v1.7.0_131 +1 checkstyle 0m 35s branch-2.8 passed +1 mvnsite 1m 13s branch-2.8 passed +1 findbugs 2m 27s branch-2.8 passed +1 javadoc 0m 51s branch-2.8 passed with JDK v1.8.0_131 +1 javadoc 0m 58s branch-2.8 passed with JDK v1.7.0_131       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 0s the patch passed +1 compile 2m 7s the patch passed with JDK v1.8.0_131 +1 javac 2m 7s the patch passed +1 compile 2m 19s the patch passed with JDK v1.7.0_131 +1 javac 2m 19s the patch passed +1 checkstyle 0m 32s the patch passed +1 mvnsite 1m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 50s the patch passed +1 javadoc 0m 46s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 58s the patch passed with JDK v1.7.0_131       Other Tests +1 unit 2m 25s hadoop-yarn-common in the patch passed with JDK v1.7.0_131. -1 unit 75m 33s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 195m 46s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerLazyPreemption   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:d946387 JIRA Issue YARN-6428 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876193/YARN-6428-branch-2.8.0003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1e029bc2e7a6 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / 5ad710b Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/16333/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16333/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16333/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        Naganarasimha Naganarasimha G R added a comment -

        Thanks for the contribution Bibin A Chundatt and additional reviews from Sunil Govind. Have committed to trunk, branch-2 & branch-2.8

        Show
        Naganarasimha Naganarasimha G R added a comment - Thanks for the contribution Bibin A Chundatt and additional reviews from Sunil Govind . Have committed to trunk, branch-2 & branch-2.8
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11979 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11979/)
        YARN-6428. Queue AM limit is not honored in CS always. Contributed by (naganarasimha_gr: rev 626d730bfc4c1f6ea54e8a246a0a676cdb74937f)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11979 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11979/ ) YARN-6428 . Queue AM limit is not honored in CS always. Contributed by (naganarasimha_gr: rev 626d730bfc4c1f6ea54e8a246a0a676cdb74937f) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java

          People

          • Assignee:
            bibinchundatt Bibin A Chundatt
            Reporter:
            bibinchundatt Bibin A Chundatt
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development