Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4565

When sizeBasedWeight enabled for FairOrderingPolicy in CapacityScheduler, Sometimes lead to situation where all queue resources consumed by AMs only

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When sizeBasedWeight enabled for FairOrderingPolicy in CapacityScheduler, Sometimes lead to situation where all queue resources consumed by AMs only,

      So from users perpective it appears that all application in queue are stuck, whole queue capacity is comsumed by AMs

      1. YARN-4565.1.patch
        8 kB
        Wangda Tan
      2. YARN-4565.2.patch
        8 kB
        Wangda Tan
      3. YARN-4565.3.patch
        8 kB
        Wangda Tan

        Activity

        Hide
        karams Karam Singh added a comment -

        Came across this issue while experimenting with Fairness in queue with CapacityScheduler.
        Ecountered a situation when FairOrderingPolicy with SizeBasedWeight is enabled on queue in CapacityScheduler, while running GridMix V3 that all queue queue resources are consume AMs

        Following are setting:
        Cluster Total memory capacity 864GB, Global AMResourcePercent=0.1 Global MaxApplications=10000, minAllocationMb=2048, AM memory=2048, mapMemory=reduceMemory=2048

        Queue Settings:
        Capacity=10
        MaxCapacity=80
        UserLimitFactor=8,
        UserLimitPercent=100,
        FairOrderingPolicy with SizeBasedWeight=True

        According to this at max only 35 AMs can run at a time simultaneously and total 345 containers can run in queue,
        Which was verified While running GridMixV3 (which submits 760 applications) with FairOderingPolicy Only (without SizeBasedWeight)
        While when ran same test with FairOderingPolicy with SizeBasedWeight=true, 345 AMs(applications) running and since all queue resources are used by AMs no more containers can run, causing all application to get stuck.

        Looks like sizeBasedWeight somehow changes/overrides amResoucePercent.

        Show
        karams Karam Singh added a comment - Came across this issue while experimenting with Fairness in queue with CapacityScheduler. Ecountered a situation when FairOrderingPolicy with SizeBasedWeight is enabled on queue in CapacityScheduler, while running GridMix V3 that all queue queue resources are consume AMs Following are setting: Cluster Total memory capacity 864GB, Global AMResourcePercent=0.1 Global MaxApplications=10000, minAllocationMb=2048, AM memory=2048, mapMemory=reduceMemory=2048 Queue Settings: Capacity=10 MaxCapacity=80 UserLimitFactor=8, UserLimitPercent=100, FairOrderingPolicy with SizeBasedWeight=True According to this at max only 35 AMs can run at a time simultaneously and total 345 containers can run in queue, Which was verified While running GridMixV3 (which submits 760 applications) with FairOderingPolicy Only (without SizeBasedWeight) While when ran same test with FairOderingPolicy with SizeBasedWeight=true, 345 AMs(applications) running and since all queue resources are used by AMs no more containers can run, causing all application to get stuck. Looks like sizeBasedWeight somehow changes/overrides amResoucePercent.
        Hide
        karams Karam Singh added a comment -

        GridMix V3 Trace information with GridMix run settings for test
        Trace runs 760 jobs with settings:
        gridmix.client.pending.queue.depth=10,
        gridmix.job-submission.policy=REPLAY
        gridmix.client.submit.threads=5, gridmix.submit.multiplier=0.0001
        gridmix.job.type=SLEEPJOB, mapreduce.framework.name=yarn
        gridmix.sleep.max-map-time=5000, gridmix.sleep.max-reduce-time=5000
        gridmix.user.resolve.class=org.apache.hadoop.mapred.gridmix.RoundRobinUserResolver
        With Users file containing 4 users for RoundRobinUserResolver

        Debugging with Tan, Wangda

        Show
        karams Karam Singh added a comment - GridMix V3 Trace information with GridMix run settings for test Trace runs 760 jobs with settings: gridmix.client.pending.queue.depth=10, gridmix.job-submission.policy=REPLAY gridmix.client.submit.threads=5, gridmix.submit.multiplier=0.0001 gridmix.job.type=SLEEPJOB, mapreduce.framework.name=yarn gridmix.sleep.max-map-time=5000, gridmix.sleep.max-reduce-time=5000 gridmix.user.resolve.class=org.apache.hadoop.mapred.gridmix.RoundRobinUserResolver With Users file containing 4 users for RoundRobinUserResolver Debugging with Tan, Wangda
        Hide
        karams Karam Singh added a comment -

        From offline chat with Tan, Wangda
        I think I found what happened:
        When application added to scheduler, CapacityScheduler#allocate will be called:
        if (updateDemandForQueue != null)

        { updateDemandForQueue.getOrderingPolicy().demandUpdated(application); }

        And in FairOrderingPolicy#demandUpdated, when sizeBasedWeight is enabled:
        if (sizeBasedWeight)

        { entityRequiresReordering(schedulableEntity); }

        It will call reordering the schedulableEntity. It will essentially insert the entity to the TreeSet.
        This could happen before the application (schedulableEntity) is not in active application list. So we can get application's container allocated before application activated.

        Show
        karams Karam Singh added a comment - From offline chat with Tan, Wangda I think I found what happened: When application added to scheduler, CapacityScheduler#allocate will be called: if (updateDemandForQueue != null) { updateDemandForQueue.getOrderingPolicy().demandUpdated(application); } And in FairOrderingPolicy#demandUpdated, when sizeBasedWeight is enabled: if (sizeBasedWeight) { entityRequiresReordering(schedulableEntity); } It will call reordering the schedulableEntity. It will essentially insert the entity to the TreeSet. This could happen before the application (schedulableEntity) is not in active application list. So we can get application's container allocated before application activated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached ver.1 patch for review.

        Show
        leftnoteasy Wangda Tan added a comment - Attached ver.1 patch for review.
        Hide
        leftnoteasy Wangda Tan added a comment -

        branch-2.7 doesn't have this feature, removed from affected version

        Show
        leftnoteasy Wangda Tan added a comment - branch-2.7 doesn't have this feature, removed from affected version
        Hide
        Naganarasimha Naganarasimha G R added a comment -

        Hi Tan, Wangda,
        Overall the approach seems to be good, but few queries

        • earlier synchronization was only for modifying entitiesToReorder but removeSchedulableEntity is made synchronized instead of only covering entitiesToReorder.remove(s.getId()); would it be better to have synchronized(this) block here ?
        • though the current approach is fine Any particular reason not to use application.isWaitingForAMContainer() than modifying AbstractComparatorOrderingPolicy, as its much simple?
        Show
        Naganarasimha Naganarasimha G R added a comment - Hi Tan, Wangda , Overall the approach seems to be good, but few queries earlier synchronization was only for modifying entitiesToReorder but removeSchedulableEntity is made synchronized instead of only covering entitiesToReorder.remove(s.getId()); would it be better to have synchronized(this) block here ? though the current approach is fine Any particular reason not to use application.isWaitingForAMContainer() than modifying AbstractComparatorOrderingPolicy , as its much simple?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 40s trunk passed
        +1 compile 0m 29s trunk passed with JDK v1.8.0_66
        +1 compile 0m 30s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 39s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 1m 11s trunk passed
        -1 javadoc 0m 21s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.8.0_66.
        +1 javadoc 0m 27s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 31s the patch passed
        +1 compile 0m 24s the patch passed with JDK v1.8.0_66
        +1 javac 0m 24s the patch passed
        +1 compile 0m 28s the patch passed with JDK v1.7.0_91
        +1 javac 0m 28s the patch passed
        -1 checkstyle 0m 13s Patch generated 1 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 41, now 42).
        +1 mvnsite 0m 35s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 18s the patch passed
        -1 javadoc 0m 18s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        +1 javadoc 0m 26s the patch passed with JDK v1.7.0_91
        -1 unit 65m 36s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        -1 unit 66m 18s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 17s Patch does not generate ASF License warnings.
        149m 41s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
        JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781340/YARN-4565.1.patch
        JIRA Issue YARN-4565
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux be0d8d561752 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 109e528
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10216/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10216/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10216/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10216/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10216/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10216/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10216/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10216/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Max memory used 75MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10216/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 40s trunk passed +1 compile 0m 29s trunk passed with JDK v1.8.0_66 +1 compile 0m 30s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 39s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 11s trunk passed -1 javadoc 0m 21s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.8.0_66. +1 javadoc 0m 27s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 31s the patch passed +1 compile 0m 24s the patch passed with JDK v1.8.0_66 +1 javac 0m 24s the patch passed +1 compile 0m 28s the patch passed with JDK v1.7.0_91 +1 javac 0m 28s the patch passed -1 checkstyle 0m 13s Patch generated 1 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 41, now 42). +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 18s the patch passed -1 javadoc 0m 18s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. +1 javadoc 0m 26s the patch passed with JDK v1.7.0_91 -1 unit 65m 36s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 66m 18s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 149m 41s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781340/YARN-4565.1.patch JIRA Issue YARN-4565 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux be0d8d561752 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 109e528 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10216/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10216/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10216/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10216/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10216/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10216/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10216/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10216/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10216/console This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Naganarasimha G R, suggestions make sense to me, updated patch to use isWaitingForAMContainer

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Naganarasimha G R , suggestions make sense to me, updated patch to use isWaitingForAMContainer
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 50s trunk passed
        +1 compile 0m 30s trunk passed with JDK v1.8.0_66
        +1 compile 0m 33s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 40s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 1m 12s trunk passed
        -1 javadoc 0m 21s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.8.0_66.
        +1 javadoc 0m 27s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 32s the patch passed
        +1 compile 0m 24s the patch passed with JDK v1.8.0_66
        +1 javac 0m 24s the patch passed
        +1 compile 0m 28s the patch passed with JDK v1.7.0_91
        +1 javac 0m 28s the patch passed
        -1 checkstyle 0m 15s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 184, now 185).
        +1 mvnsite 0m 36s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 18s the patch passed
        -1 javadoc 0m 19s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        +1 javadoc 0m 25s the patch passed with JDK v1.7.0_91
        -1 unit 65m 35s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        -1 unit 66m 21s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 17s Patch does not generate ASF License warnings.
        149m 53s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
        JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781657/YARN-4565.2.patch
        JIRA Issue YARN-4565
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 874f2b9301ce 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 9f4bf3b
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10232/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10232/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10232/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10232/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10232/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10232/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10232/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10232/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Max memory used 75MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10232/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 50s trunk passed +1 compile 0m 30s trunk passed with JDK v1.8.0_66 +1 compile 0m 33s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 40s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 12s trunk passed -1 javadoc 0m 21s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.8.0_66. +1 javadoc 0m 27s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 32s the patch passed +1 compile 0m 24s the patch passed with JDK v1.8.0_66 +1 javac 0m 24s the patch passed +1 compile 0m 28s the patch passed with JDK v1.7.0_91 +1 javac 0m 28s the patch passed -1 checkstyle 0m 15s Patch generated 2 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 184, now 185). +1 mvnsite 0m 36s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 18s the patch passed -1 javadoc 0m 19s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. +1 javadoc 0m 25s the patch passed with JDK v1.7.0_91 -1 unit 65m 35s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 66m 21s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 149m 53s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781657/YARN-4565.2.patch JIRA Issue YARN-4565 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 874f2b9301ce 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9f4bf3b Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10232/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10232/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/10232/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10232/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10232/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10232/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10232/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10232/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10232/console This message was automatically generated.
        Hide
        Naganarasimha Naganarasimha G R added a comment -

        Hi Tan, Wangda,
        Patch is generally fine with very small nits

        • one Checkstyle issue is related to patch(Unused import ) .
        • In TestFairOrderingPolicy
          • MockNM nm1 = rm.registerNode("h1:1234", 10 * GB); // label = x i think assigning to a variable is not required here
          • OrderingPolicy policy = lq.getOrderingPolicy(); we can use generics (OrderingPolicy<FiCaSchedulerApp>) here to avoid warnings
          • Assert.assertTrue(((FairOrderingPolicy)policy).getSizeBasedWeight()); similar to the above comment for FairOrderingPolicy
        Show
        Naganarasimha Naganarasimha G R added a comment - Hi Tan, Wangda , Patch is generally fine with very small nits one Checkstyle issue is related to patch(Unused import ) . In TestFairOrderingPolicy MockNM nm1 = rm.registerNode("h1:1234", 10 * GB); // label = x i think assigning to a variable is not required here OrderingPolicy policy = lq.getOrderingPolicy(); we can use generics (OrderingPolicy<FiCaSchedulerApp>) here to avoid warnings Assert.assertTrue(((FairOrderingPolicy)policy).getSizeBasedWeight()); similar to the above comment for FairOrderingPolicy
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Naganarasimha G R, attached ver.3 patch addressed all your comments.

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Naganarasimha G R , attached ver.3 patch addressed all your comments.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 33s trunk passed
        +1 compile 0m 26s trunk passed with JDK v1.8.0_66
        +1 compile 0m 30s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 35s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 1m 11s trunk passed
        +1 javadoc 0m 20s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 27s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 32s the patch passed
        +1 compile 0m 24s the patch passed with JDK v1.8.0_66
        +1 javac 0m 24s the patch passed
        +1 compile 0m 28s the patch passed with JDK v1.7.0_91
        +1 javac 0m 28s the patch passed
        -1 checkstyle 0m 15s Patch generated 1 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 184, now 184).
        +1 mvnsite 0m 34s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 18s the patch passed
        +1 javadoc 0m 18s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 24s the patch passed with JDK v1.7.0_91
        -1 unit 60m 6s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        -1 unit 61m 13s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 18s Patch does not generate ASF License warnings.
        138m 40s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
        JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782190/YARN-4565.3.patch
        JIRA Issue YARN-4565
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7a52bc152b02 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 7b0964f
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10276/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10276/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10276/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10276/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10276/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10276/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10276/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 33s trunk passed +1 compile 0m 26s trunk passed with JDK v1.8.0_66 +1 compile 0m 30s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 11s trunk passed +1 javadoc 0m 20s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 27s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 32s the patch passed +1 compile 0m 24s the patch passed with JDK v1.8.0_66 +1 javac 0m 24s the patch passed +1 compile 0m 28s the patch passed with JDK v1.7.0_91 +1 javac 0m 28s the patch passed -1 checkstyle 0m 15s Patch generated 1 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 184, now 184). +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 18s the patch passed +1 javadoc 0m 18s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_91 -1 unit 60m 6s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 61m 13s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 138m 40s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782190/YARN-4565.3.patch JIRA Issue YARN-4565 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7a52bc152b02 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7b0964f Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10276/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10276/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10276/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10276/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10276/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10276/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10276/console This message was automatically generated.
        Hide
        Naganarasimha Naganarasimha G R added a comment -

        +1, Patch LGTM

        Show
        Naganarasimha Naganarasimha G R added a comment - +1, Patch LGTM
        Hide
        leftnoteasy Wangda Tan added a comment -
        Show
        leftnoteasy Wangda Tan added a comment - Thanks Naganarasimha G R !
        Hide
        jianhe Jian He added a comment -

        Committed to trunk and branch-2, branch-2.8 , thanks Wangda !

        Thanks Karam Singh for reporting and analysis !
        Thanks Naganarasimha G R for reviewing the patch !

        Show
        jianhe Jian He added a comment - Committed to trunk and branch-2, branch-2.8 , thanks Wangda ! Thanks Karam Singh for reporting and analysis ! Thanks Naganarasimha G R for reviewing the patch !
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9136 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9136/)
        YARN-4565. Fix a bug that leads to AM resource limit not hornored when (jianhe: rev edc43a9097530fd469dee47d4fefd091818331e5)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9136 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9136/ ) YARN-4565 . Fix a bug that leads to AM resource limit not hornored when (jianhe: rev edc43a9097530fd469dee47d4fefd091818331e5) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java hadoop-yarn-project/CHANGES.txt

          People

          • Assignee:
            leftnoteasy Wangda Tan
            Reporter:
            karams Karam Singh
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development