Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4161

Capacity Scheduler : Assign single or multiple containers per heart beat driven by configuration

    Details

      Description

      Capacity Scheduler right now schedules multiple containers per heart beat if there are more resources available in the node.
      This approach works fine however in some cases its not distribute the load across the cluster hence throughput of the cluster suffers. I am adding feature to drive that using configuration by that we can control the number of containers assigned per heart beat.

      1. YARN-4161.002.patch
        18 kB
        Wei Yan
      2. YARN-4161.003.patch
        17 kB
        Wei Yan
      3. YARN-4161.004.patch
        19 kB
        Wei Yan
      4. YARN-4161.005.patch
        18 kB
        Wei Yan
      5. YARN-4161.006.patch
        18 kB
        Wei Yan
      6. YARN-4161.branch-2.006.patch
        18 kB
        Wangda Tan
      7. YARN-4161.patch
        16 kB
        Mayank Bansal
      8. YARN-4161.patch.1
        16 kB
        Wei Yan

        Activity

        Hide
        leftnoteasy Wangda Tan added a comment -

        Committed to branch-2 as well.

        Show
        leftnoteasy Wangda Tan added a comment - Committed to branch-2 as well.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Test failure is not related, pushing this to branch-2.

        Show
        leftnoteasy Wangda Tan added a comment - Test failure is not related, pushing this to branch-2.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 17m 53s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              branch-2 Compile Tests
        +1 mvninstall 7m 5s branch-2 passed
        +1 compile 0m 39s branch-2 passed with JDK v1.8.0_144
        +1 compile 0m 35s branch-2 passed with JDK v1.7.0_131
        +1 checkstyle 0m 27s branch-2 passed
        +1 mvnsite 0m 43s branch-2 passed
        +1 findbugs 1m 19s branch-2 passed
        +1 javadoc 0m 22s branch-2 passed with JDK v1.8.0_144
        +1 javadoc 0m 26s branch-2 passed with JDK v1.7.0_131
              Patch Compile Tests
        +1 mvninstall 0m 34s the patch passed
        +1 compile 0m 34s the patch passed with JDK v1.8.0_144
        +1 javac 0m 34s the patch passed
        +1 compile 0m 34s the patch passed with JDK v1.7.0_131
        +1 javac 0m 34s the patch passed
        +1 checkstyle 0m 25s the patch passed
        +1 mvnsite 0m 37s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 24s the patch passed
        +1 javadoc 0m 20s the patch passed with JDK v1.8.0_144
        +1 javadoc 0m 22s the patch passed with JDK v1.7.0_131
              Other Tests
        -1 unit 50m 18s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        134m 36s



        Reason Tests
        JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
          hadoop.yarn.server.resourcemanager.TestLeaderElectorService
          hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:5e40efe
        JIRA Issue YARN-4161
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880681/YARN-4161.branch-2.006.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 577cc68bf93b 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 050c50e
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16748/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16748/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16748/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 17m 53s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       branch-2 Compile Tests +1 mvninstall 7m 5s branch-2 passed +1 compile 0m 39s branch-2 passed with JDK v1.8.0_144 +1 compile 0m 35s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 27s branch-2 passed +1 mvnsite 0m 43s branch-2 passed +1 findbugs 1m 19s branch-2 passed +1 javadoc 0m 22s branch-2 passed with JDK v1.8.0_144 +1 javadoc 0m 26s branch-2 passed with JDK v1.7.0_131       Patch Compile Tests +1 mvninstall 0m 34s the patch passed +1 compile 0m 34s the patch passed with JDK v1.8.0_144 +1 javac 0m 34s the patch passed +1 compile 0m 34s the patch passed with JDK v1.7.0_131 +1 javac 0m 34s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 0m 37s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 24s the patch passed +1 javadoc 0m 20s the patch passed with JDK v1.8.0_144 +1 javadoc 0m 22s the patch passed with JDK v1.7.0_131       Other Tests -1 unit 50m 18s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 134m 36s Reason Tests JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.server.resourcemanager.TestLeaderElectorService   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue YARN-4161 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880681/YARN-4161.branch-2.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 577cc68bf93b 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 050c50e Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/16748/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16748/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16748/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached patch for branch-2 to run Jenkins.

        Show
        leftnoteasy Wangda Tan added a comment - Attached patch for branch-2 to run Jenkins.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Committed to trunk, thanks Wei Yan and reviews from Sunil G.

        Show
        leftnoteasy Wangda Tan added a comment - Committed to trunk, thanks Wei Yan and reviews from Sunil G .
        Hide
        sunilg Sunil G added a comment -

        Thanks Wei Yan
        That clarifies my doubt.. multiple-assignments-enabled is by default TRUE and maxAssignPerHeartbeat by default is -1. Hence by default, old behavior is covered.

        Looks fine then. I could get this in later today if no issues or Wangda ll also help as mentioned above,

        Show
        sunilg Sunil G added a comment - Thanks Wei Yan That clarifies my doubt.. multiple-assignments-enabled is by default TRUE and maxAssignPerHeartbeat by default is -1. Hence by default, old behavior is covered. Looks fine then. I could get this in later today if no issues or Wangda ll also help as mentioned above,
        Hide
        ywskycn Wei Yan added a comment -

        Yes, here we introduce one more condition check. But by default, the new condition check is always positive. So when upgrading to new YARN but with old config files, the behavior is still the same. The new condition comes to play only after users make some changes there, in that case, users should be aware of this new things.

        Show
        ywskycn Wei Yan added a comment - Yes, here we introduce one more condition check. But by default, the new condition check is always positive. So when upgrading to new YARN but with old config files, the behavior is still the same. The new condition comes to play only after users make some changes there, in that case, users should be aware of this new things.
        Hide
        sunilg Sunil G added a comment -

        Extremely sorry for sharing a quick doubt last moment here.

        yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments

        After this patch, above config can work only if below conditions are met:

        1. yarn.scheduler.capacity.per-node-heartbeat.multiple-assignments-enabled should be enabled.
        2. (maxAssignPerHeartbeat == -1 || assignedContainers < maxAssignPerHeartbeat) should be +ve.

        This means that, existing behavior related to maximum-offswitch-assignments wont work provided above 2 cases are true. I think this is a regression. So i would like to point this to Wangda Tan as well. Thoughts?

        Show
        sunilg Sunil G added a comment - Extremely sorry for sharing a quick doubt last moment here. yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments After this patch, above config can work only if below conditions are met: yarn.scheduler.capacity.per-node-heartbeat.multiple-assignments-enabled should be enabled. (maxAssignPerHeartbeat == -1 || assignedContainers < maxAssignPerHeartbeat) should be +ve. This means that, existing behavior related to maximum-offswitch-assignments wont work provided above 2 cases are true. I think this is a regression. So i would like to point this to Wangda Tan as well. Thoughts?
        Hide
        sunilg Sunil G added a comment -

        +1 LGTM

        Show
        sunilg Sunil G added a comment - +1 LGTM
        Hide
        leftnoteasy Wangda Tan added a comment -

        And reassigned to Wei Yan.

        Show
        leftnoteasy Wangda Tan added a comment - And reassigned to Wei Yan .
        Hide
        leftnoteasy Wangda Tan added a comment -

        Latest patch looks good, +1, will commit tomorrow if nobody against.

        Show
        leftnoteasy Wangda Tan added a comment - Latest patch looks good, +1, will commit tomorrow if nobody against.
        Hide
        ywskycn Wei Yan added a comment -

        Sunil G could u help review the latest patch? thanks.

        Show
        ywskycn Wei Yan added a comment - Sunil G could u help review the latest patch? thanks.
        Hide
        sunilg Sunil G added a comment -

        Yes. Test case failures are unrelated. I will check latest patch and will share comments if any.

        Show
        sunilg Sunil G added a comment - Yes. Test case failures are unrelated. I will check latest patch and will share comments if any.
        Hide
        ywskycn Wei Yan added a comment -

        The TestDelegationTokenRenewer test error looks un-related. Sunil G, any idea?

        Show
        ywskycn Wei Yan added a comment - The TestDelegationTokenRenewer test error looks un-related. Sunil G , any idea?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 23s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 14m 26s trunk passed
        +1 compile 0m 36s trunk passed
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 0m 37s trunk passed
        +1 findbugs 1m 4s trunk passed
        +1 javadoc 0m 23s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 33s the patch passed
        +1 compile 0m 34s the patch passed
        +1 javac 0m 34s the patch passed
        +1 checkstyle 0m 26s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 327 unchanged - 2 fixed = 327 total (was 329)
        +1 mvnsite 0m 36s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 9s the patch passed
        +1 javadoc 0m 20s the patch passed
              Other Tests
        -1 unit 42m 48s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        65m 57s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-4161
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878833/YARN-4161.006.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux bf6c3876a324 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ac9489f
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16538/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16538/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16538/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 14m 26s trunk passed +1 compile 0m 36s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 37s trunk passed +1 findbugs 1m 4s trunk passed +1 javadoc 0m 23s trunk passed       Patch Compile Tests +1 mvninstall 0m 33s the patch passed +1 compile 0m 34s the patch passed +1 javac 0m 34s the patch passed +1 checkstyle 0m 26s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 327 unchanged - 2 fixed = 327 total (was 329) +1 mvnsite 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 9s the patch passed +1 javadoc 0m 20s the patch passed       Other Tests -1 unit 42m 48s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 65m 57s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-4161 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878833/YARN-4161.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bf6c3876a324 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ac9489f Default Java 1.8.0_131 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/16538/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16538/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16538/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ywskycn Wei Yan added a comment -

        Good catch, Sunil G. Update a new patch to include that check.

        Show
        ywskycn Wei Yan added a comment - Good catch, Sunil G . Update a new patch to include that check.
        Hide
        sunilg Sunil G added a comment -

        One doubt:
        I think we are not checking below condition in canAllocateMore. I might have lost context, please help to correct me if I am wrong.

              if (assignment.getAssignmentInformation().getNumReservations() == 0) {		
                return true;		
              }
        
        Show
        sunilg Sunil G added a comment - One doubt: I think we are not checking below condition in canAllocateMore . I might have lost context, please help to correct me if I am wrong. if (assignment.getAssignmentInformation().getNumReservations() == 0) { return true ; }
        Hide
        ywskycn Wei Yan added a comment -

        The failed testcase passed locally, looks like not related to this patch.
        Thanks, Tan, Wangda. For the documentation part, created YARN-6851 and will put details there.

        Show
        ywskycn Wei Yan added a comment - The failed testcase passed locally, looks like not related to this patch. Thanks, Tan, Wangda . For the documentation part, created YARN-6851 and will put details there.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 12m 38s trunk passed
        +1 compile 0m 31s trunk passed
        +1 checkstyle 0m 30s trunk passed
        +1 mvnsite 0m 35s trunk passed
        +1 findbugs 0m 55s trunk passed
        +1 javadoc 0m 22s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 31s the patch passed
        +1 compile 0m 29s the patch passed
        +1 javac 0m 29s the patch passed
        +1 checkstyle 0m 28s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 327 unchanged - 2 fixed = 327 total (was 329)
        +1 mvnsite 0m 32s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 2s the patch passed
        +1 javadoc 0m 18s the patch passed
              Other Tests
        -1 unit 44m 56s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        65m 37s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-4161
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878045/YARN-4161.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 5a5473706e3f 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c21c260
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16492/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16492/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16492/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 12m 38s trunk passed +1 compile 0m 31s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 35s trunk passed +1 findbugs 0m 55s trunk passed +1 javadoc 0m 22s trunk passed       Patch Compile Tests +1 mvninstall 0m 31s the patch passed +1 compile 0m 29s the patch passed +1 javac 0m 29s the patch passed +1 checkstyle 0m 28s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 327 unchanged - 2 fixed = 327 total (was 329) +1 mvnsite 0m 32s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 2s the patch passed +1 javadoc 0m 18s the patch passed       Other Tests -1 unit 44m 56s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 65m 37s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-4161 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878045/YARN-4161.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5a5473706e3f 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c21c260 Default Java 1.8.0_131 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/16492/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16492/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16492/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Wei Yan, in general the patch looks good to me. And could you open a follow up JIRA to track documentation changes (or add them to the same JIRA if you like). Now we have 3 knobs to control #containers allocated for each node heartbeat. It's better to clearly document them.

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Wei Yan , in general the patch looks good to me. And could you open a follow up JIRA to track documentation changes (or add them to the same JIRA if you like). Now we have 3 knobs to control #containers allocated for each node heartbeat. It's better to clearly document them.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 15s trunk passed
        +1 compile 0m 34s trunk passed
        +1 checkstyle 0m 30s trunk passed
        +1 mvnsite 0m 36s trunk passed
        +1 findbugs 0m 59s trunk passed
        +1 javadoc 0m 21s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 33s the patch passed
        +1 compile 0m 31s the patch passed
        +1 javac 0m 31s the patch passed
        -0 checkstyle 0m 27s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 3 new + 324 unchanged - 5 fixed = 327 total (was 329)
        +1 mvnsite 0m 34s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 4s the patch passed
        +1 javadoc 0m 18s the patch passed
              Other Tests
        +1 unit 43m 26s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        64m 58s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-4161
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878032/YARN-4161.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 4c985ef3d95a 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 077fcf6
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16490/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16490/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16490/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 15s trunk passed +1 compile 0m 34s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 36s trunk passed +1 findbugs 0m 59s trunk passed +1 javadoc 0m 21s trunk passed       Patch Compile Tests +1 mvninstall 0m 33s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed -0 checkstyle 0m 27s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 3 new + 324 unchanged - 5 fixed = 327 total (was 329) +1 mvnsite 0m 34s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 4s the patch passed +1 javadoc 0m 18s the patch passed       Other Tests +1 unit 43m 26s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 64m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-4161 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878032/YARN-4161.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4c985ef3d95a 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 077fcf6 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16490/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16490/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16490/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ywskycn Wei Yan added a comment -

        Update patch to fix checkstyle issues. The test errors look not related to this patch.

        Show
        ywskycn Wei Yan added a comment - Update patch to fix checkstyle issues. The test errors look not related to this patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 41s trunk passed
        +1 compile 0m 37s trunk passed
        +1 checkstyle 0m 30s trunk passed
        +1 mvnsite 0m 38s trunk passed
        +1 findbugs 0m 57s trunk passed
        +1 javadoc 0m 22s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 34s the patch passed
        +1 compile 0m 33s the patch passed
        +1 javac 0m 33s the patch passed
        -0 checkstyle 0m 29s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 47 new + 327 unchanged - 2 fixed = 374 total (was 329)
        +1 mvnsite 0m 35s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 11s the patch passed
        +1 javadoc 0m 20s the patch passed
              Other Tests
        -1 unit 42m 42s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        64m 55s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-4161
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878013/YARN-4161.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7cfb2da750ea 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 04ff412
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16488/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16488/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16488/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16488/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 41s trunk passed +1 compile 0m 37s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 38s trunk passed +1 findbugs 0m 57s trunk passed +1 javadoc 0m 22s trunk passed       Patch Compile Tests +1 mvninstall 0m 34s the patch passed +1 compile 0m 33s the patch passed +1 javac 0m 33s the patch passed -0 checkstyle 0m 29s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 47 new + 327 unchanged - 2 fixed = 374 total (was 329) +1 mvnsite 0m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 11s the patch passed +1 javadoc 0m 20s the patch passed       Other Tests -1 unit 42m 42s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 64m 55s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-4161 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878013/YARN-4161.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7cfb2da750ea 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 04ff412 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16488/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16488/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16488/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16488/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ywskycn Wei Yan added a comment -

        rebase the patch

        Show
        ywskycn Wei Yan added a comment - rebase the patch
        Hide
        sunilg Sunil G added a comment -

        Wei Yan could u please help to rebase the patch

        Show
        sunilg Sunil G added a comment - Wei Yan could u please help to rebase the patch
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 5s YARN-4161 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue YARN-4161
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875321/YARN-4161.002.patch
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16436/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s YARN-4161 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-4161 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875321/YARN-4161.002.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/16436/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ywskycn Wei Yan added a comment -

        A new diff based on previous comments:
        (1) Rename the config fields to follow CS style;
        (2) Align the checking with max-offswitch one.

        Show
        ywskycn Wei Yan added a comment - A new diff based on previous comments: (1) Rename the config fields to follow CS style; (2) Align the checking with max-offswitch one.
        Hide
        sunilg Sunil G added a comment - - edited

        Yes. so you ok with existing two config fields?

        Yes. We can do two configs. Any other options with one config param may confuse user more.

        Yes, now checking is little confused. I refactored to more clearer.

        Makes sene

        For now, I more prefer to do it in scheduler-level

        Lets do this at scheduler level for now. I think we can extend to queue level a little later when its really needed. Else it may complicate a little more.

        Show
        sunilg Sunil G added a comment - - edited Yes. so you ok with existing two config fields? Yes. We can do two configs. Any other options with one config param may confuse user more. Yes, now checking is little confused. I refactored to more clearer. Makes sene For now, I more prefer to do it in scheduler-level Lets do this at scheduler level for now. I think we can extend to queue level a little later when its really needed. Else it may complicate a little more.
        Hide
        ywskycn Wei Yan added a comment -

        I got your point. I was trying to point out to an existing configuration in CS named yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments. I thought of keep this syntax and it seems more of similar nature to me.

        i c. In that case, I agree with you to change to the same syntax as the offswitch one.

        Thanks for clarifying. I understood the reason behind that. On an another note, maxAssign=0 is meaning less. Correct? So value less than 0 could be considered for infinite. And a boolean variable to switch-on / off the feature.

        Yes. so you ok with existing two config fields?

        I think assignMultiple should not impact existing "maximum-offswitch-assignments" feature. I can see that current patch will skip when assignMultiple is configured as false.

        Yes, now checking is little confused. I refactored to more clearer.

          private boolean canAllocateMore(CSAssignment assignment, int offswitchCount,
              int assignedContainers) {
            // Current assignment shouldn't be empty
            if (assignment == null
                    || Resources.equals(assignment.getResource(), Resources.none())) {
              return false;
            }
        
            // offswitch assignment should be under threshold
            if (offswitchCount >= offswitchPerHeartbeatLimit) {
              return false;
            }
        
            // assignMultiple should be ON, and assignedContainers should be under threshold
            return !(!assignMultiple
                    || (maxAllocationPerNode != -1 && maxAllocationPerNode <= assignedContainers));
          }
        

        Could we rename assignMultiple to something like assign-multiple-containers-per-heartbeat

        As mentioned above, I'll rename configs to follow CS's offswitch one.

        Are you planning to consider this per queue as well ? Or only at CS level from top for now.

        For now, I more prefer to do it in scheduler-level. As this feature is something that works in cluster-level, to balance the workload across NMs. I don't have a concrete case that we need queue-level. Do u have cases that it would be better to do per-queue config?

        Show
        ywskycn Wei Yan added a comment - I got your point. I was trying to point out to an existing configuration in CS named yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments. I thought of keep this syntax and it seems more of similar nature to me. i c. In that case, I agree with you to change to the same syntax as the offswitch one. Thanks for clarifying. I understood the reason behind that. On an another note, maxAssign=0 is meaning less. Correct? So value less than 0 could be considered for infinite. And a boolean variable to switch-on / off the feature. Yes. so you ok with existing two config fields? I think assignMultiple should not impact existing "maximum-offswitch-assignments" feature. I can see that current patch will skip when assignMultiple is configured as false. Yes, now checking is little confused. I refactored to more clearer. private boolean canAllocateMore(CSAssignment assignment, int offswitchCount, int assignedContainers) { // Current assignment shouldn't be empty if (assignment == null || Resources.equals(assignment.getResource(), Resources.none())) { return false ; } // offswitch assignment should be under threshold if (offswitchCount >= offswitchPerHeartbeatLimit) { return false ; } // assignMultiple should be ON, and assignedContainers should be under threshold return !(!assignMultiple || (maxAllocationPerNode != -1 && maxAllocationPerNode <= assignedContainers)); } Could we rename assignMultiple to something like assign-multiple-containers-per-heartbeat As mentioned above, I'll rename configs to follow CS's offswitch one. Are you planning to consider this per queue as well ? Or only at CS level from top for now. For now, I more prefer to do it in scheduler-level. As this feature is something that works in cluster-level, to balance the workload across NMs. I don't have a concrete case that we need queue-level. Do u have cases that it would be better to do per-queue config?
        Hide
        sunilg Sunil G added a comment -

        I got your point. I was trying to point out to an existing configuration in CS named yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments. I thought of keep this syntax and it seems more of similar nature to me.

        We still need assignMultiple one, as maxAssign=-1 mostly means (assignMultiple is enabled, and the max allowed allocation is unlimited).

        Thanks for clarifying. I understood the reason behind that. On an another note, maxAssign=0 is meaning less. Correct? So value less than 0 could be considered for infinite. And a boolean variable to switch-on / off the feature.

        Few more doubts.

        1. I think assignMultiple should not impact existing "maximum-offswitch-assignments" feature. I can see that current patch will skip when assignMultiple is configured as false.
        2. Could we rename assignMultiple to something like assign-multiple-containers-per-heartbeat
        3. Are you planning to consider this per queue as well ? Or only at CS level from top for now.
        Show
        sunilg Sunil G added a comment - I got your point. I was trying to point out to an existing configuration in CS named yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments . I thought of keep this syntax and it seems more of similar nature to me. We still need assignMultiple one, as maxAssign=-1 mostly means (assignMultiple is enabled, and the max allowed allocation is unlimited). Thanks for clarifying. I understood the reason behind that. On an another note, maxAssign=0 is meaning less. Correct? So value less than 0 could be considered for infinite. And a boolean variable to switch-on / off the feature. Few more doubts. I think assignMultiple should not impact existing "maximum-offswitch-assignments" feature. I can see that current patch will skip when assignMultiple is configured as false. Could we rename assignMultiple to something like assign-multiple-containers-per-heartbeat Are you planning to consider this per queue as well ? Or only at CS level from top for now.
        Hide
        ywskycn Wei Yan added a comment -

        Thanks for the comments Sunil G.

        For the configuration names, I followed the same names in FairScheduler. Renaming to different ones may make users little confused.

        We still need assignMultiple one, as maxAssign=-1 mostly means (assignMultiple is enabled, and the max allowed allocation is unlimited).

        Show
        ywskycn Wei Yan added a comment - Thanks for the comments Sunil G . For the configuration names, I followed the same names in FairScheduler. Renaming to different ones may make users little confused. We still need assignMultiple one, as maxAssign=-1 mostly means (assignMultiple is enabled, and the max allowed allocation is unlimited).
        Hide
        sunilg Sunil G added a comment -

        Thanks Wei Yan for rebasing the patch.
        Few high level comments

        1. Could you please name patch to something like YARN-4161.002.patch etc by following the naming convention. It will be more easier.
        2. I thoughts we can rename the configuration to "yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments". More thoughts are welcome
        3. Once 2 is confirmed, rename all variables and getters/setter in align with config name
        4. I prefer not to have a config like "yarn.scheduler.capacity.assignmultiple". If we can make the default of "yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments" to -1, we can assume the feature is off and we can continue allocate any numbers of containers as possible in a heartbeat. Once configured, we can assume that feature is on and restrict as per the limit imposed.
        Show
        sunilg Sunil G added a comment - Thanks Wei Yan for rebasing the patch. Few high level comments Could you please name patch to something like YARN-4161 .002.patch etc by following the naming convention. It will be more easier. I thoughts we can rename the configuration to "yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments". More thoughts are welcome Once 2 is confirmed, rename all variables and getters/setter in align with config name I prefer not to have a config like "yarn.scheduler.capacity.assignmultiple". If we can make the default of "yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments" to -1, we can assume the feature is off and we can continue allocate any numbers of containers as possible in a heartbeat. Once configured, we can assume that feature is on and restrict as per the limit imposed.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Wei Yan for the patch, Sunil G could you help review the patch?

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Wei Yan for the patch, Sunil G could you help review the patch?
        Hide
        ywskycn Wei Yan added a comment -

        Rebase for trunk, by moving the assignMultiple check to scheduler-level, instead of queue-level.
        Tan, Wangda, could u help take a look?

        Show
        ywskycn Wei Yan added a comment - Rebase for trunk, by moving the assignMultiple check to scheduler-level, instead of queue-level. Tan, Wangda , could u help take a look?
        Hide
        mshen Min Shen added a comment -

        The patch does not apply to trunk.
        Mayank Bansal, could you please rebase your patch?

        Show
        mshen Min Shen added a comment - The patch does not apply to trunk. Mayank Bansal , could you please rebase your patch?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 5s YARN-4161 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780627/YARN-4161.patch
        JIRA Issue YARN-4161
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10160/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 5s YARN-4161 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780627/YARN-4161.patch JIRA Issue YARN-4161 Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10160/console This message was automatically generated.
        Hide
        mayank_bansal Mayank Bansal added a comment -

        Attaching patch

        Show
        mayank_bansal Mayank Bansal added a comment - Attaching patch
        Hide
        winghc301 winghc added a comment -

        we just encountered this problem when we submit large job in our production env. using ambari, which only support capacity scheduler. Any patch there?

        Show
        winghc301 winghc added a comment - we just encountered this problem when we submit large job in our production env. using ambari, which only support capacity scheduler. Any patch there?
        Hide
        leftnoteasy Wangda Tan added a comment -

        This will be very useful, thanks Mayank Bansal. Converted this to an independent issue.

        Show
        leftnoteasy Wangda Tan added a comment - This will be very useful, thanks Mayank Bansal . Converted this to an independent issue.

          People

          • Assignee:
            ywskycn Wei Yan
            Reporter:
            mayank_bansal Mayank Bansal
          • Votes:
            2 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development