Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5035

FairScheduler: Adjust maxAssign dynamically when assignMultiple is turned on

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fairscheduler
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Introducing a new configuration "yarn.scheduler.fair.dynamic.max.assign" to dynamically determine the resources to assign per heartbeat when assignmultiple is turned on. When turned on, the scheduler allocates roughly half of the remaining resources overriding any max.assign settings configured. This is turned ON by default.
      Show
      Introducing a new configuration "yarn.scheduler.fair.dynamic.max.assign" to dynamically determine the resources to assign per heartbeat when assignmultiple is turned on. When turned on, the scheduler allocates roughly half of the remaining resources overriding any max.assign settings configured. This is turned ON by default.

      Description

      When assignMultiple is turned on, today the number of containers allocated to node within a single heartbeat is determined by maxAssign. Picking a higher value for this prevents the spreading of the workload, and a smaller number leads to slow ramp up.

      The proposal is to make it dynamic - half of the unallocated resources on the node.

      1. yarn-5035-1.patch
        12 kB
        Karthik Kambatla
      2. yarn-5035-2.patch
        13 kB
        Karthik Kambatla
      3. yarn-5035-3.patch
        13 kB
        Karthik Kambatla

        Issue Links

          Activity

          Hide
          asuresh Arun Suresh added a comment -

          Karthik Kambatla, You think it is possible to generalize this further ?
          I was thinking of integrating YARN-4056 with the fair scheduler. My initial thought process was to replace the assignMultiple with the Bundler.. which would be able to fit an optimal number of containers on a node (optimal, because the containers selected will have complementary resource requirements and thus will have better packing)
          This way, you don't really have to worry about an upper limit..

          Thoughts ?

          Show
          asuresh Arun Suresh added a comment - Karthik Kambatla , You think it is possible to generalize this further ? I was thinking of integrating YARN-4056 with the fair scheduler. My initial thought process was to replace the assignMultiple with the Bundler.. which would be able to fit an optimal number of containers on a node (optimal, because the containers selected will have complementary resource requirements and thus will have better packing) This way, you don't really have to worry about an upper limit.. Thoughts ?
          Hide
          ywskycn Wei Yan added a comment -

          But if continuous-scheduling is turned on, we can always set maxAssign=1 to get a balanced workload distribution, right?

          Show
          ywskycn Wei Yan added a comment - But if continuous-scheduling is turned on, we can always set maxAssign=1 to get a balanced workload distribution, right?
          Hide
          kasha Karthik Kambatla added a comment -

          I was thinking of integrating YARN-4056 with the fair scheduler. My initial thought process was to replace the assignMultiple with the Bundler.

          Sounds like a good long-term approach. Until we gain confidence, I would like for the bundler to be opt in. Would like to get this in sooner to help with more immediate concerns.

          But if continuous-scheduling is turned on, we can always set maxAssign=1 to get a balanced workload distribution, right?

          Yes. We have seen some issues with continuous-scheduling on large clusters - the continuous-scheduling thread hogs the CPU rendering other functionality slow/unresponsive.

          Show
          kasha Karthik Kambatla added a comment - I was thinking of integrating YARN-4056 with the fair scheduler. My initial thought process was to replace the assignMultiple with the Bundler. Sounds like a good long-term approach. Until we gain confidence, I would like for the bundler to be opt in. Would like to get this in sooner to help with more immediate concerns. But if continuous-scheduling is turned on, we can always set maxAssign=1 to get a balanced workload distribution, right? Yes. We have seen some issues with continuous-scheduling on large clusters - the continuous-scheduling thread hogs the CPU rendering other functionality slow/unresponsive.
          Hide
          kasha Karthik Kambatla added a comment -

          Patch that implements dynamic max assign.

          Show
          kasha Karthik Kambatla added a comment - Patch that implements dynamic max assign.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 8m 3s trunk passed
          +1 compile 2m 20s trunk passed
          +1 checkstyle 0m 46s trunk passed
          +1 mvnsite 0m 55s trunk passed
          +1 mvneclipse 0m 23s trunk passed
          0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 1m 7s trunk passed
          +1 javadoc 0m 35s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 45s the patch passed
          +1 compile 2m 28s the patch passed
          +1 javac 2m 28s the patch passed
          -1 checkstyle 0m 36s hadoop-yarn-project/hadoop-yarn: patch generated 1 new + 300 unchanged - 4 fixed = 301 total (was 304)
          +1 mvnsite 0m 42s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 1m 1s the patch passed
          +1 javadoc 0m 27s the patch passed
          -1 unit 30m 16s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 unit 0m 7s hadoop-yarn-site in the patch passed.
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          52m 15s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806214/yarn-5035-1.patch
          JIRA Issue YARN-5035
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a9a0edc11920 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f69f5ab
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11694/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11694/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11694/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11694/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11694/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 8m 3s trunk passed +1 compile 2m 20s trunk passed +1 checkstyle 0m 46s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 23s trunk passed 0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 1m 7s trunk passed +1 javadoc 0m 35s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 45s the patch passed +1 compile 2m 28s the patch passed +1 javac 2m 28s the patch passed -1 checkstyle 0m 36s hadoop-yarn-project/hadoop-yarn: patch generated 1 new + 300 unchanged - 4 fixed = 301 total (was 304) +1 mvnsite 0m 42s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. 0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 1m 1s the patch passed +1 javadoc 0m 27s the patch passed -1 unit 30m 16s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 0m 7s hadoop-yarn-site in the patch passed. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 52m 15s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806214/yarn-5035-1.patch JIRA Issue YARN-5035 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a9a0edc11920 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f69f5ab Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11694/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11694/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11694/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11694/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/11694/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Thanks for the patch, Karthik Kambatla. Looks to me like you can never assign the last container on a node because Resources.multiplyTo() will round down. Also you have an "upto" in the MD code that you should fix.

          Maybe I'm backseat designing a bit, but I wonder if it would be useful to have a configurable minimum to the dynamic assign. I could see it being useful to say I want to schedule half the node or 2 containers, which ever is more.

          Show
          templedf Daniel Templeton added a comment - Thanks for the patch, Karthik Kambatla . Looks to me like you can never assign the last container on a node because Resources.multiplyTo() will round down. Also you have an "upto" in the MD code that you should fix. Maybe I'm backseat designing a bit, but I wonder if it would be useful to have a configurable minimum to the dynamic assign. I could see it being useful to say I want to schedule half the node or 2 containers, which ever is more.
          Hide
          kasha Karthik Kambatla added a comment -

          Looks to me like you can never assign the last container on a node because Resources.multiplyTo() will round down

          assignmultiple and maxassign come into play only after we allocate one container. So, we shouldn't run into this. Also, the intent is for a rough half of the resources so we allocate multiple containers without overloading any one machine. Let me augment the test in any case to make sure we fully allocate the node.

          Also you have an "upto" in the MD code that you should fix.

          Given the allocation is roughly half the containers. I should change this to "about".

          Show
          kasha Karthik Kambatla added a comment - Looks to me like you can never assign the last container on a node because Resources.multiplyTo() will round down assignmultiple and maxassign come into play only after we allocate one container. So, we shouldn't run into this. Also, the intent is for a rough half of the resources so we allocate multiple containers without overloading any one machine. Let me augment the test in any case to make sure we fully allocate the node. Also you have an "upto" in the MD code that you should fix. Given the allocation is roughly half the containers. I should change this to "about".
          Hide
          kasha Karthik Kambatla added a comment -

          Updated patch that improves the test and the docs.

          Show
          kasha Karthik Kambatla added a comment - Updated patch that improves the test and the docs.
          Hide
          kasha Karthik Kambatla added a comment -

          Marking this an incompatible change as this might affect those who have configured max.assign. I think it is okay to do this; in my experience, people rarely fiddle with max.assign and those who do, do so by trial and error to arrive at a behavior similar to the one being introduced here.

          Show
          kasha Karthik Kambatla added a comment - Marking this an incompatible change as this might affect those who have configured max.assign. I think it is okay to do this; in my experience, people rarely fiddle with max.assign and those who do, do so by trial and error to arrive at a behavior similar to the one being introduced here.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 21s Maven dependency ordering for branch
          +1 mvninstall 6m 26s trunk passed
          +1 compile 1m 56s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 0m 46s trunk passed
          +1 mvneclipse 0m 22s trunk passed
          0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 0m 54s trunk passed
          +1 javadoc 0m 29s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 0m 35s the patch passed
          +1 compile 1m 53s the patch passed
          +1 javac 1m 53s the patch passed
          +1 checkstyle 0m 37s hadoop-yarn-project/hadoop-yarn: patch generated 0 new + 300 unchanged - 4 fixed = 300 total (was 304)
          +1 mvnsite 0m 43s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 0m 56s the patch passed
          +1 javadoc 0m 26s the patch passed
          -1 unit 32m 13s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 unit 0m 6s hadoop-yarn-site in the patch passed.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          50m 57s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens
            hadoop.yarn.server.resourcemanager.TestClientRMTokens



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806263/yarn-5035-2.patch
          JIRA Issue YARN-5035
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 742a33efc6c7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 013532a
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11704/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11704/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11704/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11704/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 21s Maven dependency ordering for branch +1 mvninstall 6m 26s trunk passed +1 compile 1m 56s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 0m 46s trunk passed +1 mvneclipse 0m 22s trunk passed 0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 0m 54s trunk passed +1 javadoc 0m 29s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 35s the patch passed +1 compile 1m 53s the patch passed +1 javac 1m 53s the patch passed +1 checkstyle 0m 37s hadoop-yarn-project/hadoop-yarn: patch generated 0 new + 300 unchanged - 4 fixed = 300 total (was 304) +1 mvnsite 0m 43s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. 0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 0m 56s the patch passed +1 javadoc 0m 26s the patch passed -1 unit 32m 13s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 0m 6s hadoop-yarn-site in the patch passed. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 50m 57s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806263/yarn-5035-2.patch JIRA Issue YARN-5035 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 742a33efc6c7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 013532a Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11704/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11704/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11704/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/11704/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          kasha Karthik Kambatla added a comment -

          The unit test failures are unrelated.

          Daniel Templeton - are you comfortable with the latest patch?

          Show
          kasha Karthik Kambatla added a comment - The unit test failures are unrelated. Daniel Templeton - are you comfortable with the latest patch?
          Hide
          asuresh Arun Suresh added a comment - - edited

          Karthik Kambatla, the patch looks pretty straightforward. Minor comments:

          1. In shouldContinueAssignment() :
            if (maxAssignDynamic) {
                  return Resources.lessThan(DOMINANT_RESOURCE_CALCULATOR,
            	          getClusterResource(), assignedResource, maxResourcesToAssign);
            ....
            

            Dont you think you should use a Resources.fitsIn() check rather than the DRF less than ?

          2. I think we should make the 0.5f configurable ?
          Show
          asuresh Arun Suresh added a comment - - edited Karthik Kambatla , the patch looks pretty straightforward. Minor comments: In shouldContinueAssignment() : if (maxAssignDynamic) { return Resources.lessThan(DOMINANT_RESOURCE_CALCULATOR, getClusterResource(), assignedResource, maxResourcesToAssign); .... Dont you think you should use a Resources.fitsIn() check rather than the DRF less than ? I think we should make the 0.5f configurable ?
          Hide
          kasha Karthik Kambatla added a comment -

          Updated patch to use fitsIn. Note that fitsIn checks <= and not <. This leads to allocating one container more than floor(available/2). This is okay - the goal here is to have a maxassign that is (1) proportional to available resources, and (2) considers resources and not containers.

          On making this ratio of 0.5 configurable, I considered it but decided against for two reasons:

          1. I don't anticipate users will change this. In that case, a boolean config is simpler than a number that they have to think about.
          2. 1/2 + 1/4 + ... converges nicely and is somewhat easier to reason about than say 1/3 + 1/9 + .... Also, if we really need a config for this, we could add one in the future.
          Show
          kasha Karthik Kambatla added a comment - Updated patch to use fitsIn. Note that fitsIn checks <= and not <. This leads to allocating one container more than floor(available/2). This is okay - the goal here is to have a maxassign that is (1) proportional to available resources, and (2) considers resources and not containers. On making this ratio of 0.5 configurable, I considered it but decided against for two reasons: I don't anticipate users will change this. In that case, a boolean config is simpler than a number that they have to think about. 1/2 + 1/4 + ... converges nicely and is somewhat easier to reason about than say 1/3 + 1/9 + ... . Also, if we really need a config for this, we could add one in the future.
          Hide
          asuresh Arun Suresh added a comment -

          1/2 + 1/4 + ... converges nicely and is somewhat easier to reason about than say 1/3 + 1/9 + ....

          Actually, given that it only converges at infinity, dont think it makes a difference.... But I do agree, keeping it as 0.5 for the time-being should be fine.

          +1 to the latest patch (pending jenkins)

          Show
          asuresh Arun Suresh added a comment - 1/2 + 1/4 + ... converges nicely and is somewhat easier to reason about than say 1/3 + 1/9 + .... Actually, given that it only converges at infinity, dont think it makes a difference.... But I do agree, keeping it as 0.5 for the time-being should be fine. +1 to the latest patch (pending jenkins)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 6m 16s trunk passed
          +1 compile 2m 2s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 0m 47s trunk passed
          +1 mvneclipse 0m 21s trunk passed
          0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 0m 55s trunk passed
          +1 javadoc 0m 31s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 0m 36s the patch passed
          +1 compile 1m 57s the patch passed
          +1 javac 1m 57s the patch passed
          +1 checkstyle 0m 38s hadoop-yarn-project/hadoop-yarn: patch generated 0 new + 300 unchanged - 4 fixed = 300 total (was 304)
          +1 mvnsite 0m 45s the patch passed
          +1 mvneclipse 0m 18s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 1m 0s the patch passed
          +1 javadoc 0m 27s the patch passed
          -1 unit 38m 14s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 unit 0m 6s hadoop-yarn-site in the patch passed.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          57m 3s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
            hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806294/yarn-5035-3.patch
          JIRA Issue YARN-5035
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f291ab30b8db 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / fed9bf0
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11717/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11717/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11717/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11717/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 16s trunk passed +1 compile 2m 2s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 0m 47s trunk passed +1 mvneclipse 0m 21s trunk passed 0 findbugs 0m 0s Skipped branch modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 0m 55s trunk passed +1 javadoc 0m 31s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 36s the patch passed +1 compile 1m 57s the patch passed +1 javac 1m 57s the patch passed +1 checkstyle 0m 38s hadoop-yarn-project/hadoop-yarn: patch generated 0 new + 300 unchanged - 4 fixed = 300 total (was 304) +1 mvnsite 0m 45s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. 0 findbugs 0m 0s Skipped patch modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 1m 0s the patch passed +1 javadoc 0m 27s the patch passed -1 unit 38m 14s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 0m 6s hadoop-yarn-site in the patch passed. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 57m 3s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806294/yarn-5035-3.patch JIRA Issue YARN-5035 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f291ab30b8db 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fed9bf0 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11717/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11717/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11717/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/11717/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          +1 for me as well.

          Show
          templedf Daniel Templeton added a comment - +1 for me as well.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9868 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9868/)
          YARN-5035. FairScheduler: Adjust maxAssign dynamically when (kasha: rev 04ded558b03ee0fbf68a611cf1f25508b4447e44)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9868 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9868/ ) YARN-5035 . FairScheduler: Adjust maxAssign dynamically when (kasha: rev 04ded558b03ee0fbf68a611cf1f25508b4447e44) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          Hide
          kasha Karthik Kambatla added a comment -

          Just committed this to trunk and branch-2.

          Thanks Arun Suresh and Daniel Templeton for your reviews.

          Show
          kasha Karthik Kambatla added a comment - Just committed this to trunk and branch-2. Thanks Arun Suresh and Daniel Templeton for your reviews.
          Hide
          kasha Karthik Kambatla added a comment -

          Pulled this into branch-2.8 as well.

          Show
          kasha Karthik Kambatla added a comment - Pulled this into branch-2.8 as well.

            People

            • Assignee:
              kasha Karthik Kambatla
              Reporter:
              kasha Karthik Kambatla
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development