Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4462

FairScheduler: Disallow preemption from a queue

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: fairscheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When scheduler preemption is enabled, applications could be preempted if they obtain resource over they should take.
      When a mapreduce application is preempted some resource, it just runs slower. However, when the preempted application is a long-run service, such as tomcat running in slider, the service would fail.
      So we should have a flag for application to indicate the scheduler that those application should not be preempted.

      1. YARN-4462.001.patch
        20 kB
        Tao Jie
      2. YARN-4462.002.patch
        20 kB
        Tao Jie
      3. YARN-4462.003.patch
        20 kB
        Tao Jie
      4. YARN-4462.004.patch
        28 kB
        Tao Jie

        Issue Links

          Activity

          Hide
          sunilg Sunil G added a comment -

          Hi Tao Jie
          Trying to understand the problem here. SO you mentioned that few applications were preempted and became slow, correct?. This can happen only if the resource usage went across the desired configured limit for that Queue. And there were demands (some applications waiting to get containers) to an under-serving queue. This would have resulted in preemption.

          In preemption, we will be considering the last submitted applications as first candidate to preempt (within each application, will consider the lowest priority containers) within queue. Few tuning could be done to avoid some sudden trigger of preemption such as "dead-zone" etc. Pls check the preemption related configurations for same.

          Now coming to the suggestion, I fee its kind of a hack over preemption. Preemption policy now chooses the apps/containers based on above explained criteria. And if we have more apps labelled with "do-not-preempt", I feel it may not give a fair preemption across applications in a queue. I am not very sure whether we need to give some extra power to a queue/user to save some v.imp apps. But needs to see how far it will help in reality.

          Show
          sunilg Sunil G added a comment - Hi Tao Jie Trying to understand the problem here. SO you mentioned that few applications were preempted and became slow, correct?. This can happen only if the resource usage went across the desired configured limit for that Queue. And there were demands (some applications waiting to get containers) to an under-serving queue. This would have resulted in preemption. In preemption, we will be considering the last submitted applications as first candidate to preempt (within each application, will consider the lowest priority containers) within queue. Few tuning could be done to avoid some sudden trigger of preemption such as "dead-zone" etc. Pls check the preemption related configurations for same. Now coming to the suggestion, I fee its kind of a hack over preemption. Preemption policy now chooses the apps/containers based on above explained criteria. And if we have more apps labelled with "do-not-preempt", I feel it may not give a fair preemption across applications in a queue. I am not very sure whether we need to give some extra power to a queue/user to save some v.imp apps. But needs to see how far it will help in reality.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Tao Jie
          Why not make use of existing configuration under a queue yarn.scheduler.capacity.<queue-path>.disable_preemption so that we can selectively disable preemption for certain queues (rather configure most of the queues to disable excluding few) and submit the non preemptable apps in those queues where we have explicitly disabled preemption.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Tao Jie Why not make use of existing configuration under a queue yarn.scheduler.capacity.<queue-path>.disable_preemption so that we can selectively disable preemption for certain queues (rather configure most of the queues to disable excluding few) and submit the non preemptable apps in those queues where we have explicitly disabled preemption.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          IIRC its introduced as part of YARN-2056 and available in 2.7.0

          Show
          Naganarasimha Naganarasimha G R added a comment - IIRC its introduced as part of YARN-2056 and available in 2.7.0
          Hide
          Tao Jie Tao Jie added a comment -

          Hi, Naganarasimha G R
          Thank you for reply!
          That is THE feature we need. I agree that queue-level preemption control is better than application-level. However we apply FairScheduler in our system. I would like to bring queue-level-disable-preemption into FairScheduler.

          Show
          Tao Jie Tao Jie added a comment - Hi, Naganarasimha G R Thank you for reply! That is THE feature we need. I agree that queue-level preemption control is better than application-level. However we apply FairScheduler in our system. I would like to bring queue-level-disable-preemption into FairScheduler.
          Hide
          Tao Jie Tao Jie added a comment -

          Hi, Sunil G
          Thank you for reply!
          I agree that "do-not-preempt" application breaks fairness of resource allocation.
          In reality, cost of preempting a long-run service(especially service needs all-or-nothing resource) may be much more than preempting a MR job. When a MR job and a long-run service running in one queue and need to be preempted, I prefer to preempt the MR job before the long-run service even if the MR job is submitted earlier.
          We do have related configuration to adjust logic of preemption(maybe higher app priority). We can also have explicit flag to tell the scheduler that just do not preempt this queue or app, isn't it?

          Show
          Tao Jie Tao Jie added a comment - Hi, Sunil G Thank you for reply! I agree that "do-not-preempt" application breaks fairness of resource allocation. In reality, cost of preempting a long-run service(especially service needs all-or-nothing resource) may be much more than preempting a MR job. When a MR job and a long-run service running in one queue and need to be preempted, I prefer to preempt the MR job before the long-run service even if the MR job is submitted earlier. We do have related configuration to adjust logic of preemption(maybe higher app priority). We can also have explicit flag to tell the scheduler that just do not preempt this queue or app, isn't it?
          Hide
          sunilg Sunil G added a comment -

          Yes Tao Jie. I understood your scenaruo where cost of one long running application is more. YARN-4108 comes up with this improvement. There will be policies which user can configure to selected app. This will surely cover your case.

          Show
          sunilg Sunil G added a comment - Yes Tao Jie. I understood your scenaruo where cost of one long running application is more. YARN-4108 comes up with this improvement. There will be policies which user can configure to selected app. This will surely cover your case.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          hi Tao Jie,

          However we apply FairScheduler in our system. I would like to bring queue-level-disable-preemption into FairScheduler.

          I am not sure about similar support in FS. May be you can update the title accordingly so that right person can comment on it.

          Show
          Naganarasimha Naganarasimha G R added a comment - hi Tao Jie , However we apply FairScheduler in our system. I would like to bring queue-level-disable-preemption into FairScheduler. I am not sure about similar support in FS. May be you can update the title accordingly so that right person can comment on it.
          Hide
          kasha Karthik Kambatla added a comment -

          Today, FairScheduler allows disabling preemption for a queue. It makes sense to add the ability to disable preemption from a queue as well.

          I have a couple of other FairScheduler items I am working on before I am able to get to this. Let me know if anyone is interested in working on this.

          Show
          kasha Karthik Kambatla added a comment - Today, FairScheduler allows disabling preemption for a queue. It makes sense to add the ability to disable preemption from a queue as well. I have a couple of other FairScheduler items I am working on before I am able to get to this. Let me know if anyone is interested in working on this.
          Hide
          Tao Jie Tao Jie added a comment -

          hi Karthik Kambatla
          I have done some attempt of this on FairScheduler based on 2.6.0, and I would like to work on this.

          Show
          Tao Jie Tao Jie added a comment - hi Karthik Kambatla I have done some attempt of this on FairScheduler based on 2.6.0, and I would like to work on this.
          Hide
          kasha Karthik Kambatla added a comment -

          All yours, Tao Jie.

          Show
          kasha Karthik Kambatla added a comment - All yours, Tao Jie .
          Hide
          Tao Jie Tao Jie added a comment -

          In the attached patch, we add disable preemption tag for queues in fair-scheduler.xml. eg:

          <queue name="queueA">
            <minResources>1024mb,0vcores</minResources>
            <disablePreemption/>
          </queue>
          

          With the disablePreemption tag, this queue and its descendants could not be preempted.
          Karthik Kambatla, would you give it a quick review?

          Show
          Tao Jie Tao Jie added a comment - In the attached patch, we add disable preemption tag for queues in fair-scheduler.xml. eg: <queue name= "queueA" > <minResources>1024mb,0vcores</minResources> <disablePreemption/> </queue> With the disablePreemption tag, this queue and its descendants could not be preempted. Karthik Kambatla , would you give it a quick review?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 9m 11s trunk passed
          +1 compile 0m 38s trunk passed with JDK v1.8.0_66
          +1 compile 0m 35s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 39s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 1m 22s trunk passed
          +1 javadoc 0m 25s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 30s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 36s the patch passed
          +1 compile 0m 33s the patch passed with JDK v1.8.0_66
          +1 javac 0m 33s the patch passed
          +1 compile 0m 33s the patch passed with JDK v1.7.0_91
          +1 javac 0m 33s the patch passed
          -1 checkstyle 0m 15s Patch generated 6 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 76, now 79).
          +1 mvnsite 0m 43s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          -1 whitespace 0m 0s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 1m 29s the patch passed
          +1 javadoc 0m 22s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 28s the patch passed with JDK v1.7.0_91
          -1 unit 66m 43s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
          -1 unit 66m 29s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 35s Patch generated 1 ASF License warnings.
          154m 21s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778969/YARN-4462.001.patch
          JIRA Issue YARN-4462
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3b38f33eaa2d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8b1837f
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10086/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10086/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10086/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10086/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10086/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10086/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10086/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10086/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10086/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 11s trunk passed +1 compile 0m 38s trunk passed with JDK v1.8.0_66 +1 compile 0m 35s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 39s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 22s trunk passed +1 javadoc 0m 25s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 30s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 36s the patch passed +1 compile 0m 33s the patch passed with JDK v1.8.0_66 +1 javac 0m 33s the patch passed +1 compile 0m 33s the patch passed with JDK v1.7.0_91 +1 javac 0m 33s the patch passed -1 checkstyle 0m 15s Patch generated 6 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 76, now 79). +1 mvnsite 0m 43s the patch passed +1 mvneclipse 0m 17s the patch passed -1 whitespace 0m 0s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 1m 29s the patch passed +1 javadoc 0m 22s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 28s the patch passed with JDK v1.7.0_91 -1 unit 66m 43s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 66m 29s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 35s Patch generated 1 ASF License warnings. 154m 21s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778969/YARN-4462.001.patch JIRA Issue YARN-4462 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3b38f33eaa2d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8b1837f Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10086/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10086/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10086/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10086/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10086/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10086/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10086/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10086/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10086/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 47s trunk passed
          +1 compile 0m 33s trunk passed with JDK v1.8.0_66
          +1 compile 0m 31s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 37s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 11s trunk passed
          +1 javadoc 0m 22s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 26s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 34s the patch passed
          +1 compile 0m 26s the patch passed with JDK v1.8.0_66
          +1 javac 0m 26s the patch passed
          +1 compile 0m 31s the patch passed with JDK v1.7.0_91
          +1 javac 0m 31s the patch passed
          -1 checkstyle 0m 12s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 76, now 77).
          +1 mvnsite 0m 37s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          -1 whitespace 0m 0s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 1m 18s the patch passed
          +1 javadoc 0m 21s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 27s the patch passed with JDK v1.7.0_91
          -1 unit 59m 7s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
          -1 unit 60m 31s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 23s Patch generated 1 ASF License warnings.
          138m 46s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
            hadoop.yarn.server.resourcemanager.TestClientRMTokens



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779385/YARN-4462.002.patch
          JIRA Issue YARN-4462
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cbd68ef3cdf9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0af492b
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10088/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10088/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10088/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10088/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10088/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10088/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10088/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10088/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10088/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 47s trunk passed +1 compile 0m 33s trunk passed with JDK v1.8.0_66 +1 compile 0m 31s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 37s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 11s trunk passed +1 javadoc 0m 22s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 26s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 34s the patch passed +1 compile 0m 26s the patch passed with JDK v1.8.0_66 +1 javac 0m 26s the patch passed +1 compile 0m 31s the patch passed with JDK v1.7.0_91 +1 javac 0m 31s the patch passed -1 checkstyle 0m 12s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 76, now 77). +1 mvnsite 0m 37s the patch passed +1 mvneclipse 0m 15s the patch passed -1 whitespace 0m 0s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 1m 18s the patch passed +1 javadoc 0m 21s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 27s the patch passed with JDK v1.7.0_91 -1 unit 59m 7s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 60m 31s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 23s Patch generated 1 ASF License warnings. 138m 46s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779385/YARN-4462.002.patch JIRA Issue YARN-4462 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cbd68ef3cdf9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0af492b Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10088/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10088/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10088/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10088/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10088/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10088/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10088/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10088/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10088/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 1s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 14s trunk passed
          +1 compile 0m 30s trunk passed with JDK v1.8.0_66
          +1 compile 0m 33s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 39s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 18s trunk passed
          +1 javadoc 0m 25s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 29s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 36s the patch passed
          +1 compile 0m 30s the patch passed with JDK v1.8.0_66
          +1 javac 0m 30s the patch passed
          +1 compile 0m 33s the patch passed with JDK v1.7.0_91
          +1 javac 0m 33s the patch passed
          -1 checkstyle 0m 14s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 77, now 78).
          +1 mvnsite 0m 39s the patch passed
          +1 mvneclipse 0m 16s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 26s the patch passed
          +1 javadoc 0m 26s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 31s the patch passed with JDK v1.7.0_91
          -1 unit 66m 17s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
          -1 unit 67m 2s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 25s Patch generated 1 ASF License warnings.
          152m 41s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779417/YARN-4462.003.patch
          JIRA Issue YARN-4462
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 09883a8548af 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a308e86
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10089/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10089/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10089/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10089/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10089/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10089/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10089/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10089/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 1s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 14s trunk passed +1 compile 0m 30s trunk passed with JDK v1.8.0_66 +1 compile 0m 33s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 39s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 18s trunk passed +1 javadoc 0m 25s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 29s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 36s the patch passed +1 compile 0m 30s the patch passed with JDK v1.8.0_66 +1 javac 0m 30s the patch passed +1 compile 0m 33s the patch passed with JDK v1.7.0_91 +1 javac 0m 33s the patch passed -1 checkstyle 0m 14s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 77, now 78). +1 mvnsite 0m 39s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 26s the patch passed +1 javadoc 0m 26s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 31s the patch passed with JDK v1.7.0_91 -1 unit 66m 17s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 67m 2s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 25s Patch generated 1 ASF License warnings. 152m 41s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779417/YARN-4462.003.patch JIRA Issue YARN-4462 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 09883a8548af 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a308e86 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10089/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10089/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10089/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10089/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10089/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10089/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10089/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10089/console This message was automatically generated.
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks for working on this, Tao Jie. The approach looks good to me. Comments on the patch itself:

          1. We should make the config less ambiguous. When we say disable preemption, it is not clear we are disabling preempting for/from the queue. I would like to be as explicit as possible. How about a config that is more descriptive - allowPreemptionFrom or preemptionFrom. This could take true or false to conform with other FairScheduler configs. Also, it would be nice to have a global option, so folks could pick a default enable/disable based on the number of queues they have.
          2. Would like the fields and methods updated accordingly. For instance, FairSchedulerQueueInfo and FSQueue could have isPreemptable instead of isPreemptionDisabled.
          3. FSQueue#setPreemptionDisabled is not used anywhere
          4. Can we move the new test added to TestFairSchedulerPreemption. We should move the rest of them also, but may be that is better done on a separate JIRA.
          5. Can we also add a test to verify the behavior where preemption is disabled on all queues?
          Show
          kasha Karthik Kambatla added a comment - Thanks for working on this, Tao Jie . The approach looks good to me. Comments on the patch itself: We should make the config less ambiguous. When we say disable preemption, it is not clear we are disabling preempting for/from the queue. I would like to be as explicit as possible. How about a config that is more descriptive - allowPreemptionFrom or preemptionFrom. This could take true or false to conform with other FairScheduler configs. Also, it would be nice to have a global option, so folks could pick a default enable/disable based on the number of queues they have. Would like the fields and methods updated accordingly. For instance, FairSchedulerQueueInfo and FSQueue could have isPreemptable instead of isPreemptionDisabled. FSQueue#setPreemptionDisabled is not used anywhere Can we move the new test added to TestFairSchedulerPreemption. We should move the rest of them also, but may be that is better done on a separate JIRA. Can we also add a test to verify the behavior where preemption is disabled on all queues?
          Hide
          Tao Jie Tao Jie added a comment -

          Karthik Kambatla , Thank you for your comments. I will make some improvements as you suggested.
          I have more consideration about:

          Also, it would be nice to have a global option, so folks could pick a default enable/disable based on the number of queues they have.

          I am not sure if it is necessary to have a global option for enable/disable preemption from one queue, since I don't think it is a reasonable scenario to set this default value to non-preemptable.
          Now option for allowPreemptionFrom is set to true as default. In most scenario, we have most queues preemptable and few queues non-preemptable, we just add option allowPreemptionFrom with value of false for those non-preemptable.
          Suppose there is another scenario in which we set global option for allowPreemptionFrom to true. We should have most queues non-preemptable and only few(maybe none) queues preemptable. There are very large probability of failure to find candidate containers to be preempted. And it breaks fairness among queues.
          Also it would become confusing when we config queue hierachy. Now we set non-preemptable option either in parent queue or in its leaf queue, the leaf queue is non-preemptable. We try to find preemptable queues from root to leaf recursively and return when we get non-preemptable. It makes sence. However, when we set global default value to non-preemptable and set one parent queue or leaf queue value to preemptable, I would be confusing about whether the leaf queue should be preemptable or not.

          Show
          Tao Jie Tao Jie added a comment - Karthik Kambatla , Thank you for your comments. I will make some improvements as you suggested. I have more consideration about: Also, it would be nice to have a global option, so folks could pick a default enable/disable based on the number of queues they have. I am not sure if it is necessary to have a global option for enable/disable preemption from one queue, since I don't think it is a reasonable scenario to set this default value to non-preemptable. Now option for allowPreemptionFrom is set to true as default. In most scenario, we have most queues preemptable and few queues non-preemptable, we just add option allowPreemptionFrom with value of false for those non-preemptable. Suppose there is another scenario in which we set global option for allowPreemptionFrom to true. We should have most queues non-preemptable and only few(maybe none) queues preemptable. There are very large probability of failure to find candidate containers to be preempted. And it breaks fairness among queues. Also it would become confusing when we config queue hierachy. Now we set non-preemptable option either in parent queue or in its leaf queue, the leaf queue is non-preemptable. We try to find preemptable queues from root to leaf recursively and return when we get non-preemptable. It makes sence. However, when we set global default value to non-preemptable and set one parent queue or leaf queue value to preemptable, I would be confusing about whether the leaf queue should be preemptable or not.
          Hide
          Tao Jie Tao Jie added a comment -

          Updates:
          1, Rename config in fair-scheduler.xml from <disablePreemption/> to <allowPreemptionFrom>false</allowPreemptionFrom>
          2, Rename fields and methods accordingly
          3, Add test case: all queues are not preemptable.

          Show
          Tao Jie Tao Jie added a comment - Updates: 1, Rename config in fair-scheduler.xml from <disablePreemption/> to <allowPreemptionFrom>false</allowPreemptionFrom> 2, Rename fields and methods accordingly 3, Add test case: all queues are not preemptable.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 37s trunk passed
          +1 compile 0m 28s trunk passed with JDK v1.8.0_66
          +1 compile 0m 30s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 37s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 13s trunk passed
          +1 javadoc 0m 22s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 27s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 31s the patch passed
          +1 compile 0m 23s the patch passed with JDK v1.8.0_66
          +1 javac 0m 23s the patch passed
          +1 compile 0m 28s the patch passed with JDK v1.7.0_91
          +1 javac 0m 28s the patch passed
          -1 checkstyle 0m 14s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 76, now 76).
          +1 mvnsite 0m 35s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 1s Patch has no whitespace issues.
          +1 findbugs 1m 16s the patch passed
          +1 javadoc 0m 19s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 24s the patch passed with JDK v1.7.0_91
          -1 unit 65m 14s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
          -1 unit 65m 36s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 19s Patch does not generate ASF License warnings.
          148m 23s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782254/YARN-4462.004.patch
          JIRA Issue YARN-4462
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ce89f92b5ba1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3ddb92b
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10280/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10280/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10280/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10280/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10280/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10280/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10280/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 37s trunk passed +1 compile 0m 28s trunk passed with JDK v1.8.0_66 +1 compile 0m 30s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 37s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 13s trunk passed +1 javadoc 0m 22s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 27s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 31s the patch passed +1 compile 0m 23s the patch passed with JDK v1.8.0_66 +1 javac 0m 23s the patch passed +1 compile 0m 28s the patch passed with JDK v1.7.0_91 +1 javac 0m 28s the patch passed -1 checkstyle 0m 14s Patch generated 3 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 76, now 76). +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 1s Patch has no whitespace issues. +1 findbugs 1m 16s the patch passed +1 javadoc 0m 19s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_91 -1 unit 65m 14s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 65m 36s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 148m 23s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782254/YARN-4462.004.patch JIRA Issue YARN-4462 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ce89f92b5ba1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3ddb92b Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10280/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10280/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10280/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10280/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10280/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10280/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10280/console This message was automatically generated.
          Hide
          kasha Karthik Kambatla added a comment -

          I don't think it is a reasonable scenario to set this default value to non-preemptable.

          Fair point. I agree it makes more sense to not allow setting a default value of non-preemptable.

          The patch itself looks good to me, but for the test being added to TestFairScheduler instead of TestFairSchedulerPreemption. Filed YARN-4648 to track that.

          +1 on this patch. Checking it in.

          Show
          kasha Karthik Kambatla added a comment - I don't think it is a reasonable scenario to set this default value to non-preemptable. Fair point. I agree it makes more sense to not allow setting a default value of non-preemptable. The patch itself looks good to me, but for the test being added to TestFairScheduler instead of TestFairSchedulerPreemption. Filed YARN-4648 to track that. +1 on this patch. Checking it in.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9194 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9194/)
          YARN-4462. FairScheduler: Disallow preemption from a queue. (Tao Jie via (kasha: rev fb238d7e5dcd96466c8938b13ca7f13cedecb08a)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerPage.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerQueueInfo.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9194 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9194/ ) YARN-4462 . FairScheduler: Disallow preemption from a queue. (Tao Jie via (kasha: rev fb238d7e5dcd96466c8938b13ca7f13cedecb08a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerPage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerQueueInfo.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java hadoop-yarn-project/CHANGES.txt
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks for reporting and working on this, Tao Jie. Users have been interested in this for a while.

          Just committed to trunk and branch-2.

          Show
          kasha Karthik Kambatla added a comment - Thanks for reporting and working on this, Tao Jie . Users have been interested in this for a while. Just committed to trunk and branch-2.

            People

            • Assignee:
              Tao Jie Tao Jie
              Reporter:
              Tao Jie Tao Jie
            • Votes:
              1 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development