Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, CLI/REST API is provided in Resource Manager to allow users to specify labels for nodes. For labels which may change over time, users will have to start a cron job to update the labels. This has the following limitations:

      • The cron job needs to be run in the YARN admin user.
      • This makes it a little complicate to maintain as users will have to make sure this service/daemon is alive.
        Adding a Node Labels Provider in Resource Manager will provide user more flexibility.
      1. YARN-3964.016.patch
        52 kB
        Dian Fu
      2. YARN-3964.015.patch
        52 kB
        Dian Fu
      3. YARN-3964.014.patch
        51 kB
        Dian Fu
      4. YARN-3964.013.patch
        51 kB
        Dian Fu
      5. YARN-3964.012.patch
        50 kB
        Dian Fu
      6. YARN-3964.011.patch
        49 kB
        Dian Fu
      7. YARN-3964.010.patch
        49 kB
        Dian Fu
      8. YARN-3964.009.patch
        49 kB
        Dian Fu
      9. YARN-3964.008.patch
        49 kB
        Dian Fu
      10. YARN-3964.007.patch
        44 kB
        Dian Fu
      11. YARN-3964.007.patch
        44 kB
        Dian Fu
      12. YARN-3964.006.patch
        33 kB
        Dian Fu
      13. YARN-3964.005.patch
        47 kB
        Dian Fu
      14. YARN-3964.004.patch
        45 kB
        Dian Fu
      15. YARN-3964.003.patch
        43 kB
        Dian Fu
      16. YARN-3964.002.patch
        42 kB
        Dian Fu
      17. YARN-3964.1.patch
        45 kB
        Dian Fu
      18. YARN-3964 design doc.pdf
        343 kB
        Dian Fu

        Issue Links

          Activity

          Hide
          dian.fu Dian Fu added a comment -

          A simple design doc attached. Any feedback will be welcome.

          Show
          dian.fu Dian Fu added a comment - A simple design doc attached. Any feedback will be welcome.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Hi Dian Fu,
          I just went through the design doc.

          According to the design doc, configuration of centralized provider also needs to consider 1) frequency of invoking fetch script. 2) permission of script. 3) a customized provider 4) configuration RM classpath to make use of it. It's not simple comparing to use cron + script + REST-API/CLI.

          To some points in the design doc:

          Why need provider instead of REST API – REST API requires admin privilege.

          Configuration/executing provider script also needs YARN's admin permission.

          User can provide their own node labels provider plug-in which can fetch labels from a database, from a remote server, etc.

          Using a cron job invoke customized script can also achieve this.

          And this approach also needs to add a plugin inside RM, which is could be unsafe and sometimes cause RM dies if the provider has some issues.

          IMHO, this will add unnecessary complexity to RM, risks to RM, and cannot make easier configuration. Please correct me if I missed anything.

          Thanks,

          Show
          leftnoteasy Wangda Tan added a comment - Hi Dian Fu , I just went through the design doc. According to the design doc, configuration of centralized provider also needs to consider 1) frequency of invoking fetch script. 2) permission of script. 3) a customized provider 4) configuration RM classpath to make use of it. It's not simple comparing to use cron + script + REST-API/CLI. To some points in the design doc: Why need provider instead of REST API – REST API requires admin privilege. Configuration/executing provider script also needs YARN's admin permission. User can provide their own node labels provider plug-in which can fetch labels from a database, from a remote server, etc. Using a cron job invoke customized script can also achieve this. And this approach also needs to add a plugin inside RM, which is could be unsafe and sometimes cause RM dies if the provider has some issues. IMHO, this will add unnecessary complexity to RM, risks to RM, and cannot make easier configuration. Please correct me if I missed anything. Thanks,
          Hide
          dian.fu Dian Fu added a comment -

          Hi Wangda Tan,
          Thanks a lot for your review and feedback.

          According to the design doc, configuration of centralized provider also needs to consider 1) frequency of invoking fetch script.

          User just need to configure it via configuration and a default value can be provided if user doesn't configure.

          2) permission of script.

          Sorry, I'm not sure what's your point? You mean the execution permission? I don't think that would be a problem. Actually the provider don't have to be script based.

          3) a customized provider

          The implementation of a customized provider will be very simple. User just need to implement interface NodeLabelsFetcher.

          4) configuration RM classpath to make use of it.

          Yes, that's true.

          Configuration/executing provider script also needs YARN's admin permission.

          The provider will be part of Resource Manager, we don't need to start up a separate process to execute it.

          To clearly express the thoughts, I have uploaded an initial patch.
          Thanks again for your comments.

          Show
          dian.fu Dian Fu added a comment - Hi Wangda Tan , Thanks a lot for your review and feedback. According to the design doc, configuration of centralized provider also needs to consider 1) frequency of invoking fetch script. User just need to configure it via configuration and a default value can be provided if user doesn't configure. 2) permission of script. Sorry, I'm not sure what's your point? You mean the execution permission? I don't think that would be a problem. Actually the provider don't have to be script based. 3) a customized provider The implementation of a customized provider will be very simple. User just need to implement interface NodeLabelsFetcher . 4) configuration RM classpath to make use of it. Yes, that's true. Configuration/executing provider script also needs YARN's admin permission. The provider will be part of Resource Manager, we don't need to start up a separate process to execute it. To clearly express the thoughts, I have uploaded an initial patch. Thanks again for your comments.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 20m 26s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 8 new or modified test files.
          +1 javac 7m 46s There were no new javac warning messages.
          +1 javadoc 9m 38s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 20s The applied patch generated 2 new checkstyle issues (total was 211, now 212).
          -1 checkstyle 2m 45s The applied patch generated 9 new checkstyle issues (total was 0, now 9).
          -1 whitespace 0m 3s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 24s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          -1 findbugs 6m 9s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 6m 57s Tests passed in hadoop-yarn-client.
          +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 6m 15s Tests passed in hadoop-yarn-server-nodemanager.
          +1 yarn tests 51m 55s Tests passed in hadoop-yarn-server-resourcemanager.
              115m 25s  



          Reason Tests
          FindBugs module:hadoop-yarn-server-resourcemanager



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12746997/YARN-3964.1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 206d493
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/testrun_hadoop-yarn-client.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8647/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8647/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 26s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 8 new or modified test files. +1 javac 7m 46s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 20s The applied patch generated 2 new checkstyle issues (total was 211, now 212). -1 checkstyle 2m 45s The applied patch generated 9 new checkstyle issues (total was 0, now 9). -1 whitespace 0m 3s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 24s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 6m 9s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 6m 57s Tests passed in hadoop-yarn-client. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common. +1 yarn tests 6m 15s Tests passed in hadoop-yarn-server-nodemanager. +1 yarn tests 51m 55s Tests passed in hadoop-yarn-server-resourcemanager.     115m 25s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12746997/YARN-3964.1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 206d493 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/testrun_hadoop-yarn-client.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8647/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8647/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8647/console This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Hi Dian Fu,
          Thanks for comments and the patch. I took a quick look at the patch, some problems I can see now:

          • It involves some unnecessary interface/parameter to NodeLabelsProvider, this also leads to unnecessary changes to NM
          • Fetcher implementation is polling updated labels for ALL NMs in the cluster, if a cluster has several thousands of NMs, this can be inefficient.

          My biggest concern is still about if this change is must-to-have:
          Since we already have a set of APIs to do this, I can't see a big add-on value of doing this inside RM.
          For example, we provided submitApplication REST API so that downstream applications can use, YARN RM won't do things beyond parse the submission request and launch the application. Complex requirements such as cron job will be handled outside of YARN, such as Oozie/Slider, YARN doesn't support "ApplicationSubmissionProvider" inside of RM even if sometimes it sounds more flexible.

          IMHO, pluggable functionality needs only be added to RM if it is necessary, otherwise it becomes over design. Resource scheduler is one example, it coupled with lots of components so it's very hard to be pulled to outside of RM. RMNodeLabelsProvider only couples with RMNodeLabelsManager in the patch, adding labels to RMNodeLabelsManager inside RM is as same as doing it via REST API / CLI.

          Please let me know if you have any other concerns/comments.

          +Devaraj K.

          Show
          leftnoteasy Wangda Tan added a comment - Hi Dian Fu , Thanks for comments and the patch. I took a quick look at the patch, some problems I can see now: It involves some unnecessary interface/parameter to NodeLabelsProvider, this also leads to unnecessary changes to NM Fetcher implementation is polling updated labels for ALL NMs in the cluster, if a cluster has several thousands of NMs, this can be inefficient. My biggest concern is still about if this change is must-to-have: Since we already have a set of APIs to do this, I can't see a big add-on value of doing this inside RM. For example, we provided submitApplication REST API so that downstream applications can use, YARN RM won't do things beyond parse the submission request and launch the application. Complex requirements such as cron job will be handled outside of YARN, such as Oozie/Slider, YARN doesn't support "ApplicationSubmissionProvider" inside of RM even if sometimes it sounds more flexible. IMHO, pluggable functionality needs only be added to RM if it is necessary, otherwise it becomes over design. Resource scheduler is one example, it coupled with lots of components so it's very hard to be pulled to outside of RM. RMNodeLabelsProvider only couples with RMNodeLabelsManager in the patch, adding labels to RMNodeLabelsManager inside RM is as same as doing it via REST API / CLI. Please let me know if you have any other concerns/comments. + Devaraj K .
          Hide
          dian.fu Dian Fu added a comment -

          Hi Wangda Tan,
          Really thanks for your comments.

          I took a quick look at the patch, some problems I can see now:

          • It involves some unnecessary interface/parameter to NodeLabelsProvider, this also leads to unnecessary changes to NM

          This patch tries to move NodeLabelsProvider from hadoop-yarn-server-nodemanager to hadoop-yarn-server-common to make it usable by both NM and RM. But it's fine to keep it untouched.

          • Fetcher implementation is polling updated labels for ALL NMs in the cluster, if a cluster has several thousands of NMs, this can be inefficient.

          Good advice. We can solve this issue by updating the labels for ALL NMs in one request, not one by one. Will update the patch accordingly.

          My biggest concern is still about if this change is must-to-have:
          Since we already have a set of APIs to do this, I can't see a big add-on value of doing this inside RM.

          I understand your concern and agree that with a cron job, some scripts and REST API, we do be able to achieve the functionality. While this improvement will have its value. It can largely decrease the amount of additional work to do and other difficulties for integrating a label source. Also it increases the usability of the label feature from management perspective. We know, a lot of times, how a technology will be adapted by users depends largely on how easily the technology can be used or integrated. Although this is not a "must-to-have", this improvement take the label feature a step further from the integration point of view.

          For large clusters, it's usually not practical to manage the label of all nodes manually. Enterprises usually use some kind of label or label policy storage. This improvement can help address this requirement perfectly with the minimized additional development work. Also, this feature can be used as a different use case than synchronizing the labels through REST API because the configuration of a label provider mechanism at the YARN side means the management operations (usually done by administrator) instead of REST API operation of a client, adding the trustfulness of label source.

          Further more, we will target to make this change to be simple, light weight and strait-forward . It will not bring any additional complexity to YARN architecture but provide a flexible functionality for label integration.

          Thank you again for your feedback.

          Show
          dian.fu Dian Fu added a comment - Hi Wangda Tan , Really thanks for your comments. I took a quick look at the patch, some problems I can see now: It involves some unnecessary interface/parameter to NodeLabelsProvider, this also leads to unnecessary changes to NM This patch tries to move NodeLabelsProvider from hadoop-yarn-server-nodemanager to hadoop-yarn-server-common to make it usable by both NM and RM. But it's fine to keep it untouched. Fetcher implementation is polling updated labels for ALL NMs in the cluster, if a cluster has several thousands of NMs, this can be inefficient. Good advice. We can solve this issue by updating the labels for ALL NMs in one request, not one by one. Will update the patch accordingly. My biggest concern is still about if this change is must-to-have: Since we already have a set of APIs to do this, I can't see a big add-on value of doing this inside RM. I understand your concern and agree that with a cron job, some scripts and REST API, we do be able to achieve the functionality. While this improvement will have its value. It can largely decrease the amount of additional work to do and other difficulties for integrating a label source. Also it increases the usability of the label feature from management perspective. We know, a lot of times, how a technology will be adapted by users depends largely on how easily the technology can be used or integrated. Although this is not a "must-to-have", this improvement take the label feature a step further from the integration point of view. For large clusters, it's usually not practical to manage the label of all nodes manually. Enterprises usually use some kind of label or label policy storage. This improvement can help address this requirement perfectly with the minimized additional development work. Also, this feature can be used as a different use case than synchronizing the labels through REST API because the configuration of a label provider mechanism at the YARN side means the management operations (usually done by administrator) instead of REST API operation of a client, adding the trustfulness of label source. Further more, we will target to make this change to be simple, light weight and strait-forward . It will not bring any additional complexity to YARN architecture but provide a flexible functionality for label integration. Thank you again for your feedback.
          Hide
          dian.fu Dian Fu added a comment -

          Updated the patch with the following updates:

          • remove the interface modification to NodeLabelsProvider
          • improve Fetcher implementation to update node labels in batch
          Show
          dian.fu Dian Fu added a comment - Updated the patch with the following updates: remove the interface modification to NodeLabelsProvider improve Fetcher implementation to update node labels in batch
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 20m 44s Pre-patch trunk has 7 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 8 new or modified test files.
          +1 javac 7m 56s There were no new javac warning messages.
          +1 javadoc 9m 44s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 21s The applied patch generated 2 new checkstyle issues (total was 211, now 212).
          -1 checkstyle 2m 45s The applied patch generated 9 new checkstyle issues (total was 0, now 9).
          -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 24s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 6m 15s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          -1 yarn tests 0m 21s Tests failed in hadoop-yarn-api.
          +1 yarn tests 7m 0s Tests passed in hadoop-yarn-client.
          +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 6m 38s Tests passed in hadoop-yarn-server-nodemanager.
          -1 yarn tests 56m 43s Tests failed in hadoop-yarn-server-resourcemanager.
              121m 20s  



          Reason Tests
          FindBugs module:hadoop-yarn-server-resourcemanager
          Failed unit tests hadoop.yarn.conf.TestYarnConfigurationFields
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
            hadoop.yarn.server.resourcemanager.TestRMAdminService



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12749814/YARN-3964.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / fa1d84a
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/testrun_hadoop-yarn-client.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8822/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8822/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 44s Pre-patch trunk has 7 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 8 new or modified test files. +1 javac 7m 56s There were no new javac warning messages. +1 javadoc 9m 44s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 21s The applied patch generated 2 new checkstyle issues (total was 211, now 212). -1 checkstyle 2m 45s The applied patch generated 9 new checkstyle issues (total was 0, now 9). -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 24s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 6m 15s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. -1 yarn tests 0m 21s Tests failed in hadoop-yarn-api. +1 yarn tests 7m 0s Tests passed in hadoop-yarn-client. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common. +1 yarn tests 6m 38s Tests passed in hadoop-yarn-server-nodemanager. -1 yarn tests 56m 43s Tests failed in hadoop-yarn-server-resourcemanager.     121m 20s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Failed unit tests hadoop.yarn.conf.TestYarnConfigurationFields   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation   hadoop.yarn.server.resourcemanager.TestRMAdminService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749814/YARN-3964.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / fa1d84a Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/testrun_hadoop-yarn-client.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8822/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8822/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8822/console This message was automatically generated.
          Hide
          dian.fu Dian Fu added a comment -

          Updated the patch to fix the warnings and test failures.

          Show
          dian.fu Dian Fu added a comment - Updated the patch to fix the warnings and test failures.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 20m 19s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 8 new or modified test files.
          +1 javac 8m 17s There were no new javac warning messages.
          +1 javadoc 10m 10s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 23s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          -1 checkstyle 3m 6s The applied patch generated 2 new checkstyle issues (total was 0, now 2).
          +1 whitespace 0m 3s The patch has no lines that end in whitespace.
          +1 install 1m 27s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          -1 findbugs 8m 17s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          +1 yarn tests 7m 2s Tests passed in hadoop-yarn-client.
          -1 yarn tests 2m 0s Tests failed in hadoop-yarn-common.
          +1 yarn tests 0m 26s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 6m 32s Tests passed in hadoop-yarn-server-nodemanager.
          -1 yarn tests 53m 53s Tests failed in hadoop-yarn-server-resourcemanager.
              123m 23s  



          Reason Tests
          FindBugs module:hadoop-yarn-server-resourcemanager
          Failed unit tests hadoop.yarn.util.TestRackResolver
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12750043/YARN-3964.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 1c12adb
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/testrun_hadoop-yarn-client.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8831/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8831/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 19s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 8 new or modified test files. +1 javac 8m 17s There were no new javac warning messages. +1 javadoc 10m 10s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 23s The applied patch generated 1 new checkstyle issues (total was 211, now 211). -1 checkstyle 3m 6s The applied patch generated 2 new checkstyle issues (total was 0, now 2). +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 27s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. -1 findbugs 8m 17s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 7m 2s Tests passed in hadoop-yarn-client. -1 yarn tests 2m 0s Tests failed in hadoop-yarn-common. +1 yarn tests 0m 26s Tests passed in hadoop-yarn-server-common. +1 yarn tests 6m 32s Tests passed in hadoop-yarn-server-nodemanager. -1 yarn tests 53m 53s Tests failed in hadoop-yarn-server-resourcemanager.     123m 23s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Failed unit tests hadoop.yarn.util.TestRackResolver   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12750043/YARN-3964.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1c12adb Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/testrun_hadoop-yarn-client.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8831/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8831/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8831/console This message was automatically generated.
          Hide
          dian.fu Dian Fu added a comment -

          Updated the patch to address the checkstyle warnings and also add more comments. The test failures aren't relevent to this patch.

          Show
          dian.fu Dian Fu added a comment - Updated the patch to address the checkstyle warnings and also add more comments. The test failures aren't relevent to this patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 21m 31s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 8 new or modified test files.
          +1 javac 10m 14s There were no new javac warning messages.
          +1 javadoc 12m 30s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 52s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 3s The patch has no lines that end in whitespace.
          +1 install 1m 44s mvn install still works.
          +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
          -1 findbugs 12m 24s The patch appears to introduce 8 new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api.
          -1 yarn tests 7m 30s Tests failed in hadoop-yarn-client.
          -1 yarn tests 2m 18s Tests failed in hadoop-yarn-common.
          +1 yarn tests 0m 30s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 6m 57s Tests passed in hadoop-yarn-server-nodemanager.
          -1 yarn tests 55m 20s Tests failed in hadoop-yarn-server-resourcemanager.
              136m 32s  



          Reason Tests
          FindBugs module:hadoop-yarn-server-common
          FindBugs module:hadoop-yarn-server-resourcemanager
          Failed unit tests hadoop.yarn.client.api.impl.TestYarnClient
            hadoop.yarn.util.TestRackResolver
          Timed out tests org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12750244/YARN-3964.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 40f8151
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
          Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-client.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8840/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8840/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 21m 31s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 8 new or modified test files. +1 javac 10m 14s There were no new javac warning messages. +1 javadoc 12m 30s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 52s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 44s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. -1 findbugs 12m 24s The patch appears to introduce 8 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api. -1 yarn tests 7m 30s Tests failed in hadoop-yarn-client. -1 yarn tests 2m 18s Tests failed in hadoop-yarn-common. +1 yarn tests 0m 30s Tests passed in hadoop-yarn-server-common. +1 yarn tests 6m 57s Tests passed in hadoop-yarn-server-nodemanager. -1 yarn tests 55m 20s Tests failed in hadoop-yarn-server-resourcemanager.     136m 32s   Reason Tests FindBugs module:hadoop-yarn-server-common FindBugs module:hadoop-yarn-server-resourcemanager Failed unit tests hadoop.yarn.client.api.impl.TestYarnClient   hadoop.yarn.util.TestRackResolver Timed out tests org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12750244/YARN-3964.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 40f8151 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-client.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8840/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8840/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8840/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12750244/YARN-3964.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 65ccf2b
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8965/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12750244/YARN-3964.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 65ccf2b Console output https://builds.apache.org/job/PreCommit-YARN-Build/8965/console This message was automatically generated.
          Hide
          dian.fu Dian Fu added a comment -

          rebase the patch.

          Show
          dian.fu Dian Fu added a comment - rebase the patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 23m 46s Pre-patch trunk has 7 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 9 new or modified test files.
          +1 javac 9m 14s There were no new javac warning messages.
          +1 javadoc 11m 2s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 3m 21s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          -1 checkstyle 4m 21s The applied patch generated 1 new checkstyle issues (total was 0, now 1).
          +1 whitespace 0m 3s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          -1 findbugs 8m 8s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 27s Tests passed in hadoop-yarn-api.
          -1 yarn tests 7m 6s Tests failed in hadoop-yarn-client.
          +1 yarn tests 2m 12s Tests passed in hadoop-yarn-common.
          +1 yarn tests 0m 27s Tests passed in hadoop-yarn-server-common.
          -1 yarn tests 8m 1s Tests failed in hadoop-yarn-server-nodemanager.
          -1 yarn tests 55m 12s Tests failed in hadoop-yarn-server-resourcemanager.
              133m 0s  



          Reason Tests
          FindBugs module:hadoop-yarn-server-resourcemanager
          Failed unit tests hadoop.yarn.client.api.impl.TestYarnClient
            hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels
          Timed out tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12753693/YARN-3964.005.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 9b68577
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/testrun_hadoop-yarn-client.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9020/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9020/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 23m 46s Pre-patch trunk has 7 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 9 new or modified test files. +1 javac 9m 14s There were no new javac warning messages. +1 javadoc 11m 2s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 3m 21s The applied patch generated 1 new checkstyle issues (total was 211, now 211). -1 checkstyle 4m 21s The applied patch generated 1 new checkstyle issues (total was 0, now 1). +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 8m 8s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 27s Tests passed in hadoop-yarn-api. -1 yarn tests 7m 6s Tests failed in hadoop-yarn-client. +1 yarn tests 2m 12s Tests passed in hadoop-yarn-common. +1 yarn tests 0m 27s Tests passed in hadoop-yarn-server-common. -1 yarn tests 8m 1s Tests failed in hadoop-yarn-server-nodemanager. -1 yarn tests 55m 12s Tests failed in hadoop-yarn-server-resourcemanager.     133m 0s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Failed unit tests hadoop.yarn.client.api.impl.TestYarnClient   hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels Timed out tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12753693/YARN-3964.005.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 9b68577 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/testrun_hadoop-yarn-client.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9020/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9020/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9020/console This message was automatically generated.
          Hide
          devaraj.k Devaraj K added a comment -

          Thanks Dian Fu for the patch.

          Patch has gone stale, Can you please update the patch? And also please take care of the above jenkins warnings in the updated patch.

          Show
          devaraj.k Devaraj K added a comment - Thanks Dian Fu for the patch. Patch has gone stale, Can you please update the patch? And also please take care of the above jenkins warnings in the updated patch.
          Hide
          dian.fu Dian Fu added a comment -

          Thanks Devaraj K for taking look at the patch. Attaching rebased patch.

          Show
          dian.fu Dian Fu added a comment - Thanks Devaraj K for taking look at the patch. Attaching rebased patch.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Dian Fu, Devaraj K, I plan to take a look at this patch tomorrow, could you wait my review before commit it?

          Thanks,

          Show
          leftnoteasy Wangda Tan added a comment - Dian Fu , Devaraj K , I plan to take a look at this patch tomorrow, could you wait my review before commit it? Thanks,
          Hide
          devaraj.k Devaraj K added a comment -

          Wangda Tan, Sure, Thanks for your interest.

          Show
          devaraj.k Devaraj K added a comment - Wangda Tan , Sure, Thanks for your interest.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 20m 15s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 7 new or modified test files.
          +1 javac 7m 48s There were no new javac warning messages.
          +1 javadoc 9m 56s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 10s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 3s The patch has no lines that end in whitespace.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 5m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          +1 yarn tests 6m 57s Tests passed in hadoop-yarn-client.
          +1 yarn tests 1m 59s Tests passed in hadoop-yarn-common.
          +1 yarn tests 58m 28s Tests passed in hadoop-yarn-server-resourcemanager.
              116m 46s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12761374/YARN-3964.006.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c9cb6a5
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9227/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9227/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/9227/artifact/patchprocess/testrun_hadoop-yarn-client.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9227/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9227/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9227/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9227/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 15s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 7 new or modified test files. +1 javac 7m 48s There were no new javac warning messages. +1 javadoc 9m 56s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 10s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 5m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 6m 57s Tests passed in hadoop-yarn-client. +1 yarn tests 1m 59s Tests passed in hadoop-yarn-common. +1 yarn tests 58m 28s Tests passed in hadoop-yarn-server-resourcemanager.     116m 46s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761374/YARN-3964.006.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c9cb6a5 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9227/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9227/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/9227/artifact/patchprocess/testrun_hadoop-yarn-client.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9227/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9227/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9227/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9227/console This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Dian Fu.

          Some comments:
          1) I suggest to make this to be an explicit node label configuration type: yarn.node-labels.configuration-type. Currently it has "centralized/distributed", I think you may add a "delegated-centralized" (or other better name).
          Other configurations in your patch look fine to me.

          2) Some comments of organization of Updater/Provider:

          • Updater is a subclass of AbstractService, but no need to be abstract. I'm not sure what's the purpose of adding an AbstractNodeLabelsUpdater. Provider will be initialized by Updater, and Updater will call Provider's method periodically and notify RMNodeLabelsManager.
          • Provider is an interface, minor comments to your patch:
            • Why need a Configuration in getNodeLabels method?
            • Returns Set<NodeLabel> instead of Set<String>

          3) There're some methods / comments include "Fetcher", could you replace them to "Provider"?

          4) Instead of adding a new checkAndThrowIfNodeLabelsFetcherConfigured, I suggest to reuse the checkAndThrowIfDistributedNodeLabelConfEnabled: You can rename it to something like checkAndThrowIfNodeLabelCannotBeUpdatedManually, which will check yarn.node-labels.configuration-type, we only allow manually update labels when type=centralized configured.

          5) You can add a method to get RMNodeLabelsUpdater from RMContext, and remove it from ResourceTrackerService constructor.

          6) Add a test of RMNodeLabelsUpdater? It seems can only update labels-on-node once for every node.

          7) I think we need to make sure labels will be updated synchronizedly when a node is registering, this can avoid a node's labels initialized after it registered a while.

          8) If you agree with #7, I think wait/notify implementation of Updater could be removed, you can use synchronized lock instead. Code using wait/notify has bad readability and will likely introduce bugs.

          Thanks,
          Wangda

          Show
          leftnoteasy Wangda Tan added a comment - Dian Fu . Some comments: 1) I suggest to make this to be an explicit node label configuration type: yarn.node-labels.configuration-type . Currently it has "centralized/distributed", I think you may add a "delegated-centralized" (or other better name). Other configurations in your patch look fine to me. 2) Some comments of organization of Updater/Provider: Updater is a subclass of AbstractService, but no need to be abstract. I'm not sure what's the purpose of adding an AbstractNodeLabelsUpdater. Provider will be initialized by Updater, and Updater will call Provider's method periodically and notify RMNodeLabelsManager. Provider is an interface, minor comments to your patch: Why need a Configuration in getNodeLabels method? Returns Set<NodeLabel> instead of Set<String> 3) There're some methods / comments include "Fetcher", could you replace them to "Provider"? 4) Instead of adding a new checkAndThrowIfNodeLabelsFetcherConfigured, I suggest to reuse the checkAndThrowIfDistributedNodeLabelConfEnabled: You can rename it to something like checkAndThrowIfNodeLabelCannotBeUpdatedManually, which will check yarn.node-labels.configuration-type , we only allow manually update labels when type=centralized configured. 5) You can add a method to get RMNodeLabelsUpdater from RMContext, and remove it from ResourceTrackerService constructor. 6) Add a test of RMNodeLabelsUpdater? It seems can only update labels-on-node once for every node. 7) I think we need to make sure labels will be updated synchronizedly when a node is registering, this can avoid a node's labels initialized after it registered a while. 8) If you agree with #7, I think wait/notify implementation of Updater could be removed, you can use synchronized lock instead. Code using wait/notify has bad readability and will likely introduce bugs. Thanks, Wangda
          Hide
          dian.fu Dian Fu added a comment -

          Thanks Wangda Tan for your detailed review. Make sense to me and will update the patch to incorporate your comments ASAP.

          Show
          dian.fu Dian Fu added a comment - Thanks Wangda Tan for your detailed review. Make sense to me and will update the patch to incorporate your comments ASAP.
          Hide
          dian.fu Dian Fu added a comment -

          Hi Wangda Tan,
          Thanks a lot for the review. Updated the patch according to your review comments.

          Show
          dian.fu Dian Fu added a comment - Hi Wangda Tan , Thanks a lot for the review. Updated the patch according to your review comments.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 48s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 8m 4s There were no new javac warning messages.
          +1 javadoc 9m 54s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 58s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 3s The patch has no lines that end in whitespace.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          +1 findbugs 4m 39s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 2s Tests passed in hadoop-yarn-common.
          -1 yarn tests 40m 6s Tests failed in hadoop-yarn-server-resourcemanager.
              90m 14s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing
            hadoop.yarn.server.resourcemanager.resourcetracker.TestRMNMRPCResponseId
            hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerEventLog
            hadoop.yarn.server.resourcemanager.TestApplicationCleanup
            hadoop.yarn.server.resourcemanager.TestRMHA
            hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs
            hadoop.yarn.server.resourcemanager.TestResourceManager
            hadoop.yarn.server.resourcemanager.TestApplicationMasterService
            hadoop.yarn.server.resourcemanager.nodelabels.TestRMNodeLabelsManager
            hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels
            hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
            hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
            hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestWorkPreservingRMRestartForNodeLabel
            hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicy
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.resourcetracker.TestNMReconnect
            hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
            hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
            hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor
            hadoop.yarn.server.resourcemanager.TestRMProxyUsersConf
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior
            hadoop.yarn.server.resourcemanager.TestApplicationACLs
            hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler
            hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
            hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
            hadoop.yarn.server.resourcemanager.scheduler.TestQueueMetrics
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
            hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerFairShare
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestAppRunnability
            hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSLeafQueue
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestSchedulingUpdate
            hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
            hadoop.yarn.server.resourcemanager.nodelabels.TestRMNodeLabelsUpdater
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12761855/YARN-3964.007.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 7c5c099
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9238/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9238/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9238/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9238/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9238/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9238/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 48s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 8m 4s There were no new javac warning messages. +1 javadoc 9m 54s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 58s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 4m 39s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 2s Tests passed in hadoop-yarn-common. -1 yarn tests 40m 6s Tests failed in hadoop-yarn-server-resourcemanager.     90m 14s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing   hadoop.yarn.server.resourcemanager.resourcetracker.TestRMNMRPCResponseId   hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerEventLog   hadoop.yarn.server.resourcemanager.TestApplicationCleanup   hadoop.yarn.server.resourcemanager.TestRMHA   hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs   hadoop.yarn.server.resourcemanager.TestResourceManager   hadoop.yarn.server.resourcemanager.TestApplicationMasterService   hadoop.yarn.server.resourcemanager.nodelabels.TestRMNodeLabelsManager   hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels   hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA   hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue   hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestWorkPreservingRMRestartForNodeLabel   hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicy   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.resourcetracker.TestNMReconnect   hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation   hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs   hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor   hadoop.yarn.server.resourcemanager.TestRMProxyUsersConf   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior   hadoop.yarn.server.resourcemanager.TestApplicationACLs   hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage   hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler   hadoop.yarn.server.resourcemanager.scheduler.TestQueueMetrics   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate   hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerFairShare   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption   hadoop.yarn.server.resourcemanager.scheduler.fair.TestAppRunnability   hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSLeafQueue   hadoop.yarn.server.resourcemanager.scheduler.fair.TestSchedulingUpdate   hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter   hadoop.yarn.server.resourcemanager.nodelabels.TestRMNodeLabelsUpdater   hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761855/YARN-3964.007.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7c5c099 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9238/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9238/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9238/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9238/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9238/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9238/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Dian Fu,
          Thanks for working on this jira, i will take a look at this jira today before it one small query ? Purpose of YARN-3743 and this seems to be same, can you close the older one as its same as this one.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Dian Fu , Thanks for working on this jira, i will take a look at this jira today before it one small query ? Purpose of YARN-3743 and this seems to be same, can you close the older one as its same as this one.
          Hide
          dian.fu Dian Fu added a comment -

          Hi Naganarasimha G R,
          Thanks in advance for your review. Have marked YARN-3743 as duplicate with this one.

          Show
          dian.fu Dian Fu added a comment - Hi Naganarasimha G R , Thanks in advance for your review. Have marked YARN-3743 as duplicate with this one.
          Hide
          dian.fu Dian Fu added a comment -

          Attaching the 007 patch to trigger the jenkins again as I have checked the test failures in my local environment and all of them have passed.

          Show
          dian.fu Dian Fu added a comment - Attaching the 007 patch to trigger the jenkins again as I have checked the test failures in my local environment and all of them have passed.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 54s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 7m 57s There were no new javac warning messages.
          +1 javadoc 10m 11s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 53s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 4s The patch has no lines that end in whitespace.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 42s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 7s Tests passed in hadoop-yarn-common.
          -1 yarn tests 29m 1s Tests failed in hadoop-yarn-server-resourcemanager.
              79m 24s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates
            hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestApplicationACLs
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
            hadoop.yarn.server.resourcemanager.nodelabels.TestRMNodeLabelsManager
            hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler
            hadoop.yarn.server.resourcemanager.TestClientRMService
            hadoop.yarn.server.resourcemanager.TestApplicationCleanup
            hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels
            hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart
            hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesHttpStaticUserPermissions
            hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicy
            hadoop.yarn.server.resourcemanager.TestMoveApplication
            hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
            hadoop.yarn.server.resourcemanager.TestRM
            hadoop.yarn.server.resourcemanager.resourcetracker.TestRMNMRPCResponseId
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication
            hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
            hadoop.yarn.server.resourcemanager.nodelabels.TestRMNodeLabelsUpdater
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
            hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.rmapp.attempt.TestAMLivelinessMonitor
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
            hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor
            hadoop.yarn.server.resourcemanager.TestResourceManager
            hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
            hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId
            hadoop.yarn.server.resourcemanager.TestResourceTrackerService
            hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl
            hadoop.yarn.server.resourcemanager.resourcetracker.TestNMReconnect



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762033/YARN-3964.007.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 1f707ec
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9250/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9250/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9250/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9250/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9250/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9250/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 54s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 7m 57s There were no new javac warning messages. +1 javadoc 10m 11s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 53s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 4s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 42s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 7s Tests passed in hadoop-yarn-common. -1 yarn tests 29m 1s Tests failed in hadoop-yarn-server-resourcemanager.     79m 24s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates   hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestApplicationACLs   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.resourcemanager.nodelabels.TestRMNodeLabelsManager   hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler   hadoop.yarn.server.resourcemanager.TestClientRMService   hadoop.yarn.server.resourcemanager.TestApplicationCleanup   hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels   hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart   hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesHttpStaticUserPermissions   hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicy   hadoop.yarn.server.resourcemanager.TestMoveApplication   hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA   hadoop.yarn.server.resourcemanager.TestRM   hadoop.yarn.server.resourcemanager.resourcetracker.TestRMNMRPCResponseId   hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication   hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps   hadoop.yarn.server.resourcemanager.nodelabels.TestRMNodeLabelsUpdater   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication   hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.rmapp.attempt.TestAMLivelinessMonitor   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes   hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor   hadoop.yarn.server.resourcemanager.TestResourceManager   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage   hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId   hadoop.yarn.server.resourcemanager.TestResourceTrackerService   hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl   hadoop.yarn.server.resourcemanager.resourcetracker.TestNMReconnect Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762033/YARN-3964.007.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1f707ec checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9250/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9250/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9250/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9250/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9250/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9250/console This message was automatically generated.
          Hide
          dian.fu Dian Fu added a comment -

          Hi Wangda Tan, Naganarasimha G R, Devaraj K,
          I have run test-patch.sh in a clean environment (one newly created virtual machine) and all the tests have passed. Most of the test failures jenkins reported are caused by

          java.lang.NoSuchMethodError: org.apache.hadoop.yarn.conf.YarnConfiguration.isDelegatedCentralizedNodeLabelConfiguration(Lorg/apache/hadoop/conf/Configuration;)Z
          

          But method YarnConfiguration.isDelegatedCentralizedNodeLabelConfiguration does exist and the signature is also correct. Could you go ahead the review? I will also continue to investigate why the tests failed in jenkins machines.

          Show
          dian.fu Dian Fu added a comment - Hi Wangda Tan , Naganarasimha G R , Devaraj K , I have run test-patch.sh in a clean environment (one newly created virtual machine) and all the tests have passed. Most of the test failures jenkins reported are caused by java.lang.NoSuchMethodError: org.apache.hadoop.yarn.conf.YarnConfiguration.isDelegatedCentralizedNodeLabelConfiguration(Lorg/apache/hadoop/conf/Configuration;)Z But method YarnConfiguration.isDelegatedCentralizedNodeLabelConfiguration does exist and the signature is also correct. Could you go ahead the review? I will also continue to investigate why the tests failed in jenkins machines.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Few concerns what i have with this approach is :

          • CommonNodeLabelsManager.isDistributedNodeLabelConfiguration is set in distributed mode and we avoid reading and storing to FileSystemNodeLabelsStore. But in your case its not done, hence LabelStore Editlogs goes on increasing over time as we not checking for weather the labels are not modified for a given node while replacing. Eventually FileSystemNodeLabelsStore.recover during startup(/failover) might be come slow.
          • During update even if one node has Labels which is not part of CLuster Labels it fails to update for other nodes is that fine ?

          few nits :

          1. RMNodeLabelsUpdater
            • IMO we can name this class as RMDelegatedNodeLabelsUpdater/DelegatedNodeLabelsUpdater
            • NodeLabelsUpdaterThread can it be better handled using TimerTask or even Better have ScheduledThreadPoolExecutor which doesnt have impact of change in internal clock time and on any exception in the task, Executor itself doesnt fail.
            • In NodeLabelsUpdaterThread.run, On exception during update i think we should log LOG.error instead of warn
            • can we merge nodeLabelsUpdated and updateNodeLabelsInternal, as nodeLabelsUpdated is used only by updateNodeLabelsInternal
          2. AdminService & RMWebServices
            • checkAndThrowIfCentralizedNodeLabelConfNotEnabled => verifyCentralizedNodeLabelConfEnabled and moved to NodeLabelsUtils
          3. NodeLabelsProvider
            • The name of the interface seems to be same with the one in NM, i feel we can name it better as RMNodeLabelsProvider/NodeLabelsMappingProvider to avoid confusions
          Show
          Naganarasimha Naganarasimha G R added a comment - Few concerns what i have with this approach is : CommonNodeLabelsManager.isDistributedNodeLabelConfiguration is set in distributed mode and we avoid reading and storing to FileSystemNodeLabelsStore . But in your case its not done, hence LabelStore Editlogs goes on increasing over time as we not checking for weather the labels are not modified for a given node while replacing. Eventually FileSystemNodeLabelsStore.recover during startup(/failover) might be come slow. During update even if one node has Labels which is not part of CLuster Labels it fails to update for other nodes is that fine ? few nits : RMNodeLabelsUpdater IMO we can name this class as RMDelegatedNodeLabelsUpdater / DelegatedNodeLabelsUpdater NodeLabelsUpdaterThread can it be better handled using TimerTask or even Better have ScheduledThreadPoolExecutor which doesnt have impact of change in internal clock time and on any exception in the task, Executor itself doesnt fail. In NodeLabelsUpdaterThread.run , On exception during update i think we should log LOG.error instead of warn can we merge nodeLabelsUpdated and updateNodeLabelsInternal , as nodeLabelsUpdated is used only by updateNodeLabelsInternal AdminService & RMWebServices checkAndThrowIfCentralizedNodeLabelConfNotEnabled => verifyCentralizedNodeLabelConfEnabled and moved to NodeLabelsUtils NodeLabelsProvider The name of the interface seems to be same with the one in NM, i feel we can name it better as RMNodeLabelsProvider / NodeLabelsMappingProvider to avoid confusions
          Hide
          dian.fu Dian Fu added a comment -

          Hi Naganarasimha G R,
          Thanks a lot for your review. Have updated the patch according to your comments.

          CommonNodeLabelsManager.isDistributedNodeLabelConfiguration is set in distributed mode and we avoid reading and storing to FileSystemNodeLabelsStore. But in your case its not done, hence LabelStore Editlogs goes on increasing over time as we not checking for weather the labels are not modified for a given node while replacing. Eventually FileSystemNodeLabelsStore.recover during startup(/failover) might be come slow.

          In fact, in delegated-centralized mode, we should also avoid storing the node->labels map to FileSystemNodeLabelsStore. The new patch should have fixed this.

          During update even if one node has Labels which is not part of CLuster Labels it fails to update for other nodes is that fine ?

          This issue also exists when updating multiple nodes labels through CLI/REST. Maybe we can address this in a separate JIRA if we think this is a problem.

          Show
          dian.fu Dian Fu added a comment - Hi Naganarasimha G R , Thanks a lot for your review. Have updated the patch according to your comments. CommonNodeLabelsManager.isDistributedNodeLabelConfiguration is set in distributed mode and we avoid reading and storing to FileSystemNodeLabelsStore. But in your case its not done, hence LabelStore Editlogs goes on increasing over time as we not checking for weather the labels are not modified for a given node while replacing. Eventually FileSystemNodeLabelsStore.recover during startup(/failover) might be come slow. In fact, in delegated-centralized mode, we should also avoid storing the node->labels map to FileSystemNodeLabelsStore. The new patch should have fixed this. During update even if one node has Labels which is not part of CLuster Labels it fails to update for other nodes is that fine ? This issue also exists when updating multiple nodes labels through CLI/REST. Maybe we can address this in a separate JIRA if we think this is a problem.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 45s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 7m 44s There were no new javac warning messages.
          +1 javadoc 10m 19s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 57s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 4s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 45s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 3s Tests passed in hadoop-yarn-common.
          +1 yarn tests 55m 44s Tests passed in hadoop-yarn-server-resourcemanager.
              105m 47s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762352/YARN-3964.008.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 83e65c5
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9264/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9264/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9264/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9264/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9264/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9264/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 45s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 7m 44s There were no new javac warning messages. +1 javadoc 10m 19s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 57s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 4s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 45s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 3s Tests passed in hadoop-yarn-common. +1 yarn tests 55m 44s Tests passed in hadoop-yarn-server-resourcemanager.     105m 47s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762352/YARN-3964.008.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 83e65c5 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9264/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9264/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9264/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9264/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9264/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9264/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Dian Fu,
          For reworking on the comments, few nits & queries :

          1. DelegatedNodeLabelsUpdater
            • If updateInterval is disabled then only on register of nodes the labels are fetched, is that fine or is it better to fetch once and not run timer ?
            • Can there be chances that NodeLabelsMappingProvider requires configuration to initialize itself ? would it better to added it as service ?
            • updateNodeLabels(NodeId node) on exception we need to log with error instead of warn
          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Dian Fu , For reworking on the comments, few nits & queries : DelegatedNodeLabelsUpdater If updateInterval is disabled then only on register of nodes the labels are fetched, is that fine or is it better to fetch once and not run timer ? Can there be chances that NodeLabelsMappingProvider requires configuration to initialize itself ? would it better to added it as service ? updateNodeLabels(NodeId node) on exception we need to log with error instead of warn
          Hide
          sunilg Sunil G added a comment -

          Hi Dian Fu
          Couple of points.
          1. Is it better to check for yarn.node-labels.enabled also in DelegatedNodeLabelsUpdater init. Else timer thread will print exception every time.
          2. createNodeLabelsMappingProvider throws exception from serviceInit if NodeLabelsMappingProvider class is not configured. This will disrupt RM. I feel you can create a SimpleNodeLabelsMappingProvider and keep that as default, and timer can also be stopped in this case from provider. Thoughts?

          Show
          sunilg Sunil G added a comment - Hi Dian Fu Couple of points. 1. Is it better to check for yarn.node-labels.enabled also in DelegatedNodeLabelsUpdater init. Else timer thread will print exception every time. 2. createNodeLabelsMappingProvider throws exception from serviceInit if NodeLabelsMappingProvider class is not configured. This will disrupt RM. I feel you can create a SimpleNodeLabelsMappingProvider and keep that as default, and timer can also be stopped in this case from provider. Thoughts?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Sunil G,
          Good points sunil, But i feel we can have different approach

          Is it better to check for yarn.node-labels.enabled also in DelegatedNodeLabelsUpdater init.

          Instead we can avoid creating DelegatedNodeLabelsUpdater in createDelegatedNodeLabelsUpdater if yarn.node-labels.enabled is set as false.

          createNodeLabelsMappingProvider throws exception from serviceInit if NodeLabelsMappingProvider class is not configured.

          If the configuration is explicitly set for delegated-centralized then i prefer to fail fast if provider is not configured properly, similarly what we plan to do in NM side when node labels script path are not configured but provider is set to script . Thoughts ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Sunil G , Good points sunil, But i feel we can have different approach Is it better to check for yarn.node-labels.enabled also in DelegatedNodeLabelsUpdater init. Instead we can avoid creating DelegatedNodeLabelsUpdater in createDelegatedNodeLabelsUpdater if yarn.node-labels.enabled is set as false. createNodeLabelsMappingProvider throws exception from serviceInit if NodeLabelsMappingProvider class is not configured. If the configuration is explicitly set for delegated-centralized then i prefer to fail fast if provider is not configured properly, similarly what we plan to do in NM side when node labels script path are not configured but provider is set to script . Thoughts ?
          Hide
          sunilg Sunil G added a comment -

          Hi Naganarasimha G R
          Yes, We could not even create provider in case of point 1. Much better.

          For point2, I am still not very convinced. Fail-fast may be too aggressive right?. We could think in both way also here. As we are giving the user the flexibility of providing a provider, implementation can be anything. So keeping a dummy one if no config is present and give a warning is also an option. I had this thought only bcz we are making RM fail-fast which I am doubting whether its too aggressive or not.

          Show
          sunilg Sunil G added a comment - Hi Naganarasimha G R Yes, We could not even create provider in case of point 1. Much better. For point2, I am still not very convinced. Fail-fast may be too aggressive right?. We could think in both way also here. As we are giving the user the flexibility of providing a provider, implementation can be anything. So keeping a dummy one if no config is present and give a warning is also an option. I had this thought only bcz we are making RM fail-fast which I am doubting whether its too aggressive or not.
          Hide
          dian.fu Dian Fu added a comment -

          Hi Naganarasimha G R, Sunil G,
          Thanks a lot for your review. Updated the patch according to your comments with the following exceptions:

          If updateInterval is disabled then only on register of nodes the labels are fetched, is that fine or is it better to fetch once and not run timer ?

          You mean not only fetching the labels on register, but also fetching the labels once for all the nodes when all nodes are registered? If it's so, I think this's unnecessary and difficult to implement. Firstly, the labels of all the registered nodes have been fetched once on register. Secondly, it's difficult to determine when to fetch the labels for all the nodes as some nodes may still not registered. Thoughts?

          createNodeLabelsMappingProvider throws exception from serviceInit if NodeLabelsMappingProvider class is not configured.

          I would agree with Naganarasimha G R on this point. By default, centralized node label configuration type is enabled. If user explicitly set the node label configuration type to delegated-centralized, we can assume that he knows what does it mean for delegated-centralized and he knows he should provide a NodeLabelsMappingProvider. On the other hand, providing a SimpleNodeLabelsMappingProvider and make it as default would also be acceptable for me. If we decide to do this, then we should log the following with INFO to let user know which NodeLabelsMappingProvider is being used.

          {{LOG.debug("Node labels mapping provider class is : "
          157	          + nodeLabelsMappingProvider.getClass().toString());}} 
          

          Thoughts?

          Show
          dian.fu Dian Fu added a comment - Hi Naganarasimha G R , Sunil G , Thanks a lot for your review. Updated the patch according to your comments with the following exceptions: If updateInterval is disabled then only on register of nodes the labels are fetched, is that fine or is it better to fetch once and not run timer ? You mean not only fetching the labels on register, but also fetching the labels once for all the nodes when all nodes are registered? If it's so, I think this's unnecessary and difficult to implement. Firstly, the labels of all the registered nodes have been fetched once on register. Secondly, it's difficult to determine when to fetch the labels for all the nodes as some nodes may still not registered. Thoughts? createNodeLabelsMappingProvider throws exception from serviceInit if NodeLabelsMappingProvider class is not configured. I would agree with Naganarasimha G R on this point. By default, centralized node label configuration type is enabled. If user explicitly set the node label configuration type to delegated-centralized , we can assume that he knows what does it mean for delegated-centralized and he knows he should provide a NodeLabelsMappingProvider . On the other hand, providing a SimpleNodeLabelsMappingProvider and make it as default would also be acceptable for me. If we decide to do this, then we should log the following with INFO to let user know which NodeLabelsMappingProvider is being used. {{LOG.debug("Node labels mapping provider class is : " 157 + nodeLabelsMappingProvider.getClass().toString());}} Thoughts?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Sunil G,
          Point2 is debatable, my 2 cents here is, if delegated-centralized was the default i can agree, and if we have a default implementation which serves some purpose like MemoryRMStateStore then fine, if not i am not too convinced to create a dummy provider instance when user is explicitly setting delegated-centralized, cc/ Wangda Tan ur thoughts ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Sunil G , Point2 is debatable, my 2 cents here is, if delegated-centralized was the default i can agree, and if we have a default implementation which serves some purpose like MemoryRMStateStore then fine, if not i am not too convinced to create a dummy provider instance when user is explicitly setting delegated-centralized , cc/ Wangda Tan ur thoughts ?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Dian Fu,

          Secondly, it's difficult to determine when to fetch the labels for all the nodes as some nodes may still not registered. Thoughts?

          makes sense, missed the point that the provider provides labels for the requested nodes only !.
          Also have a feeling that default interval time can be as big as 30 mins as we are updating the labels for all nodes without checking if its modified or not. Thoughts ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Dian Fu , Secondly, it's difficult to determine when to fetch the labels for all the nodes as some nodes may still not registered. Thoughts? makes sense, missed the point that the provider provides labels for the requested nodes only !. Also have a feeling that default interval time can be as big as 30 mins as we are updating the labels for all nodes without checking if its modified or not. Thoughts ?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 20m 1s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 8m 3s There were no new javac warning messages.
          +1 javadoc 10m 13s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 3s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 3s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 4s Tests passed in hadoop-yarn-common.
          -1 yarn tests 60m 11s Tests failed in hadoop-yarn-server-resourcemanager.
              111m 2s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.resourcemanager.nodelabels.TestDelegatedNodeLabelsUpdater



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762490/YARN-3964.009.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 67b0e96
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9269/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9269/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9269/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9269/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9269/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9269/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 1s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 8m 3s There were no new javac warning messages. +1 javadoc 10m 13s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 3s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 4s Tests passed in hadoop-yarn-common. -1 yarn tests 60m 11s Tests failed in hadoop-yarn-server-resourcemanager.     111m 2s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.nodelabels.TestDelegatedNodeLabelsUpdater Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762490/YARN-3964.009.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 67b0e96 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9269/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9269/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9269/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9269/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9269/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9269/console This message was automatically generated.
          Hide
          dian.fu Dian Fu added a comment -

          Updated the patch to fix the test failures and change the default interval time to 30 minutes.

          Show
          dian.fu Dian Fu added a comment - Updated the patch to fix the test failures and change the default interval time to 30 minutes.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 20m 59s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 9m 4s There were no new javac warning messages.
          +1 javadoc 11m 40s There were no new javadoc warning messages.
          +1 release audit 0m 28s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 5s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 4s The patch has no lines that end in whitespace.
          +1 install 1m 52s mvn install still works.
          +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse.
          +1 findbugs 5m 39s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 27s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 14s Tests passed in hadoop-yarn-common.
          -1 yarn tests 63m 50s Tests failed in hadoop-yarn-server-resourcemanager.
              119m 35s  



          Reason Tests
          Timed out tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762564/YARN-3964.011.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / bf37d3d
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9276/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9276/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9276/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9276/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9276/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9276/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 59s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 9m 4s There were no new javac warning messages. +1 javadoc 11m 40s There were no new javadoc warning messages. +1 release audit 0m 28s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 5s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 4s The patch has no lines that end in whitespace. +1 install 1m 52s mvn install still works. +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse. +1 findbugs 5m 39s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 27s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 14s Tests passed in hadoop-yarn-common. -1 yarn tests 63m 50s Tests failed in hadoop-yarn-server-resourcemanager.     119m 35s   Reason Tests Timed out tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762564/YARN-3964.011.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / bf37d3d checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9276/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9276/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9276/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9276/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9276/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9276/console This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks for updates, Dian Fu!

          Point2 is debatable, my 2 cents here is, if delegated-centralized was the default i can agree, and if we have a default implementation which serves some purpose like MemoryRMStateStore then fine, if not i am not too convinced to create a dummy provider instance when user is explicitly setting delegated-centralized, cc/ Wangda Tan ur thoughts ?

          I would prefer to the fail-fast solution, since it is a configuration issue, skip it may cause RM in an unexpected behavior. This is a little different from RMStore failure, this is configuration issue and another one is runtime issue. User should fix configuration before start RM.

          Some comments:

          • DelegatedNodeLabelsUpdater -> RMDelegatedNodeLabelsUpdater
          • Add a comment to DelegatedNodeLabelsUpdater explains how it works and which configuration controls enable it.
          • NodeLabelsMappingProvider -> RMNodeLabelsMappingProvider

          And I'm not sure if you noticed YARN-4106 and YARN-4176, I think the two issues are related to what you're trying to do. Maybe it's better to consider them within your patch.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks for updates, Dian Fu ! Point2 is debatable, my 2 cents here is, if delegated-centralized was the default i can agree, and if we have a default implementation which serves some purpose like MemoryRMStateStore then fine, if not i am not too convinced to create a dummy provider instance when user is explicitly setting delegated-centralized, cc/ Wangda Tan ur thoughts ? I would prefer to the fail-fast solution, since it is a configuration issue, skip it may cause RM in an unexpected behavior. This is a little different from RMStore failure, this is configuration issue and another one is runtime issue. User should fix configuration before start RM. Some comments: DelegatedNodeLabelsUpdater -> RMDelegatedNodeLabelsUpdater Add a comment to DelegatedNodeLabelsUpdater explains how it works and which configuration controls enable it. NodeLabelsMappingProvider -> RMNodeLabelsMappingProvider And I'm not sure if you noticed YARN-4106 and YARN-4176 , I think the two issues are related to what you're trying to do. Maybe it's better to consider them within your patch.
          Hide
          dian.fu Dian Fu added a comment -

          Hi Wangda Tan,
          Thanks a lot for your review. Updated the patch according to your comments. Regarding to the issues YARN-4106 and YARN-4176, this patch doesn't have the same issue as currently when updating node labels, RMDelegatedNodeLabelsUpdater doesn't check if the labels for a node has changed and just call RMNodeLabelsManager#replaceLabelsOnNode. The reason is that for delegated-centralized type, we only update the nodes -> labels map stored in memory and this should be very fast. Thoughts?

          Show
          dian.fu Dian Fu added a comment - Hi Wangda Tan , Thanks a lot for your review. Updated the patch according to your comments. Regarding to the issues YARN-4106 and YARN-4176 , this patch doesn't have the same issue as currently when updating node labels, RMDelegatedNodeLabelsUpdater doesn't check if the labels for a node has changed and just call RMNodeLabelsManager#replaceLabelsOnNode . The reason is that for delegated-centralized type, we only update the nodes -> labels map stored in memory and this should be very fast. Thoughts?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Dian Fu,
          The scenario what is addressed as per YARN-4176 is : how to handle when Cluster Node label which is already mapped to a node, is removed in RM. As NM will report labels to RM only when there is any difference from previous labels sent NM, its not aware that the cluster label is removed and there is no way NM show warning or error that label reported by it is wrong.
          But according to Delegated Node Labels solution, labels for all the nodes are updated without verifying whether there is any change from previous state. Hence same issue doesn't arise directly. So to minimize the impact we have default interval of 30 mins. But one doubt what i have is, when there is large cluster like 1000 or more (practical limit what i have heard is 50000) so what would be the load on CS on refresh of Labels on all nodes though there is no change.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Dian Fu , The scenario what is addressed as per YARN-4176 is : how to handle when Cluster Node label which is already mapped to a node, is removed in RM . As NM will report labels to RM only when there is any difference from previous labels sent NM, its not aware that the cluster label is removed and there is no way NM show warning or error that label reported by it is wrong. But according to Delegated Node Labels solution, labels for all the nodes are updated without verifying whether there is any change from previous state. Hence same issue doesn't arise directly. So to minimize the impact we have default interval of 30 mins. But one doubt what i have is, when there is large cluster like 1000 or more (practical limit what i have heard is 50000) so what would be the load on CS on refresh of Labels on all nodes though there is no change.
          Hide
          sunilg Sunil G added a comment -

          Hi NGarla_Unused
          I had this concern also earlier. updateNodeLabelsInternal is looping on all nodes in cluster, and its been triggered from a timer (30 mins default). I feel as nodes are more in cluster, and response time from nodeLabelsMappingProvider.getNodeLabels may also be more. So I feel its upto the nodeLabelsMappingProvider to have a better filtering mechanism and some pre-computed information to skip non-changed nodes to improve the response time of api. Not sure whether RM could do more here.

          Show
          sunilg Sunil G added a comment - Hi NGarla_Unused I had this concern also earlier. updateNodeLabelsInternal is looping on all nodes in cluster, and its been triggered from a timer (30 mins default). I feel as nodes are more in cluster, and response time from nodeLabelsMappingProvider.getNodeLabels may also be more. So I feel its upto the nodeLabelsMappingProvider to have a better filtering mechanism and some pre-computed information to skip non-changed nodes to improve the response time of api. Not sure whether RM could do more here.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Sunil G and Tan, Wangda,
          One small optimization i can think of is

          // CapacityScheduler.updateLabelsOnNode
          // labels is same, we don't need do update
              if (node.getLabels().size() == newLabels.size()
                  && node.getLabels().containsAll(newLabels)) {
                return;
              }
          

          This piece of code i think should be done by the caller handle(SchedulerEvent event) rather than updateLabelsOnNode as its synchronized method. If req to handle it in another jira i can raise one and work on it.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Sunil G and Tan, Wangda , One small optimization i can think of is // CapacityScheduler.updateLabelsOnNode // labels is same, we don't need do update if (node.getLabels().size() == newLabels.size() && node.getLabels().containsAll(newLabels)) { return ; } This piece of code i think should be done by the caller handle(SchedulerEvent event) rather than updateLabelsOnNode as its synchronized method. If req to handle it in another jira i can raise one and work on it.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 20m 8s Pre-patch trunk compilation is healthy.
          -1 @author 0m 0s The patch appears to contain 1 @author tags which the Hadoop community has agreed to not allow in code contributions.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 8m 16s There were no new javac warning messages.
          +1 javadoc 10m 11s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 1s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 4s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          +1 findbugs 4m 39s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 1s Tests passed in hadoop-yarn-common.
          +1 yarn tests 60m 25s Tests passed in hadoop-yarn-server-resourcemanager.
              111m 11s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12764020/YARN-3964.012.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 66dad85
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9284/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9284/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9284/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9284/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9284/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9284/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 8s Pre-patch trunk compilation is healthy. -1 @author 0m 0s The patch appears to contain 1 @author tags which the Hadoop community has agreed to not allow in code contributions. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 8m 16s There were no new javac warning messages. +1 javadoc 10m 11s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 1s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 4s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 4m 39s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 1s Tests passed in hadoop-yarn-common. +1 yarn tests 60m 25s Tests passed in hadoop-yarn-server-resourcemanager.     111m 11s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12764020/YARN-3964.012.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 66dad85 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9284/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9284/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9284/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9284/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9284/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9284/console This message was automatically generated.
          Hide
          dian.fu Dian Fu added a comment -

          Hi Naganarasimha G R and Sunil G,
          Thanks for your comments.

          I feel as nodes are more in cluster, and response time from nodeLabelsMappingProvider.getNodeLabels may also be more.

          updateNodeLabelsInternal is called in two places: one place is in thread RMDelegatedNodeLabelsUpdaterTimerTask and the other place is on register.

          • When called in thread RMDelegatedNodeLabelsUpdaterTimerTask, the impact is minor as it is working in a separate thread and so the response time of RMDelegatedNodeLabelsUpdaterTimerTask won't impact other threads. Further more, only one call of nodeLabelsMappingProvider.getNodeLabels can fetch all the node labels and so the response time should not be too long.
          • When called on register, the response time of nodeLabelsMappingProvider.getNodeLabels does have impact as it may lead to longer register time. That's the reason why patch 006 doesn't update node labels synchronously on register. Thoughts?


            This piece of code i think should be done by the caller handle(SchedulerEvent event) rather than updateLabelsOnNode as its synchronized method.

            Make sense.

          Show
          dian.fu Dian Fu added a comment - Hi Naganarasimha G R and Sunil G , Thanks for your comments. I feel as nodes are more in cluster, and response time from nodeLabelsMappingProvider.getNodeLabels may also be more. updateNodeLabelsInternal is called in two places: one place is in thread RMDelegatedNodeLabelsUpdaterTimerTask and the other place is on register. When called in thread RMDelegatedNodeLabelsUpdaterTimerTask , the impact is minor as it is working in a separate thread and so the response time of RMDelegatedNodeLabelsUpdaterTimerTask won't impact other threads. Further more, only one call of nodeLabelsMappingProvider.getNodeLabels can fetch all the node labels and so the response time should not be too long. When called on register, the response time of nodeLabelsMappingProvider.getNodeLabels does have impact as it may lead to longer register time. That's the reason why patch 006 doesn't update node labels synchronously on register. Thoughts? This piece of code i think should be done by the caller handle(SchedulerEvent event) rather than updateLabelsOnNode as its synchronized method. Make sense.
          Hide
          leftnoteasy Wangda Tan added a comment -

          This piece of code i think should be done by the caller handle(SchedulerEvent event) rather than updateLabelsOnNode as its synchronized method. If req to handle it in another jira i can raise one and work on it.

          I would prefer to do this at RMNodeLabelsManager, we don't need to check this in every scheduler implementation. Which could be done in a separated JIRA.

          Show
          leftnoteasy Wangda Tan added a comment - This piece of code i think should be done by the caller handle(SchedulerEvent event) rather than updateLabelsOnNode as its synchronized method. If req to handle it in another jira i can raise one and work on it. I would prefer to do this at RMNodeLabelsManager, we don't need to check this in every scheduler implementation. Which could be done in a separated JIRA.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Dian Fu,

          RMDelegatedNodeLabelsUpdater doesn't check if the labels for a node has changed and just call RMNodeLabelsManager#replaceLabelsOnNode. The reason is that for delegated-centralized type, we only update the nodes -> labels map stored in memory and this should be very fast. Thoughts?

          Make sense to me, as I commented above, if we solve the locking issue, it should be fine since all operations are in memory.

          Show
          leftnoteasy Wangda Tan added a comment - Dian Fu , RMDelegatedNodeLabelsUpdater doesn't check if the labels for a node has changed and just call RMNodeLabelsManager#replaceLabelsOnNode. The reason is that for delegated-centralized type, we only update the nodes -> labels map stored in memory and this should be very fast. Thoughts? Make sense to me, as I commented above, if we solve the locking issue, it should be fine since all operations are in memory.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the input Tan, Wangda, Yes it would be better to be handled in RMNodeLabelsManager. i will raise a new jira and work on it.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the input Tan, Wangda , Yes it would be better to be handled in RMNodeLabelsManager. i will raise a new jira and work on it.
          Hide
          dian.fu Dian Fu added a comment -

          Thanks a lot for the comments Wangda Tan. What's your thought about the impact of the response time of nodeLabelsMappingProvider.getNodeLabels?

          Show
          dian.fu Dian Fu added a comment - Thanks a lot for the comments Wangda Tan . What's your thought about the impact of the response time of nodeLabelsMappingProvider.getNodeLabels ?
          Hide
          leftnoteasy Wangda Tan added a comment -

          What's your thought about the impact of the response time of nodeLabelsMappingProvider.getNodeLabels?

          I think it should be fine to me if we solved the locking issue.

          Show
          leftnoteasy Wangda Tan added a comment - What's your thought about the impact of the response time of nodeLabelsMappingProvider.getNodeLabels? I think it should be fine to me if we solved the locking issue.
          Hide
          dian.fu Dian Fu added a comment -

          Hi Wangda Tan, Naganarasimha G R and Sunil G,
          Thanks for your inputs. Updated the patch to eliminate the impact of the response time of nodeLabelsMappingProvider.getNodeLabels.

          Show
          dian.fu Dian Fu added a comment - Hi Wangda Tan , Naganarasimha G R and Sunil G , Thanks for your inputs. Updated the patch to eliminate the impact of the response time of nodeLabelsMappingProvider.getNodeLabels .
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 21m 29s Pre-patch trunk compilation is healthy.
          -1 @author 0m 0s The patch appears to contain 1 @author tags which the Hadoop community has agreed to not allow in code contributions.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 8m 19s There were no new javac warning messages.
          +1 javadoc 13m 12s There were no new javadoc warning messages.
          +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 14s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 6s The patch has no lines that end in whitespace.
          +1 install 2m 1s mvn install still works.
          +1 eclipse:eclipse 0m 45s The patch built with eclipse:eclipse.
          +1 findbugs 6m 19s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 34s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 28s Tests passed in hadoop-yarn-common.
          +1 yarn tests 59m 53s Tests passed in hadoop-yarn-server-resourcemanager.
              118m 33s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12764377/YARN-3964.013.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 06abc57
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9306/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9306/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9306/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9306/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9306/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9306/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 21m 29s Pre-patch trunk compilation is healthy. -1 @author 0m 0s The patch appears to contain 1 @author tags which the Hadoop community has agreed to not allow in code contributions. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 8m 19s There were no new javac warning messages. +1 javadoc 13m 12s There were no new javadoc warning messages. +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 14s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 6s The patch has no lines that end in whitespace. +1 install 2m 1s mvn install still works. +1 eclipse:eclipse 0m 45s The patch built with eclipse:eclipse. +1 findbugs 6m 19s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 34s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 28s Tests passed in hadoop-yarn-common. +1 yarn tests 59m 53s Tests passed in hadoop-yarn-server-resourcemanager.     118m 33s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12764377/YARN-3964.013.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 06abc57 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9306/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9306/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9306/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9306/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9306/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9306/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the patch Dian Fu,

          I think it should be fine to me if we solved the locking issue.

          By this comment, IIUC Tan, Wangda meant about the issue which i mentioned. And as per the approach discussed it would be better to be handled in RMNodeLabelsManager (for which i am coming up with a jira and patch) so that it solves issue in all types of Node label configuration (centralized , distributed & this). But still feel the approach in the new patch is better as it would be holding less locks on RMNodeLabelsManager and will try to club and fetch multiple requests at one shot!
          Few comments :

          1. Synchronizations in RMDelegatedNodeLabelsUpdaterTimerTask is not proper. {{ synchronized(this)}} holds lock on RMDelegatedNodeLabelsUpdaterTimerTask instance but newlyRegisteredNodes is updated in updateNodeLabels with the lock on RMDelegatedNodeLabelsUpdater instance.
          2. nodesToUpdateLabels == null is req in the below code ?
            if (nodesToUpdateLabels == null && !newlyRegisteredNodes.isEmpty()) {
            	        synchronized(this) {
            	          if (!newlyRegisteredNodes.isEmpty()) {
            	            nodesToUpdateLabels = new HashSet<NodeId>(newlyRegisteredNodes);
            	          }
            	        }
            	      }
            

            nodesToUpdateLabels == null is req ?

          3. 5 Seconds span i feel is little too much, better to have 30 seconds, if the provider is doing some operation which takes more than 5 seconds then multiple tasks can pile up.

          Hope you can share your test code for the RMNodeLabelMappingsUpdater with which it i can test, hope you also have verified it .

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the patch Dian Fu , I think it should be fine to me if we solved the locking issue. By this comment, IIUC Tan, Wangda meant about the issue which i mentioned. And as per the approach discussed it would be better to be handled in RMNodeLabelsManager (for which i am coming up with a jira and patch) so that it solves issue in all types of Node label configuration (centralized , distributed & this). But still feel the approach in the new patch is better as it would be holding less locks on RMNodeLabelsManager and will try to club and fetch multiple requests at one shot! Few comments : Synchronizations in RMDelegatedNodeLabelsUpdaterTimerTask is not proper. {{ synchronized(this)}} holds lock on RMDelegatedNodeLabelsUpdaterTimerTask instance but newlyRegisteredNodes is updated in updateNodeLabels with the lock on RMDelegatedNodeLabelsUpdater instance. nodesToUpdateLabels == null is req in the below code ? if (nodesToUpdateLabels == null && !newlyRegisteredNodes.isEmpty()) { synchronized ( this ) { if (!newlyRegisteredNodes.isEmpty()) { nodesToUpdateLabels = new HashSet<NodeId>(newlyRegisteredNodes); } } } nodesToUpdateLabels == null is req ? 5 Seconds span i feel is little too much, better to have 30 seconds, if the provider is doing some operation which takes more than 5 seconds then multiple tasks can pile up. Hope you can share your test code for the RMNodeLabelMappingsUpdater with which it i can test, hope you also have verified it .
          Hide
          dian.fu Dian Fu added a comment -

          Hi Naganarasimha G R,
          Thanks a lot for your review. Updated the patch accordingly.

          nodesToUpdateLabels == null is req ?

          Yes. If nodesToUpdateLabels isn't null, then nodesToUpdateLabels contains all the cluster nodes (also including the newly registered nodes). In this case, we don't need to add the newly registered nodes to nodesToUpdateLabels any more.

          Hope you can share your test code for the RMNodeLabelMappingsUpdater with which it i can test, hope you also have verified it .

          I have an implementation of RMNodeLabelMappingsUpdater which can fetch nodes trust status from Mt. Wilson server and translate this trust status information into node trust/untrust labels. As the Mt. Wilson is a commercial software and the environment setup of Mt. Wilson server is a little complicated, it would be difficult for you to use my test code directly. I think you can write a simple implementation of RMNodeLabelMappingsUpdater which returns hard-coded labels for the test.

          Show
          dian.fu Dian Fu added a comment - Hi Naganarasimha G R , Thanks a lot for your review. Updated the patch accordingly. nodesToUpdateLabels == null is req ? Yes. If nodesToUpdateLabels isn't null, then nodesToUpdateLabels contains all the cluster nodes (also including the newly registered nodes). In this case, we don't need to add the newly registered nodes to nodesToUpdateLabels any more. Hope you can share your test code for the RMNodeLabelMappingsUpdater with which it i can test, hope you also have verified it . I have an implementation of RMNodeLabelMappingsUpdater which can fetch nodes trust status from Mt. Wilson server and translate this trust status information into node trust/untrust labels. As the Mt. Wilson is a commercial software and the environment setup of Mt. Wilson server is a little complicated, it would be difficult for you to use my test code directly. I think you can write a simple implementation of RMNodeLabelMappingsUpdater which returns hard-coded labels for the test.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 20m 8s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 8m 21s There were no new javac warning messages.
          +1 javadoc 10m 29s There were no new javadoc warning messages.
          -1 release audit 0m 15s The applied patch generated 1 release audit warnings.
          -1 checkstyle 2m 2s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 4s The patch has no lines that end in whitespace.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 4m 44s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 2s Tests passed in hadoop-yarn-common.
          +1 yarn tests 60m 50s Tests passed in hadoop-yarn-server-resourcemanager.
              112m 4s  



          Reason Tests
          FindBugs module:hadoop-yarn-server-resourcemanager



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12764576/YARN-3964.014.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 195793c
          Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9322/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9322/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9322/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9322/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9322/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9322/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9322/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9322/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 8s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 8m 21s There were no new javac warning messages. +1 javadoc 10m 29s There were no new javadoc warning messages. -1 release audit 0m 15s The applied patch generated 1 release audit warnings. -1 checkstyle 2m 2s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 4s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 4m 44s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 2s Tests passed in hadoop-yarn-common. +1 yarn tests 60m 50s Tests passed in hadoop-yarn-server-resourcemanager.     112m 4s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12764576/YARN-3964.014.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 195793c Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9322/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9322/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9322/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9322/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9322/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9322/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9322/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9322/console This message was automatically generated.
          Hide
          dian.fu Dian Fu added a comment -

          Attaching a new patch to fix the findbugs warning.

          Show
          dian.fu Dian Fu added a comment - Attaching a new patch to fix the findbugs warning.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 51s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 7m 58s There were no new javac warning messages.
          +1 javadoc 10m 6s There were no new javadoc warning messages.
          -1 release audit 0m 16s The applied patch generated 1 release audit warnings.
          -1 checkstyle 2m 0s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 4s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 3s Tests passed in hadoop-yarn-common.
          +1 yarn tests 60m 25s Tests passed in hadoop-yarn-server-resourcemanager.
              110m 29s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12764740/YARN-3964.015.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / fd026f5
          Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9328/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9328/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9328/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9328/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9328/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9328/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9328/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 51s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 7m 58s There were no new javac warning messages. +1 javadoc 10m 6s There were no new javadoc warning messages. -1 release audit 0m 16s The applied patch generated 1 release audit warnings. -1 checkstyle 2m 0s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 4s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 3s Tests passed in hadoop-yarn-common. +1 yarn tests 60m 25s Tests passed in hadoop-yarn-server-resourcemanager.     110m 29s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12764740/YARN-3964.015.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / fd026f5 Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9328/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9328/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9328/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9328/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9328/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9328/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9328/console This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Dian Fu working on this patch and reviews from Naganarasimha G R/Sunil G. I think the approach/patch generally looks good and safe to me. Devaraj K, could you take care of following review works if you have bandwidth?

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Dian Fu working on this patch and reviews from Naganarasimha G R / Sunil G . I think the approach/patch generally looks good and safe to me. Devaraj K , could you take care of following review works if you have bandwidth?
          Hide
          devaraj.k Devaraj K added a comment -

          Thanks Wangda Tan for review and confirmation, Naganarasimha G R and Sunil G for reviews.

          Thanks Dian Fu for the patch, It mostly looks good to me except these minor comments.

          1. Can you update the descriptions for the new configs added in yarn-default.xml

          +    The class to use as the node labels fetcher by ResourceManager. It should
          +    extend org.apache.hadoop.yarn.server.resourcemanager.nodelabels.
          +    RMNodeLabelsMappingProvider.
          

          Can you update the description like below,
          'When node labels "yarn.node-labels.configuration-type" is
          of type "delegated-centralized", Administrators can configure
          the class for fetching node labels by ResourceManager. Configured
          class needs to extend
          org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsMappingProvider.'

          +    The interval to use to update node labels by ResourceManager.
          

          Can we think of having it like 'This interval is used to update the node labels by ResourceManager.'? And also can we describe here that if the value is '-1' then there will not be any timer task gets created.

          2. In TestRMDelegatedNodeLabelsUpdater.java, can we have an assertion in catch block to check the expected exception message?

              } catch (Exception e) {
                // expected
              }
             

          3. Can you file a Jira to update the documentation for this?

          Show
          devaraj.k Devaraj K added a comment - Thanks Wangda Tan for review and confirmation, Naganarasimha G R and Sunil G for reviews. Thanks Dian Fu for the patch, It mostly looks good to me except these minor comments. 1. Can you update the descriptions for the new configs added in yarn-default.xml + The class to use as the node labels fetcher by ResourceManager. It should + extend org.apache.hadoop.yarn.server.resourcemanager.nodelabels. + RMNodeLabelsMappingProvider. Can you update the description like below, 'When node labels "yarn.node-labels.configuration-type" is of type "delegated-centralized", Administrators can configure the class for fetching node labels by ResourceManager. Configured class needs to extend org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsMappingProvider.' + The interval to use to update node labels by ResourceManager. Can we think of having it like 'This interval is used to update the node labels by ResourceManager.'? And also can we describe here that if the value is '-1' then there will not be any timer task gets created. 2. In TestRMDelegatedNodeLabelsUpdater.java, can we have an assertion in catch block to check the expected exception message? } catch (Exception e) { // expected } 3. Can you file a Jira to update the documentation for this?
          Hide
          dian.fu Dian Fu added a comment -

          Hi Devaraj K,
          Thanks a lot for your review. Updated the patch accordingly. Have also created ticket YARN-4240 for the documentation.

          Show
          dian.fu Dian Fu added a comment - Hi Devaraj K , Thanks a lot for your review. Updated the patch accordingly. Have also created ticket YARN-4240 for the documentation.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 22m 28s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 8m 42s There were no new javac warning messages.
          +1 javadoc 11m 13s There were no new javadoc warning messages.
          -1 release audit 0m 20s The applied patch generated 1 release audit warnings.
          -1 checkstyle 2m 6s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 4s The patch has no lines that end in whitespace.
          +1 install 1m 41s mvn install still works.
          +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
          +1 findbugs 5m 2s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 27s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 8s Tests passed in hadoop-yarn-common.
          -1 yarn tests 51m 0s Tests failed in hadoop-yarn-server-resourcemanager.
              106m 23s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesHttpStaticUserPermissions
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler
            hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens
            hadoop.yarn.server.resourcemanager.TestResourceManager
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication
            hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart
            hadoop.yarn.server.resourcemanager.TestRMAdminService
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12765542/YARN-3964.016.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 1107bd3
          Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9377/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9377/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9377/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9377/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9377/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9377/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9377/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 22m 28s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 8m 42s There were no new javac warning messages. +1 javadoc 11m 13s There were no new javadoc warning messages. -1 release audit 0m 20s The applied patch generated 1 release audit warnings. -1 checkstyle 2m 6s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 4s The patch has no lines that end in whitespace. +1 install 1m 41s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 5m 2s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 27s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 8s Tests passed in hadoop-yarn-common. -1 yarn tests 51m 0s Tests failed in hadoop-yarn-server-resourcemanager.     106m 23s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesHttpStaticUserPermissions   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler   hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens   hadoop.yarn.server.resourcemanager.TestResourceManager   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId   hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication   hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart   hadoop.yarn.server.resourcemanager.TestRMAdminService   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765542/YARN-3964.016.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1107bd3 Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9377/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9377/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9377/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9377/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9377/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9377/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9377/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 20m 15s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
          +1 javac 8m 6s There were no new javac warning messages.
          +1 javadoc 10m 27s There were no new javadoc warning messages.
          -1 release audit 0m 19s The applied patch generated 1 release audit warnings.
          -1 checkstyle 2m 0s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 5s The patch has no lines that end in whitespace.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          +1 findbugs 4m 42s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 7s Tests passed in hadoop-yarn-common.
          +1 yarn tests 56m 52s Tests passed in hadoop-yarn-server-resourcemanager.
              107m 57s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12765542/YARN-3964.016.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 1107bd3
          Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9378/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9378/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9378/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9378/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9378/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9378/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9378/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 15s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 8m 6s There were no new javac warning messages. +1 javadoc 10m 27s There were no new javadoc warning messages. -1 release audit 0m 19s The applied patch generated 1 release audit warnings. -1 checkstyle 2m 0s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 5s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 4m 42s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 7s Tests passed in hadoop-yarn-common. +1 yarn tests 56m 52s Tests passed in hadoop-yarn-server-resourcemanager.     107m 57s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765542/YARN-3964.016.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1107bd3 Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9378/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9378/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9378/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9378/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9378/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9378/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9378/console This message was automatically generated.
          Hide
          devaraj.k Devaraj K added a comment -

          Thanks Dian Fu for the updated patch.

          Latest patch looks good to me. I will commit it tomorrow if there are no further comments/objections.

          Show
          devaraj.k Devaraj K added a comment - Thanks Dian Fu for the updated patch. Latest patch looks good to me. I will commit it tomorrow if there are no further comments/objections.
          Hide
          devaraj.k Devaraj K added a comment -

          +1, committing it shortly.

          Show
          devaraj.k Devaraj K added a comment - +1, committing it shortly.
          Hide
          devaraj.k Devaraj K added a comment -

          Thanks Dian Fu for the patch.
          Thanks Wangda Tan, Naganarasimha G R and Sunil G for reviews.

          Committed to trunk and branch-2.

          Show
          devaraj.k Devaraj K added a comment - Thanks Dian Fu for the patch. Thanks Wangda Tan , Naganarasimha G R and Sunil G for reviews. Committed to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8608 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8608/)
          YARN-3964. Support NodeLabelsProvider at Resource Manager side. (devaraj: rev db9304788187c700647c4d84caeb3b5ad6d868d8)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMDelegatedNodeLabelsUpdater.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMDelegatedNodeLabelsUpdater.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeLabelsUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsMappingProvider.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8608 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8608/ ) YARN-3964 . Support NodeLabelsProvider at Resource Manager side. (devaraj: rev db9304788187c700647c4d84caeb3b5ad6d868d8) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMDelegatedNodeLabelsUpdater.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMDelegatedNodeLabelsUpdater.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeLabelsUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsMappingProvider.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #510 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/510/)
          YARN-3964. Support NodeLabelsProvider at Resource Manager side. (devaraj: rev db9304788187c700647c4d84caeb3b5ad6d868d8)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeLabelsUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMDelegatedNodeLabelsUpdater.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsMappingProvider.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMDelegatedNodeLabelsUpdater.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #510 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/510/ ) YARN-3964 . Support NodeLabelsProvider at Resource Manager side. (devaraj: rev db9304788187c700647c4d84caeb3b5ad6d868d8) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeLabelsUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMDelegatedNodeLabelsUpdater.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsMappingProvider.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMDelegatedNodeLabelsUpdater.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #522 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/522/)
          YARN-3964. Support NodeLabelsProvider at Resource Manager side. (devaraj: rev db9304788187c700647c4d84caeb3b5ad6d868d8)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeLabelsUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsMappingProvider.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMDelegatedNodeLabelsUpdater.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMDelegatedNodeLabelsUpdater.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #522 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/522/ ) YARN-3964 . Support NodeLabelsProvider at Resource Manager side. (devaraj: rev db9304788187c700647c4d84caeb3b5ad6d868d8) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeLabelsUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsMappingProvider.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMDelegatedNodeLabelsUpdater.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMDelegatedNodeLabelsUpdater.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1247 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1247/)
          YARN-3964. Support NodeLabelsProvider at Resource Manager side. (devaraj: rev db9304788187c700647c4d84caeb3b5ad6d868d8)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsMappingProvider.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMDelegatedNodeLabelsUpdater.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMDelegatedNodeLabelsUpdater.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeLabelsUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1247 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1247/ ) YARN-3964 . Support NodeLabelsProvider at Resource Manager side. (devaraj: rev db9304788187c700647c4d84caeb3b5ad6d868d8) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsMappingProvider.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMDelegatedNodeLabelsUpdater.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMDelegatedNodeLabelsUpdater.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeLabelsUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2456 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2456/)
          YARN-3964. Support NodeLabelsProvider at Resource Manager side. (devaraj: rev db9304788187c700647c4d84caeb3b5ad6d868d8)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeLabelsUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMDelegatedNodeLabelsUpdater.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMDelegatedNodeLabelsUpdater.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsMappingProvider.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2456 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2456/ ) YARN-3964 . Support NodeLabelsProvider at Resource Manager side. (devaraj: rev db9304788187c700647c4d84caeb3b5ad6d868d8) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeLabelsUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMDelegatedNodeLabelsUpdater.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMDelegatedNodeLabelsUpdater.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsMappingProvider.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #482 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/482/)
          YARN-3964. Support NodeLabelsProvider at Resource Manager side. (devaraj: rev db9304788187c700647c4d84caeb3b5ad6d868d8)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMDelegatedNodeLabelsUpdater.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMDelegatedNodeLabelsUpdater.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsMappingProvider.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeLabelsUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #482 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/482/ ) YARN-3964 . Support NodeLabelsProvider at Resource Manager side. (devaraj: rev db9304788187c700647c4d84caeb3b5ad6d868d8) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMDelegatedNodeLabelsUpdater.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMDelegatedNodeLabelsUpdater.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsMappingProvider.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeLabelsUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2420 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2420/)
          YARN-3964. Support NodeLabelsProvider at Resource Manager side. (devaraj: rev db9304788187c700647c4d84caeb3b5ad6d868d8)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsMappingProvider.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMDelegatedNodeLabelsUpdater.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMDelegatedNodeLabelsUpdater.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeLabelsUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2420 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2420/ ) YARN-3964 . Support NodeLabelsProvider at Resource Manager side. (devaraj: rev db9304788187c700647c4d84caeb3b5ad6d868d8) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsMappingProvider.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMDelegatedNodeLabelsUpdater.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMDelegatedNodeLabelsUpdater.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NodeLabelsUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodeLabels.java hadoop-yarn-project/CHANGES.txt

            People

            • Assignee:
              dian.fu Dian Fu
              Reporter:
              dian.fu Dian Fu
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development