Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4176

Resync NM nodelabels with RM periodically for distributed nodelabels

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This JIRA is for handling the below set of issue

      1. Distributed nodelabels after NM registered with RM if cluster nodelabels are removed and added then NM doesnt resend labels in heartbeat again untils any change in labels
      2. NM registration failed with Nodelabels should resend labels again to RM

      The above cases can be handled by resync nodeLabels with RM every x interval

      1. Add property yarn.nodemanager.node-labels.provider.resync-interval-ms and will resend nodelabels to RM based on config no matter what the registration fails or success.
      1. 0005-YARN-4176.patch
        11 kB
        Bibin A Chundatt
      2. 0004-YARN-4176.patch
        11 kB
        Bibin A Chundatt
      3. 0003-YARN-4176.patch
        9 kB
        Bibin A Chundatt
      4. 0002-YARN-4176.patch
        9 kB
        Bibin A Chundatt
      5. 0001-YARN-4176.patch
        7 kB
        Bibin A Chundatt

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #458 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/458/)
          YARN-4176. Resync NM nodelabels with RM periodically for distributed (wangda: rev 30ac69c6bd3db363248d6c742561371576006dab)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #458 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/458/ ) YARN-4176 . Resync NM nodelabels with RM periodically for distributed (wangda: rev 30ac69c6bd3db363248d6c742561371576006dab) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2397 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2397/)
          YARN-4176. Resync NM nodelabels with RM periodically for distributed (wangda: rev 30ac69c6bd3db363248d6c742561371576006dab)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2397 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2397/ ) YARN-4176 . Resync NM nodelabels with RM periodically for distributed (wangda: rev 30ac69c6bd3db363248d6c742561371576006dab) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2427 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2427/)
          YARN-4176. Resync NM nodelabels with RM periodically for distributed (wangda: rev 30ac69c6bd3db363248d6c742561371576006dab)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2427 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2427/ ) YARN-4176 . Resync NM nodelabels with RM periodically for distributed (wangda: rev 30ac69c6bd3db363248d6c742561371576006dab) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #1222 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1222/)
          YARN-4176. Resync NM nodelabels with RM periodically for distributed (wangda: rev 30ac69c6bd3db363248d6c742561371576006dab)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1222 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1222/ ) YARN-4176 . Resync NM nodelabels with RM periodically for distributed (wangda: rev 30ac69c6bd3db363248d6c742561371576006dab) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #492 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/492/)
          YARN-4176. Resync NM nodelabels with RM periodically for distributed (wangda: rev 30ac69c6bd3db363248d6c742561371576006dab)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #492 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/492/ ) YARN-4176 . Resync NM nodelabels with RM periodically for distributed (wangda: rev 30ac69c6bd3db363248d6c742561371576006dab) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #483 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/483/)
          YARN-4176. Resync NM nodelabels with RM periodically for distributed (wangda: rev 30ac69c6bd3db363248d6c742561371576006dab)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #483 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/483/ ) YARN-4176 . Resync NM nodelabels with RM periodically for distributed (wangda: rev 30ac69c6bd3db363248d6c742561371576006dab) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8573 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8573/)
          YARN-4176. Resync NM nodelabels with RM periodically for distributed (wangda: rev 30ac69c6bd3db363248d6c742561371576006dab)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8573 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8573/ ) YARN-4176 . Resync NM nodelabels with RM periodically for distributed (wangda: rev 30ac69c6bd3db363248d6c742561371576006dab) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Hide
          leftnoteasy Wangda Tan added a comment -

          Committed to trunk/branch-2, thanks Bibin A Chundatt and review from Naganarasimha G R/Steve Loughran!

          Show
          leftnoteasy Wangda Tan added a comment - Committed to trunk/branch-2, thanks Bibin A Chundatt and review from Naganarasimha G R / Steve Loughran !
          Hide
          leftnoteasy Wangda Tan added a comment -

          Latest patch LGTM, committing..

          Show
          leftnoteasy Wangda Tan added a comment - Latest patch LGTM, committing..
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Tan, Wangda and Bibin A Chundatt,
          Can we just hold on to this getting committed as we need to see the cluster Node Labels being removed holistically including changes in YARN-3964.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Tan, Wangda and Bibin A Chundatt , Can we just hold on to this getting committed as we need to see the cluster Node Labels being removed holistically including changes in YARN-3964 .
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Hi Wangda Tan

          YARN-4106 the Timer related fix for loading conf every interval is done.
          So this jira cann't completely replace the same.

          Only the heartbeat resend is common between both the jira's. YARN-4106 only failed we used to handle in this failed or success we are sending label along with heartbeat.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Hi Wangda Tan YARN-4106 the Timer related fix for loading conf every interval is done. So this jira cann't completely replace the same. Only the heartbeat resend is common between both the jira's. YARN-4106 only failed we used to handle in this failed or success we are sending label along with heartbeat.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Bibin A Chundatt, one question when looking at the patch, do you think is this patch can completely replace what YARN-4106 can do. If so, is it make sense to revert YARN-4106?

          Show
          leftnoteasy Wangda Tan added a comment - Bibin A Chundatt , one question when looking at the patch, do you think is this patch can completely replace what YARN-4106 can do. If so, is it make sense to revert YARN-4106 ?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for changes Bibin A Chundatt,
          Patch lgtm, +1.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for changes Bibin A Chundatt , Patch lgtm, +1.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          hadoop.yarn.logaggregation.TestAggregatedLogsBlock test failure is not related to patch attached.Checkstyle due to lines more than 2 K

          Show
          bibinchundatt Bibin A Chundatt added a comment - hadoop.yarn.logaggregation.TestAggregatedLogsBlock test failure is not related to patch attached.Checkstyle due to lines more than 2 K
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 32s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 53s There were no new javac warning messages.
          +1 javadoc 10m 6s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 55s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 29s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api.
          -1 yarn tests 1m 58s Tests failed in hadoop-yarn-common.
          +1 yarn tests 7m 58s Tests passed in hadoop-yarn-server-nodemanager.
              57m 23s  



          Reason Tests
          Failed unit tests hadoop.yarn.logaggregation.TestAggregatedLogsBlock



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12761859/0005-YARN-4176.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a2c76e5
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9239/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9239/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9239/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9239/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9239/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9239/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 32s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 53s There were no new javac warning messages. +1 javadoc 10m 6s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 55s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api. -1 yarn tests 1m 58s Tests failed in hadoop-yarn-common. +1 yarn tests 7m 58s Tests passed in hadoop-yarn-server-nodemanager.     57m 23s   Reason Tests Failed unit tests hadoop.yarn.logaggregation.TestAggregatedLogsBlock Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761859/0005-YARN-4176.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a2c76e5 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9239/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9239/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9239/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9239/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9239/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9239/console This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Thanks Naganarasimha G R for comments

          increase this sync duration to a greater value like 10 mins.

          The udpate interval is 10 mins so resync should be less than 10.

          All other comments have handled

          Show
          bibinchundatt Bibin A Chundatt added a comment - Thanks Naganarasimha G R for comments increase this sync duration to a greater value like 10 mins. The udpate interval is 10 mins so resync should be less than 10. All other comments have handled
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Bibin A Chundatt for working on this issue, few small nits remaining :

          1. nodeLabelsProvider and resyncInterval can be final
          2. lastNodeLabelSendMills needs to be set even in case validation fails this will avoid invalid labels to be validated for every heartbeat
          3. In TestNodeStatusUpdaterForLabels, dummyLabelsProviderRef.setNodeLabels(toNodeLabelSet("P1")); can be outside the for loop and assertNull("Null resourcelabels are expected", resourceTracker.labels); is not required in side the if check.
          4. Modify updatedLabelsSentToRM -> areLabelsSentToRM
          5. instead of logging on every resync in verifyRMHeartbeatResponseForNodeLabels we can either make it debug log or based on labels update/RM reject we can log. I prefer for the former approach.

          Apart from this every thing else is fine and have tested the patch in the cluster
          Tan, Wangda we have kept the sync interval as 2 mins so that when cluster nodelabels are added, NM HB can update latest labels fast, please share weather we need to increase this sync duration to a greater value like 10 mins.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Bibin A Chundatt for working on this issue, few small nits remaining : nodeLabelsProvider and resyncInterval can be final lastNodeLabelSendMills needs to be set even in case validation fails this will avoid invalid labels to be validated for every heartbeat In TestNodeStatusUpdaterForLabels, dummyLabelsProviderRef.setNodeLabels(toNodeLabelSet("P1")); can be outside the for loop and assertNull("Null resourcelabels are expected", resourceTracker.labels); is not required in side the if check. Modify updatedLabelsSentToRM -> areLabelsSentToRM instead of logging on every resync in verifyRMHeartbeatResponseForNodeLabels we can either make it debug log or based on labels update/RM reject we can log. I prefer for the former approach. Apart from this every thing else is fine and have tested the patch in the cluster Tan, Wangda we have kept the sync interval as 2 mins so that when cluster nodelabels are added, NM HB can update latest labels fast, please share weather we need to increase this sync duration to a greater value like 10 mins.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Hi Wangda Tan

          Could you please look into this issue

          Show
          bibinchundatt Bibin A Chundatt added a comment - Hi Wangda Tan Could you please look into this issue
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Checkstyle is due to number of lines

          File length is 2,146 lines (max allowed is 2,000).
          

          I feel can be skipped as already the number of lines were greater than 2K

          Show
          bibinchundatt Bibin A Chundatt added a comment - Checkstyle is due to number of lines File length is 2,146 lines (max allowed is 2,000). I feel can be skipped as already the number of lines were greater than 2K
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 9s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 51s There were no new javac warning messages.
          +1 javadoc 10m 8s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 50s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common.
          +1 yarn tests 7m 48s Tests passed in hadoop-yarn-server-nodemanager.
              56m 40s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12761395/0004-YARN-4176.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c9cb6a5
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9228/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9228/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9228/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9228/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9228/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9228/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 9s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 51s There were no new javac warning messages. +1 javadoc 10m 8s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 50s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common. +1 yarn tests 7m 48s Tests passed in hadoop-yarn-server-nodemanager.     56m 40s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761395/0004-YARN-4176.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c9cb6a5 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9228/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9228/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9228/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9228/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9228/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9228/console This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Hi Naganarasimha G R
          Thanks for review comments

          Attaching patch after handling the same.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Hi Naganarasimha G R Thanks for review comments Attaching patch after handling the same.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Bibin A Chundatt,
          Thanks for working on the patch. Few comments :
          NodeStatusUpdaterImpl.java

          1. diffcheckstylehadoop-yarn-server-nodemanager.txt seems like issues reported here are related to the modifications of the patch
          2. areNodeLabelsUpdated || resyncElapsed could be areNodeLabelsUpdated || isResyncIntervalElapsed(), In the cases where node labels are updated then due to short circuit isResyncIntervalElapsed will be avoided
          3. in isResyncIntervalElapsed resyncInterval need not be evaluated on every call we can push this to constructor or init
          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Bibin A Chundatt , Thanks for working on the patch. Few comments : NodeStatusUpdaterImpl.java diffcheckstylehadoop-yarn-server-nodemanager.txt seems like issues reported here are related to the modifications of the patch areNodeLabelsUpdated || resyncElapsed could be areNodeLabelsUpdated || isResyncIntervalElapsed() , In the cases where node labels are updated then due to short circuit isResyncIntervalElapsed will be avoided in isResyncIntervalElapsed resyncInterval need not be evaluated on every call we can push this to constructor or init
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Checkstyle issues not added as part of this patch to my understanding

          Show
          bibinchundatt Bibin A Chundatt added a comment - Checkstyle issues not added as part of this patch to my understanding
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 10s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 47s There were no new javac warning messages.
          +1 javadoc 10m 1s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          -1 checkstyle 2m 36s The applied patch generated 4 new checkstyle issues (total was 25, now 29).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 4m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common.
          +1 yarn tests 7m 46s Tests passed in hadoop-yarn-server-nodemanager.
              56m 33s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12761332/0003-YARN-4176.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 3a9c707
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9222/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/9222/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9222/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9222/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9222/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9222/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9222/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 10s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 47s There were no new javac warning messages. +1 javadoc 10m 1s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 211, now 211). -1 checkstyle 2m 36s The applied patch generated 4 new checkstyle issues (total was 25, now 29). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 4m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common. +1 yarn tests 7m 46s Tests passed in hadoop-yarn-server-nodemanager.     56m 33s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761332/0003-YARN-4176.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3a9c707 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9222/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/9222/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9222/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9222/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9222/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9222/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9222/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 15s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 54s There were no new javac warning messages.
          +1 javadoc 9m 58s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 50s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          +1 findbugs 4m 20s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 0s Tests passed in hadoop-yarn-common.
          -1 yarn tests 8m 15s Tests failed in hadoop-yarn-server-nodemanager.
              57m 7s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.nodemanager.TestDockerContainerExecutor
            hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12761329/0002-YARN-4176.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 3a9c707
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9221/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9221/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9221/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9221/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9221/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9221/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 15s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 54s There were no new javac warning messages. +1 javadoc 9m 58s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 50s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 4m 20s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 0s Tests passed in hadoop-yarn-common. -1 yarn tests 8m 15s Tests failed in hadoop-yarn-server-nodemanager.     57m 7s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.TestDockerContainerExecutor   hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761329/0002-YARN-4176.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3a9c707 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9221/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9221/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9221/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9221/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9221/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9221/console This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching patch after testcase correction.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching patch after testcase correction.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 11s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 8s There were no new javac warning messages.
          +1 javadoc 10m 24s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 25s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 yarn tests 0m 23s Tests failed in hadoop-yarn-api.
          -1 yarn tests 7m 53s Tests failed in hadoop-yarn-server-nodemanager.
              52m 5s  



          Reason Tests
          Failed unit tests hadoop.yarn.conf.TestYarnConfigurationFields
            hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12761291/0001-YARN-4176.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c39ddc3
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9220/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9220/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9220/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9220/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9220/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 11s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 8s There were no new javac warning messages. +1 javadoc 10m 24s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 25s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 0m 23s Tests failed in hadoop-yarn-api. -1 yarn tests 7m 53s Tests failed in hadoop-yarn-server-nodemanager.     52m 5s   Reason Tests Failed unit tests hadoop.yarn.conf.TestYarnConfigurationFields   hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12761291/0001-YARN-4176.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c39ddc3 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9220/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9220/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9220/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9220/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9220/console This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attaching patch for review.

          Naganarasimha G R

          1. Have taken care of configuration naming .
          2. Haven't changed Syste.mills as of now.
          Show
          bibinchundatt Bibin A Chundatt added a comment - Attaching patch for review. Naganarasimha G R Have taken care of configuration naming . Haven't changed Syste.mills as of now.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Steve Loughran,
          Thanks for the comments, but based on the earlier comment

          Clock-wise, how about adding a new method, `monotonicTimeMillis()`, which is just nanoTime/1e6; easy to switch from one to the other.

          i thought by nanoTime/1e6 you meant System.nanoTime()/1000000, which is similar to the modification which you had asked to refer but if that also is inconsistent, then any other option ?

          Bibin A Chundatt,
          Missed one more point let the interval configuration name be yarn.nodemanager.node-labels.resync-interval-ms , as configurations after provider was used only for config and label based provider.

          Show
          Naganarasimha Naganarasimha G R added a comment - Steve Loughran , Thanks for the comments, but based on the earlier comment Clock-wise, how about adding a new method, `monotonicTimeMillis()`, which is just nanoTime/1e6; easy to switch from one to the other. i thought by nanoTime/1e6 you meant System.nanoTime()/1000000 , which is similar to the modification which you had asked to refer but if that also is inconsistent, then any other option ? Bibin A Chundatt , Missed one more point let the interval configuration name be yarn.nodemanager.node-labels.resync-interval-ms , as configurations after provider was used only for config and label based provider.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I no longer trust nanoTime(). I'll do a blog post on it, but the summary is: on multi-core/multi-socket systems you may get either inconsistent results or time data from a clock that is even less granular than getTimeMillis

          Show
          stevel@apache.org Steve Loughran added a comment - I no longer trust nanoTime(). I'll do a blog post on it, but the summary is: on multi-core/multi-socket systems you may get either inconsistent results or time data from a clock that is even less granular than getTimeMillis
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Hi Naganarasimha G R
          Thnks for the comments.

          I would suggest only to have only 1 resync configuration and remove what we introduced for YARN-4106.

          Will be taken care

          So node labels will be sent to RM either if node labels are modified since the last heartbeat or the resync-interval has elapsed right ?

          Currently on hearbeat we are sending only when update happens. Will be or of update & resync.
          Changing to System.nanoTime() will handle too.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Hi Naganarasimha G R Thnks for the comments. I would suggest only to have only 1 resync configuration and remove what we introduced for YARN-4106 . Will be taken care So node labels will be sent to RM either if node labels are modified since the last heartbeat or the resync-interval has elapsed right ? Currently on hearbeat we are sending only when update happens. Will be or of update & resync. Changing to System.nanoTime() will handle too.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Bibin A Chundatt,
          Seems like this would be a better idea than what we did in YARN-4106, where in we used time interval of 1 min only on NM side failure. But i have few concerns/queries :

          1. I would suggest only to have only 1 resync configuration and remove what we introduced for YARN-4106.
          2. So node labels will be sent to RM either if node labels are modified since the last heartbeat or the resync-interval has elapsed right ?
          3. Earlier the way used to check the time elapse is using System.currentTimeMillis(), but i think we need to use the approach as mentioned by Steve Loughran in the comment of the jira HADOOP-12409. Hope Xianyin Xin creates a new jira and provide a clock with monotonic time by then, if not use System.nanoTime().
          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Bibin A Chundatt , Seems like this would be a better idea than what we did in YARN-4106 , where in we used time interval of 1 min only on NM side failure. But i have few concerns/queries : I would suggest only to have only 1 resync configuration and remove what we introduced for YARN-4106 . So node labels will be sent to RM either if node labels are modified since the last heartbeat or the resync-interval has elapsed right ? Earlier the way used to check the time elapse is using System.currentTimeMillis() , but i think we need to use the approach as mentioned by Steve Loughran in the comment of the jira HADOOP-12409 . Hope Xianyin Xin creates a new jira and provide a clock with monotonic time by then, if not use System.nanoTime() .

            People

            • Assignee:
              bibinchundatt Bibin A Chundatt
              Reporter:
              bibinchundatt Bibin A Chundatt
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development