Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3826

Race condition in ResourceTrackerService leads to wrong diagnostics messages

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: resourcemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Since we are calling setDiagnosticsMessage in nodeHeartbeat, which can be called concurrently, the static resync and shutdown may have wrong diagnostics messages in some cases.

      On the other side, these static members can hardly save any memory, since the normal heartbeat responses are created for each heartbeat.

      1. YARN-3826.03.patch
        5 kB
        Devaraj K
      2. YARN-3826.02.patch
        5 kB
        Chengbing Liu
      3. YARN-3826.01.patch
        3 kB
        Chengbing Liu

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #970 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/970/)
        YARN-3826. Race condition in ResourceTrackerService leads to wrong (devaraj: rev 57f1a01eda80f44d3ffcbcb93c4ee290e274946a)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerBuilderUtils.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #970 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/970/ ) YARN-3826 . Race condition in ResourceTrackerService leads to wrong (devaraj: rev 57f1a01eda80f44d3ffcbcb93c4ee290e274946a) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerBuilderUtils.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #240 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/240/)
        YARN-3826. Race condition in ResourceTrackerService leads to wrong (devaraj: rev 57f1a01eda80f44d3ffcbcb93c4ee290e274946a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerBuilderUtils.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #240 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/240/ ) YARN-3826 . Race condition in ResourceTrackerService leads to wrong (devaraj: rev 57f1a01eda80f44d3ffcbcb93c4ee290e274946a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerBuilderUtils.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2185 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2185/)
        YARN-3826. Race condition in ResourceTrackerService leads to wrong (devaraj: rev 57f1a01eda80f44d3ffcbcb93c4ee290e274946a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerBuilderUtils.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2185 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2185/ ) YARN-3826 . Race condition in ResourceTrackerService leads to wrong (devaraj: rev 57f1a01eda80f44d3ffcbcb93c4ee290e274946a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerBuilderUtils.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #237 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/237/)
        YARN-3826. Race condition in ResourceTrackerService leads to wrong (devaraj: rev 57f1a01eda80f44d3ffcbcb93c4ee290e274946a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerBuilderUtils.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #237 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/237/ ) YARN-3826 . Race condition in ResourceTrackerService leads to wrong (devaraj: rev 57f1a01eda80f44d3ffcbcb93c4ee290e274946a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerBuilderUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #228 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/228/)
        YARN-3826. Race condition in ResourceTrackerService leads to wrong (devaraj: rev 57f1a01eda80f44d3ffcbcb93c4ee290e274946a)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerBuilderUtils.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #228 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/228/ ) YARN-3826 . Race condition in ResourceTrackerService leads to wrong (devaraj: rev 57f1a01eda80f44d3ffcbcb93c4ee290e274946a) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerBuilderUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2167 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2167/)
        YARN-3826. Race condition in ResourceTrackerService leads to wrong (devaraj: rev 57f1a01eda80f44d3ffcbcb93c4ee290e274946a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerBuilderUtils.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2167 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2167/ ) YARN-3826 . Race condition in ResourceTrackerService leads to wrong (devaraj: rev 57f1a01eda80f44d3ffcbcb93c4ee290e274946a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerBuilderUtils.java hadoop-yarn-project/CHANGES.txt
        Hide
        chengbing.liu Chengbing Liu added a comment -

        Thanks Devaraj K for review and committing!

        Show
        chengbing.liu Chengbing Liu added a comment - Thanks Devaraj K for review and committing!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8065 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8065/)
        YARN-3826. Race condition in ResourceTrackerService leads to wrong (devaraj: rev 57f1a01eda80f44d3ffcbcb93c4ee290e274946a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerBuilderUtils.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8065 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8065/ ) YARN-3826 . Race condition in ResourceTrackerService leads to wrong (devaraj: rev 57f1a01eda80f44d3ffcbcb93c4ee290e274946a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerBuilderUtils.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
        Hide
        devaraj.k Devaraj K added a comment -

        Thanks Chengbing Liu.

        Committed to trunk and branch-2.

        Show
        devaraj.k Devaraj K added a comment - Thanks Chengbing Liu . Committed to trunk and branch-2.
        Hide
        devaraj.k Devaraj K added a comment -

        The timed out test is not related to the patch.

        +1, will commit it shortly.

        Show
        devaraj.k Devaraj K added a comment - The timed out test is not related to the patch. +1, will commit it shortly.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 16m 46s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 35s There were no new javac warning messages.
        +1 javadoc 9m 35s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 14s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 2m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common.
        -1 yarn tests 61m 0s Tests failed in hadoop-yarn-server-resourcemanager.
            101m 33s  



        Reason Tests
        Timed out tests org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12741817/YARN-3826.03.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / a815cc1
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8342/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
        hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8342/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8342/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8342/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8342/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 16m 46s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 14s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common. -1 yarn tests 61m 0s Tests failed in hadoop-yarn-server-resourcemanager.     101m 33s   Reason Tests Timed out tests org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12741817/YARN-3826.03.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a815cc1 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8342/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8342/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8342/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8342/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8342/console This message was automatically generated.
        Hide
        devaraj.k Devaraj K added a comment -

        Thanks Chengbing Liu for the patch.

        There are lines exceeding max line width(>80 characters), I am attaching the same patch with max line width fix.

        Show
        devaraj.k Devaraj K added a comment - Thanks Chengbing Liu for the patch. There are lines exceeding max line width(>80 characters), I am attaching the same patch with max line width fix.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 17m 1s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 36s There were no new javac warning messages.
        +1 javadoc 9m 34s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 0s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 36s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 2m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common.
        +1 yarn tests 50m 52s Tests passed in hadoop-yarn-server-resourcemanager.
            91m 29s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12741774/YARN-3826.02.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / a815cc1
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8337/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
        hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8337/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8337/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8337/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8337/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 1s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 0s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common. +1 yarn tests 50m 52s Tests passed in hadoop-yarn-server-resourcemanager.     91m 29s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12741774/YARN-3826.02.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a815cc1 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8337/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8337/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8337/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8337/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8337/console This message was automatically generated.
        Hide
        chengbing.liu Chengbing Liu added a comment -

        Sorry for the delay. I'll submit a new patch shortly.

        Show
        chengbing.liu Chengbing Liu added a comment - Sorry for the delay. I'll submit a new patch shortly.
        Hide
        devaraj.k Devaraj K added a comment -

        Chengbing Liu, Did you get any chance to look into the above comment on the patch? Thanks

        Show
        devaraj.k Devaraj K added a comment - Chengbing Liu , Did you get any chance to look into the above comment on the patch? Thanks
        Hide
        devaraj.k Devaraj K added a comment -

        Chengbing Liu, patch looks good to me overall except this comment.

        The similar code is duplicated in three places, can you extract as a method something like below and invoke from those 3 places.

          private NodeHeartbeatResponse getResponse(NodeAction action, String message) {
            NodeHeartbeatResponse response = recordFactory
                .newRecordInstance(NodeHeartbeatResponse.class);
            response.setNodeAction(action);
            response.setDiagnosticsMessage(message);
            return response;
          }
        
        Show
        devaraj.k Devaraj K added a comment - Chengbing Liu , patch looks good to me overall except this comment. The similar code is duplicated in three places, can you extract as a method something like below and invoke from those 3 places. private NodeHeartbeatResponse getResponse(NodeAction action, String message) { NodeHeartbeatResponse response = recordFactory .newRecordInstance(NodeHeartbeatResponse.class); response.setNodeAction(action); response.setDiagnosticsMessage(message); return response; }
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 32s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 30s There were no new javac warning messages.
        +1 javadoc 9m 35s There were no new javadoc warning messages.
        +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 48s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 35s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 1m 24s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 50m 43s Tests failed in hadoop-yarn-server-resourcemanager.
            89m 7s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12740355/YARN-3826.01.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 445b132
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8308/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8308/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8308/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 32s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 48s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 24s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 50m 43s Tests failed in hadoop-yarn-server-resourcemanager.     89m 7s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740355/YARN-3826.01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 445b132 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8308/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8308/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8308/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 19m 42s Findbugs (version ) appears to be broken on trunk.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 9m 30s There were no new javac warning messages.
        +1 javadoc 10m 42s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 28s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 40s mvn install still works.
        +1 eclipse:eclipse 0m 44s The patch built with eclipse:eclipse.
        +1 findbugs 1m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 61m 36s Tests failed in hadoop-yarn-server-resourcemanager.
            106m 18s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart
        Timed out tests org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12740355/YARN-3826.01.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 6c7a9d5
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8306/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8306/testReport/
        Java 1.7.0_55
        uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8306/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 19m 42s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 9m 30s There were no new javac warning messages. +1 javadoc 10m 42s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 28s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 40s mvn install still works. +1 eclipse:eclipse 0m 44s The patch built with eclipse:eclipse. +1 findbugs 1m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 61m 36s Tests failed in hadoop-yarn-server-resourcemanager.     106m 18s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart Timed out tests org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740355/YARN-3826.01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6c7a9d5 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8306/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8306/testReport/ Java 1.7.0_55 uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8306/console This message was automatically generated.
        Hide
        devaraj.k Devaraj K added a comment -

        Thanks Chengbing Liu for details.

        Show
        devaraj.k Devaraj K added a comment - Thanks Chengbing Liu for details.
        Hide
        chengbing.liu Chengbing Liu added a comment -

        Devaraj K, is it possible that two heartbeats happen at the same time? Because the following code

              shutDown.setDiagnosticsMessage(message);
              return shutDown;
        

        is not atomic, race can happen when two threads run to this point simultaneously.

        Show
        chengbing.liu Chengbing Liu added a comment - Devaraj K , is it possible that two heartbeats happen at the same time? Because the following code shutDown.setDiagnosticsMessage(message); return shutDown; is not atomic, race can happen when two threads run to this point simultaneously.
        Hide
        devaraj.k Devaraj K added a comment -

        Thanks Chengbing Liu for the patch.

        Since we are calling setDiagnosticsMessage in nodeHeartbeat, which can be called concurrently, the static resync and shutdown may have wrong diagnostics messages in some cases.

        I see diagnostics message is setting into resync/shutdown response objects before returning them. Can you give more details about a case where race condition could occur?

        these static members can hardly save any memory, since the normal heartbeat responses are created for each heartbeat.

        I agree on this.

        Show
        devaraj.k Devaraj K added a comment - Thanks Chengbing Liu for the patch. Since we are calling setDiagnosticsMessage in nodeHeartbeat, which can be called concurrently, the static resync and shutdown may have wrong diagnostics messages in some cases. I see diagnostics message is setting into resync/shutdown response objects before returning them. Can you give more details about a case where race condition could occur? these static members can hardly save any memory, since the normal heartbeat responses are created for each heartbeat. I agree on this.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 15m 55s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 34s There were no new javac warning messages.
        +1 javadoc 9m 41s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 46s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 35s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 50m 49s Tests failed in hadoop-yarn-server-resourcemanager.
            88m 43s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens
          hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12740355/YARN-3826.01.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 1c13519
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8285/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8285/testReport/
        Java 1.7.0_55
        uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8285/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 55s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 46s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 50m 49s Tests failed in hadoop-yarn-server-resourcemanager.     88m 43s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens   hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740355/YARN-3826.01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1c13519 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8285/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8285/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8285/console This message was automatically generated.
        Hide
        chengbing.liu Chengbing Liu added a comment -

        Patch uploaded.

        Show
        chengbing.liu Chengbing Liu added a comment - Patch uploaded.

          People

          • Assignee:
            chengbing.liu Chengbing Liu
            Reporter:
            chengbing.liu Chengbing Liu
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development