Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4756

Unnecessary wait in Node Status Updater during reboot

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The startStatusUpdater thread waits for the isStopped variable to be set to true, but it is waiting for the next heartbeat. During a reboot, the next heartbeat will not come and so the thread waits for a timeout. Instead, we should notify the thread to continue so that it can check the isStopped variable and exit without having to wait for a timeout.

      1. YARN-4756.001.patch
        5 kB
        Eric Badger
      2. YARN-4756.002.patch
        5 kB
        Eric Badger
      3. YARN-4756.003.patch
        5 kB
        Eric Badger
      4. YARN-4756.004.patch
        6 kB
        Eric Badger
      5. YARN-4756.005.patch
        5 kB
        Eric Badger

        Activity

        Hide
        ebadger Eric Badger added a comment -

        The optimization to notify the Node Status Updater thread to stop waiting for a heartbeat exposes a race condition in the test TestNodeManagerResync#testContainerResourceIncreaseIsSynchronizedWithRMResync. The test checks the current resources of the NM, then checks for it again since a different thread changes the current resources. However, there is no synchronization between these threads and it was only working because of the excessive wait time from the reboot. The patch adds in a barrier to synchronize these two threads.

        Show
        ebadger Eric Badger added a comment - The optimization to notify the Node Status Updater thread to stop waiting for a heartbeat exposes a race condition in the test TestNodeManagerResync#testContainerResourceIncreaseIsSynchronizedWithRMResync. The test checks the current resources of the NM, then checks for it again since a different thread changes the current resources. However, there is no synchronization between these threads and it was only working because of the excessive wait time from the reboot. The patch adds in a barrier to synchronize these two threads.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 4s YARN-4756 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791229/YARN-4756.001.patch
        JIRA Issue YARN-4756
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10700/console
        Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s YARN-4756 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791229/YARN-4756.001.patch JIRA Issue YARN-4756 Console output https://builds.apache.org/job/PreCommit-YARN-Build/10700/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ebadger Eric Badger added a comment -

        Fixing patch so that it applies to trunk instead of being dependent on the YARN-4686 patch.

        Show
        ebadger Eric Badger added a comment - Fixing patch so that it applies to trunk instead of being dependent on the YARN-4686 patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 49s trunk passed
        +1 compile 0m 22s trunk passed with JDK v1.8.0_72
        +1 compile 0m 25s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 28s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 52s trunk passed
        +1 javadoc 0m 17s trunk passed with JDK v1.8.0_72
        +1 javadoc 0m 22s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 24s the patch passed
        +1 compile 0m 19s the patch passed with JDK v1.8.0_72
        +1 javac 0m 19s the patch passed
        +1 compile 0m 23s the patch passed with JDK v1.7.0_95
        +1 javac 0m 23s the patch passed
        +1 checkstyle 0m 14s the patch passed
        +1 mvnsite 0m 26s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 2s the patch passed
        +1 javadoc 0m 15s the patch passed with JDK v1.8.0_72
        +1 javadoc 0m 19s the patch passed with JDK v1.7.0_95
        +1 unit 9m 3s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_72.
        +1 unit 9m 35s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 19s Patch does not generate ASF License warnings.
        33m 40s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791244/YARN-4756.002.patch
        JIRA Issue YARN-4756
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux bb8f2e3e7d26 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0a9f00a
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10701/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10701/console
        Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 49s trunk passed +1 compile 0m 22s trunk passed with JDK v1.8.0_72 +1 compile 0m 25s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 52s trunk passed +1 javadoc 0m 17s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 22s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 24s the patch passed +1 compile 0m 19s the patch passed with JDK v1.8.0_72 +1 javac 0m 19s the patch passed +1 compile 0m 23s the patch passed with JDK v1.7.0_95 +1 javac 0m 23s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 2s the patch passed +1 javadoc 0m 15s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 19s the patch passed with JDK v1.7.0_95 +1 unit 9m 3s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_72. +1 unit 9m 35s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 33m 40s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791244/YARN-4756.002.patch JIRA Issue YARN-4756 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bb8f2e3e7d26 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0a9f00a Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10701/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10701/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ebadger Eric Badger added a comment -

        Eric Payne, Jason Lowe, Karthik Kambatla, can one of you review this patch when you are available? This patch is a byproduct of test failures that were uncovered when fixing YARN-4686. Thanks.

        Show
        ebadger Eric Badger added a comment - Eric Payne , Jason Lowe , Karthik Kambatla , can one of you review this patch when you are available? This patch is a byproduct of test failures that were uncovered when fixing YARN-4686 . Thanks.
        Hide
        kasha Karthik Kambatla added a comment -

        The patch doesn't apply anymore. Can we update this?

        Also, we should avoid putting changes into trunk and branch-* when we know the patch is breaking tests.

        Show
        kasha Karthik Kambatla added a comment - The patch doesn't apply anymore. Can we update this? Also, we should avoid putting changes into trunk and branch-* when we know the patch is breaking tests.
        Hide
        ebadger Eric Badger added a comment -

        Karthik Kambatla, I wasn't clear in my original text. The patches in YARN-4686 do not break any extra tests. However, while exploring the fixes for those failures, I came across an unnecessary wait in the NodeStatusUpdater thread, NodeStatusUpdaterImpl:850. When a reboot happens, the isStopped variable is set to true, but the thread waits until the next heartbeat. The next heartbeat won't come and so it will wait for a heartbeat timeout. So instead of wasting this time unnecessarily, I added a notify to wake the thread up and let it know to continue in the loop, where it would find that isStopped is set to true.

        Adding in this optimization uncovered a race condition in the TestNodeManagerResync test. The test doesn't wait for the NM to completely reboot before it checks for its updated capabilities. The only reason that it worked before is because the unnecessary wait in the NodeStatusUpdater acted as a sleep that masked the race condition.

        I'm uploading a patch that removes the unnecessary wait in the NodeStatusUpdater thread and also fixes the race condition in TestNodeManagerResync that it uncovers.

        Show
        ebadger Eric Badger added a comment - Karthik Kambatla , I wasn't clear in my original text. The patches in YARN-4686 do not break any extra tests. However, while exploring the fixes for those failures, I came across an unnecessary wait in the NodeStatusUpdater thread, NodeStatusUpdaterImpl:850. When a reboot happens, the isStopped variable is set to true, but the thread waits until the next heartbeat. The next heartbeat won't come and so it will wait for a heartbeat timeout. So instead of wasting this time unnecessarily, I added a notify to wake the thread up and let it know to continue in the loop, where it would find that isStopped is set to true. Adding in this optimization uncovered a race condition in the TestNodeManagerResync test. The test doesn't wait for the NM to completely reboot before it checks for its updated capabilities. The only reason that it worked before is because the unnecessary wait in the NodeStatusUpdater acted as a sleep that masked the race condition. I'm uploading a patch that removes the unnecessary wait in the NodeStatusUpdater thread and also fixes the race condition in TestNodeManagerResync that it uncovers.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 43s trunk passed
        +1 compile 0m 22s trunk passed with JDK v1.8.0_74
        +1 compile 0m 26s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 28s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 0m 48s trunk passed
        +1 javadoc 0m 18s trunk passed with JDK v1.8.0_74
        +1 javadoc 0m 21s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 24s the patch passed
        +1 compile 0m 20s the patch passed with JDK v1.8.0_74
        +1 javac 0m 20s the patch passed
        +1 compile 0m 23s the patch passed with JDK v1.7.0_95
        +1 javac 0m 23s the patch passed
        +1 checkstyle 0m 13s the patch passed
        +1 mvnsite 0m 26s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 findbugs 1m 1s the patch passed
        +1 javadoc 0m 16s the patch passed with JDK v1.8.0_74
        +1 javadoc 0m 19s the patch passed with JDK v1.7.0_95
        +1 unit 8m 58s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74.
        +1 unit 9m 32s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 17s Patch does not generate ASF License warnings.
        33m 18s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794627/YARN-4756.003.patch
        JIRA Issue YARN-4756
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 682b9251c430 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e7ed05e
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10841/artifact/patchprocess/whitespace-eol.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10841/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10841/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 43s trunk passed +1 compile 0m 22s trunk passed with JDK v1.8.0_74 +1 compile 0m 26s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 48s trunk passed +1 javadoc 0m 18s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 21s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 24s the patch passed +1 compile 0m 20s the patch passed with JDK v1.8.0_74 +1 javac 0m 20s the patch passed +1 compile 0m 23s the patch passed with JDK v1.7.0_95 +1 javac 0m 23s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 10s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 1m 1s the patch passed +1 javadoc 0m 16s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 19s the patch passed with JDK v1.7.0_95 +1 unit 8m 58s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74. +1 unit 9m 32s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 33m 18s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794627/YARN-4756.003.patch JIRA Issue YARN-4756 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 682b9251c430 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e7ed05e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10841/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10841/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10841/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        ebadger Eric Badger added a comment -

        Fixing whitespace in patch.

        Show
        ebadger Eric Badger added a comment - Fixing whitespace in patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 0m 54s Maven dependency ordering for branch
        +1 mvninstall 6m 52s trunk passed
        +1 compile 6m 9s trunk passed with JDK v1.8.0_74
        +1 compile 6m 47s trunk passed with JDK v1.7.0_95
        +1 checkstyle 1m 6s trunk passed
        +1 mvnsite 0m 55s trunk passed
        +1 mvneclipse 0m 28s trunk passed
        +1 findbugs 1m 17s trunk passed
        +1 javadoc 0m 30s trunk passed with JDK v1.8.0_74
        +1 javadoc 0m 36s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 0m 47s the patch passed
        +1 compile 6m 0s the patch passed with JDK v1.8.0_74
        +1 javac 6m 0s the patch passed
        +1 compile 6m 46s the patch passed with JDK v1.7.0_95
        +1 javac 6m 46s the patch passed
        +1 checkstyle 1m 4s the patch passed
        +1 mvnsite 0m 55s the patch passed
        +1 mvneclipse 0m 27s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 40s the patch passed
        +1 javadoc 0m 30s the patch passed with JDK v1.8.0_74
        +1 javadoc 0m 36s the patch passed with JDK v1.7.0_95
        +1 unit 9m 10s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74.
        -1 unit 95m 32s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_74.
        -1 unit 9m 49s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_95.
        -1 unit 103m 36s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95.
        -1 asflicense 0m 36s Patch generated 15 ASF License warnings.
        264m 58s



        Reason Tests
        JDK v1.8.0_74 Failed junit tests hadoop.mapreduce.v2.TestMRJobs
          hadoop.mapreduce.v2.TestUberAM
        JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
          hadoop.mapreduce.v2.TestMRJobs
          hadoop.mapreduce.v2.TestUberAM



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794780/YARN-4756.004.patch
        JIRA Issue YARN-4756
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 85ff0470dc48 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e7ed05e
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10844/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_74.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10844/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_95.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10844/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10844/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10844/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-YARN-Build/10844/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10844/testReport/
        asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10844/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10844/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 54s Maven dependency ordering for branch +1 mvninstall 6m 52s trunk passed +1 compile 6m 9s trunk passed with JDK v1.8.0_74 +1 compile 6m 47s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 6s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 1m 17s trunk passed +1 javadoc 0m 30s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 36s trunk passed with JDK v1.7.0_95 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 0m 47s the patch passed +1 compile 6m 0s the patch passed with JDK v1.8.0_74 +1 javac 6m 0s the patch passed +1 compile 6m 46s the patch passed with JDK v1.7.0_95 +1 javac 6m 46s the patch passed +1 checkstyle 1m 4s the patch passed +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 40s the patch passed +1 javadoc 0m 30s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 36s the patch passed with JDK v1.7.0_95 +1 unit 9m 10s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74. -1 unit 95m 32s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_74. -1 unit 9m 49s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_95. -1 unit 103m 36s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95. -1 asflicense 0m 36s Patch generated 15 ASF License warnings. 264m 58s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.mapreduce.v2.TestMRJobs   hadoop.mapreduce.v2.TestUberAM JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService   hadoop.mapreduce.v2.TestMRJobs   hadoop.mapreduce.v2.TestUberAM Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794780/YARN-4756.004.patch JIRA Issue YARN-4756 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 85ff0470dc48 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e7ed05e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10844/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10844/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10844/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10844/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10844/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-YARN-Build/10844/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10844/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10844/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/10844/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        ebadger Eric Badger added a comment -

        Removing irrelevant changes from patch.

        Show
        ebadger Eric Badger added a comment - Removing irrelevant changes from patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 50s trunk passed
        +1 compile 0m 22s trunk passed with JDK v1.8.0_74
        +1 compile 0m 27s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 28s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 0m 49s trunk passed
        +1 javadoc 0m 18s trunk passed with JDK v1.8.0_74
        +1 javadoc 0m 21s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 24s the patch passed
        +1 compile 0m 22s the patch passed with JDK v1.8.0_74
        +1 javac 0m 22s the patch passed
        +1 compile 0m 23s the patch passed with JDK v1.7.0_95
        +1 javac 0m 23s the patch passed
        +1 checkstyle 0m 14s the patch passed
        +1 mvnsite 0m 25s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 59s the patch passed
        +1 javadoc 0m 15s the patch passed with JDK v1.8.0_74
        +1 javadoc 0m 19s the patch passed with JDK v1.7.0_95
        +1 unit 9m 6s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74.
        +1 unit 9m 38s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 18s Patch does not generate ASF License warnings.
        33m 51s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794833/YARN-4756.005.patch
        JIRA Issue YARN-4756
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 1eccde100775 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e7ed05e
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10847/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10847/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 50s trunk passed +1 compile 0m 22s trunk passed with JDK v1.8.0_74 +1 compile 0m 27s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 49s trunk passed +1 javadoc 0m 18s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 21s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 24s the patch passed +1 compile 0m 22s the patch passed with JDK v1.8.0_74 +1 javac 0m 22s the patch passed +1 compile 0m 23s the patch passed with JDK v1.7.0_95 +1 javac 0m 23s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 59s the patch passed +1 javadoc 0m 15s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 19s the patch passed with JDK v1.7.0_95 +1 unit 9m 6s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74. +1 unit 9m 38s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 33m 51s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794833/YARN-4756.005.patch JIRA Issue YARN-4756 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1eccde100775 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e7ed05e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10847/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10847/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        ebadger Eric Badger added a comment -

        Karthik Kambatla, does my explanation make sense? Do you have any other concerns with this patch?

        Show
        ebadger Eric Badger added a comment - Karthik Kambatla , does my explanation make sense? Do you have any other concerns with this patch?
        Hide
        kasha Karthik Kambatla added a comment -

        The patch seems reasonable to me.

        +1. Also, quite excited to see a +1 from Hadoop QA. Checking this in.

        Show
        kasha Karthik Kambatla added a comment - The patch seems reasonable to me. +1. Also, quite excited to see a +1 from Hadoop QA. Checking this in.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9576 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9576/)
        YARN-4756. Unnecessary wait in Node Status Updater during reboot. (Eric (kasha: rev e82f961a3925aadf9e53a009820a48ba9e4f78b6)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9576 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9576/ ) YARN-4756 . Unnecessary wait in Node Status Updater during reboot. (Eric (kasha: rev e82f961a3925aadf9e53a009820a48ba9e4f78b6) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java
        Hide
        kasha Karthik Kambatla added a comment -

        Thanks for working on this, Eric. Just committed to trunk, branch-2 and branch-2.8.

        Show
        kasha Karthik Kambatla added a comment - Thanks for working on this, Eric. Just committed to trunk, branch-2 and branch-2.8.

          People

          • Assignee:
            ebadger Eric Badger
            Reporter:
            ebadger Eric Badger
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development