Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6920

Fix resource leak that happens during container re-initialization.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: nodemanager
    • Labels:
      None

      Description

      Looks like TestNMClient has been failing for a while. Opening this JIRA to track the fix.

      Upon Investigation:
      Container resources

      1. YARN-6920.001.patch
        1 kB
        Arun Suresh
      2. YARN-6920.002.patch
        4 kB
        Arun Suresh
      3. YARN-6920.003.patch
        8 kB
        Arun Suresh
      4. YARN-6920.004.patch
        8 kB
        Arun Suresh

        Activity

        Hide
        asuresh Arun Suresh added a comment - - edited

        It looks like its been failing after YARN-6706. Haibo Chen Can you take a look ?

        Show
        asuresh Arun Suresh added a comment - - edited It looks like its been failing after YARN-6706 . Haibo Chen Can you take a look ?
        Hide
        haibochen Haibo Chen added a comment -

        Sure. Will take a look.

        Show
        haibochen Haibo Chen added a comment - Sure. Will take a look.
        Hide
        asuresh Arun Suresh added a comment -

        Actually, let me take over this Haibo Chen, if you don't mind.
        On further investigation, it looks like the actual issue is due to the fact that during container re-initialization, the container resources which should have been reclaimed by the ContainerScheduler before re-launching the re-initialized container, was never re-claimed - which resulted in a resource leak.
        This was not happening earlier either, but due to the fact that prior to YARN-6706, if maxOppQueueLength == 0, we never even used to perform a resource availability check, the ContainerManager test-cases used to pass
        Will post the fix shortly along with some additional assertions.

        Show
        asuresh Arun Suresh added a comment - Actually, let me take over this Haibo Chen , if you don't mind. On further investigation, it looks like the actual issue is due to the fact that during container re-initialization, the container resources which should have been reclaimed by the ContainerScheduler before re-launching the re-initialized container, was never re-claimed - which resulted in a resource leak. This was not happening earlier either, but due to the fact that prior to YARN-6706 , if maxOppQueueLength == 0, we never even used to perform a resource availability check, the ContainerManager test-cases used to pass Will post the fix shortly along with some additional assertions.
        Hide
        haibochen Haibo Chen added a comment -

        Assigned to you as such.

        Show
        haibochen Haibo Chen added a comment - Assigned to you as such.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
              trunk Compile Tests
        +1 mvninstall 15m 14s trunk passed
        +1 compile 0m 31s trunk passed
        +1 checkstyle 0m 19s trunk passed
        +1 mvnsite 0m 28s trunk passed
        -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
        +1 javadoc 0m 18s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 26s the patch passed
        +1 compile 0m 27s the patch passed
        +1 javac 0m 27s the patch passed
        +1 checkstyle 0m 17s the patch passed
        +1 mvnsite 0m 26s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 53s the patch passed
        +1 javadoc 0m 16s the patch passed
              Other Tests
        +1 unit 13m 42s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        35m 58s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6920
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879937/YARN-6920.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e7cf421177a4 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 9625a03
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16658/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16658/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16658/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 15m 14s trunk passed +1 compile 0m 31s trunk passed +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 28s trunk passed -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 0m 18s trunk passed       Patch Compile Tests +1 mvninstall 0m 26s the patch passed +1 compile 0m 27s the patch passed +1 javac 0m 27s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 53s the patch passed +1 javadoc 0m 16s the patch passed       Other Tests +1 unit 13m 42s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 35m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6920 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879937/YARN-6920.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e7cf421177a4 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9625a03 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16658/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16658/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16658/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        Think I posted an incomplete patch earlier - updated it.
        Haibo Chen / Jian He - quick review ?

        Show
        asuresh Arun Suresh added a comment - Think I posted an incomplete patch earlier - updated it. Haibo Chen / Jian He - quick review ?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 47s trunk passed
        +1 compile 0m 29s trunk passed
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 0m 28s trunk passed
        -1 findbugs 0m 42s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
        +1 javadoc 0m 18s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 24s the patch passed
        +1 compile 0m 26s the patch passed
        +1 javac 0m 26s the patch passed
        +1 checkstyle 0m 18s the patch passed
        +1 mvnsite 0m 26s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 50s the patch passed
        +1 javadoc 0m 15s the patch passed
              Other Tests
        +1 unit 13m 7s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        33m 41s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6920
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879941/YARN-6920.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2dea263a73f7 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6814324
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16659/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16659/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16659/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 47s trunk passed +1 compile 0m 29s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 28s trunk passed -1 findbugs 0m 42s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 0m 18s trunk passed       Patch Compile Tests +1 mvninstall 0m 24s the patch passed +1 compile 0m 26s the patch passed +1 javac 0m 26s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 50s the patch passed +1 javadoc 0m 15s the patch passed       Other Tests +1 unit 13m 7s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 33m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6920 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879941/YARN-6920.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2dea263a73f7 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6814324 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16659/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16659/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16659/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        haibochen Haibo Chen added a comment -

        I applied the 2rd patch, but TestNMClient is still failing for me. Am I missing something.

        Show
        haibochen Haibo Chen added a comment - I applied the 2rd patch, but TestNMClient is still failing for me. Am I missing something.
        Hide
        asuresh Arun Suresh added a comment -

        Hmmm... That could be some timing issue..
        Attaching v3 patch.. with a minor change in TestNMClient. Let's see if it runs through Jenkins fine.

        Show
        asuresh Arun Suresh added a comment - Hmmm... That could be some timing issue.. Attaching v3 patch.. with a minor change in TestNMClient . Let's see if it runs through Jenkins fine.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
              Prechecks
        +1 @author 0m 1s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 50s Maven dependency ordering for branch
        +1 mvninstall 14m 11s trunk passed
        +1 compile 9m 16s trunk passed
        +1 checkstyle 0m 59s trunk passed
        +1 mvnsite 1m 3s trunk passed
        -1 findbugs 0m 53s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
        +1 javadoc 0m 47s trunk passed
              Patch Compile Tests
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 0m 46s the patch passed
        +1 compile 5m 42s the patch passed
        +1 javac 5m 42s the patch passed
        -0 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 208 unchanged - 0 fixed = 209 total (was 208)
        +1 mvnsite 1m 2s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 40s the patch passed
        +1 javadoc 0m 45s the patch passed
              Other Tests
        +1 unit 14m 8s hadoop-yarn-server-nodemanager in the patch passed.
        -1 unit 30m 42s hadoop-yarn-client in the patch failed.
        +1 asflicense 0m 29s The patch does not generate ASF License warnings.
        93m 26s



        Reason Tests
        Failed junit tests hadoop.yarn.client.TestFederationRMFailoverProxyProvider
          hadoop.yarn.client.api.impl.TestNMClient
        Timed out junit tests org.apache.hadoop.yarn.client.api.impl.TestAMRMClient



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6920
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880081/YARN-6920.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b97802280eab 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 12e44e7
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16671/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16671/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16671/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16671/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16671/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated.       Prechecks +1 @author 0m 1s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 50s Maven dependency ordering for branch +1 mvninstall 14m 11s trunk passed +1 compile 9m 16s trunk passed +1 checkstyle 0m 59s trunk passed +1 mvnsite 1m 3s trunk passed -1 findbugs 0m 53s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 0m 47s trunk passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 0m 46s the patch passed +1 compile 5m 42s the patch passed +1 javac 5m 42s the patch passed -0 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 208 unchanged - 0 fixed = 209 total (was 208) +1 mvnsite 1m 2s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 40s the patch passed +1 javadoc 0m 45s the patch passed       Other Tests +1 unit 14m 8s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 30m 42s hadoop-yarn-client in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 93m 26s Reason Tests Failed junit tests hadoop.yarn.client.TestFederationRMFailoverProxyProvider   hadoop.yarn.client.api.impl.TestNMClient Timed out junit tests org.apache.hadoop.yarn.client.api.impl.TestAMRMClient Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6920 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880081/YARN-6920.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b97802280eab 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 12e44e7 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16671/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16671/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16671/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16671/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16671/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        Juggling around with the sleep statements and times and fixing the checkstyle.
        The other two testcase errors are unrelated.

        Show
        asuresh Arun Suresh added a comment - Juggling around with the sleep statements and times and fixing the checkstyle. The other two testcase errors are unrelated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 48s Maven dependency ordering for branch
        +1 mvninstall 14m 5s trunk passed
        +1 compile 9m 6s trunk passed
        +1 checkstyle 0m 58s trunk passed
        +1 mvnsite 1m 4s trunk passed
        -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
        +1 javadoc 0m 46s trunk passed
              Patch Compile Tests
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 0m 45s the patch passed
        +1 compile 5m 48s the patch passed
        +1 javac 5m 48s the patch passed
        +1 checkstyle 0m 56s the patch passed
        +1 mvnsite 1m 3s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 41s the patch passed
        +1 javadoc 0m 44s the patch passed
              Other Tests
        +1 unit 14m 0s hadoop-yarn-server-nodemanager in the patch passed.
        -1 unit 30m 57s hadoop-yarn-client in the patch failed.
        +1 asflicense 0m 27s The patch does not generate ASF License warnings.
        93m 6s



        Reason Tests
        Failed junit tests hadoop.yarn.client.TestFederationRMFailoverProxyProvider
        Timed out junit tests org.apache.hadoop.yarn.client.api.impl.TestAMRMClient



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6920
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880112/YARN-6920.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 69df7d87ff39 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 79df1e7
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16672/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16672/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16672/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16672/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 48s Maven dependency ordering for branch +1 mvninstall 14m 5s trunk passed +1 compile 9m 6s trunk passed +1 checkstyle 0m 58s trunk passed +1 mvnsite 1m 4s trunk passed -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 0m 46s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 45s the patch passed +1 compile 5m 48s the patch passed +1 javac 5m 48s the patch passed +1 checkstyle 0m 56s the patch passed +1 mvnsite 1m 3s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 41s the patch passed +1 javadoc 0m 44s the patch passed       Other Tests +1 unit 14m 0s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 30m 57s hadoop-yarn-client in the patch failed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 93m 6s Reason Tests Failed junit tests hadoop.yarn.client.TestFederationRMFailoverProxyProvider Timed out junit tests org.apache.hadoop.yarn.client.api.impl.TestAMRMClient Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6920 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880112/YARN-6920.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 69df7d87ff39 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 79df1e7 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16672/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html unit https://builds.apache.org/job/PreCommit-YARN-Build/16672/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16672/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16672/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 47s Maven dependency ordering for branch
        +1 mvninstall 14m 47s trunk passed
        +1 compile 8m 43s trunk passed
        +1 checkstyle 0m 58s trunk passed
        +1 mvnsite 1m 3s trunk passed
        -1 findbugs 0m 51s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
        +1 javadoc 0m 45s trunk passed
              Patch Compile Tests
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 0m 45s the patch passed
        +1 compile 5m 21s the patch passed
        +1 javac 5m 21s the patch passed
        +1 checkstyle 0m 56s the patch passed
        +1 mvnsite 1m 1s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 39s the patch passed
        +1 javadoc 0m 43s the patch passed
              Other Tests
        +1 unit 13m 14s hadoop-yarn-server-nodemanager in the patch passed.
        -1 unit 19m 31s hadoop-yarn-client in the patch failed.
        +1 asflicense 0m 29s The patch does not generate ASF License warnings.
        80m 36s



        Reason Tests
        Failed junit tests hadoop.yarn.client.api.impl.TestAMRMProxy
          hadoop.yarn.client.TestFederationRMFailoverProxyProvider



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6920
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880112/YARN-6920.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux ac3750fee0c2 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 79df1e7
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16675/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16675/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16675/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16675/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 47s Maven dependency ordering for branch +1 mvninstall 14m 47s trunk passed +1 compile 8m 43s trunk passed +1 checkstyle 0m 58s trunk passed +1 mvnsite 1m 3s trunk passed -1 findbugs 0m 51s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 0m 45s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 45s the patch passed +1 compile 5m 21s the patch passed +1 javac 5m 21s the patch passed +1 checkstyle 0m 56s the patch passed +1 mvnsite 1m 1s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 39s the patch passed +1 javadoc 0m 43s the patch passed       Other Tests +1 unit 13m 14s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 19m 31s hadoop-yarn-client in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 80m 36s Reason Tests Failed junit tests hadoop.yarn.client.api.impl.TestAMRMProxy   hadoop.yarn.client.TestFederationRMFailoverProxyProvider Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6920 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880112/YARN-6920.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ac3750fee0c2 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 79df1e7 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16675/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html unit https://builds.apache.org/job/PreCommit-YARN-Build/16675/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16675/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16675/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        TestFederationRMFailoverProxyProvider failure has been tracked by YARN-6932 and the TestAMRMClient failure is unrelated. Haibo Chen, can you check if TestNMClient works for you now ? - given that Jenkins is fine with it.

        Show
        asuresh Arun Suresh added a comment - TestFederationRMFailoverProxyProvider failure has been tracked by YARN-6932 and the TestAMRMClient failure is unrelated. Haibo Chen , can you check if TestNMClient works for you now ? - given that Jenkins is fine with it.
        Hide
        haibochen Haibo Chen added a comment -

        Hmm.. It still does not fix TestNMClient for me. Is this the same failure that you have seen?
        "java.lang.AssertionError: Exception is not expected: org.apache.hadoop.yarn.exceptions.YarnException: Cannot perform RE_INIT on [container_1501785504205_0001_01_000001]. Current state is [REINITIALIZING, isReInitializing=true]."
        Looking at the log, it looks like NM cannot find the pid file because the container launch context is empty, and therefore waits for up to maxKillWaitTime which is 2000. So it seems like a timing issue to me. Also, I set some breakpoints which happened to mess up the time on NM, and then I got the same Exception but with container being in DONE state.

        Show
        haibochen Haibo Chen added a comment - Hmm.. It still does not fix TestNMClient for me. Is this the same failure that you have seen? "java.lang.AssertionError: Exception is not expected: org.apache.hadoop.yarn.exceptions.YarnException: Cannot perform RE_INIT on [container_1501785504205_0001_01_000001] . Current state is [REINITIALIZING, isReInitializing=true] ." Looking at the log, it looks like NM cannot find the pid file because the container launch context is empty, and therefore waits for up to maxKillWaitTime which is 2000. So it seems like a timing issue to me. Also, I set some breakpoints which happened to mess up the time on NM, and then I got the same Exception but with container being in DONE state.
        Hide
        haibochen Haibo Chen added a comment -

        Can you add the error message or stack trace of what you saw, and also elaborate a little bit on how the change in ContainerScheduler fixes it?

        Show
        haibochen Haibo Chen added a comment - Can you add the error message or stack trace of what you saw, and also elaborate a little bit on how the change in ContainerScheduler fixes it?
        Hide
        asuresh Arun Suresh added a comment -

        Without this patch, the test case would time out (the testcase timeout is 200s) and you should see the following in the logs:

        ....
        2017-08-03 11:58:04,094 INFO  container.ContainerImpl (ContainerImpl.java:transition(1382)) - Relaunching Container [container_1501786677410_0001_01_000002] for re-initialization !!
        2017-08-03 11:58:04,094 INFO  container.ContainerImpl (ContainerImpl.java:handle(1691)) - Container container_1501786677410_0001_01_000002 transitioned from REINITIALIZING to SCHEDULED
        2017-08-03 11:58:04,094 WARN  scheduler.ContainerScheduler (ContainerScheduler.java:pickOpportunisticContainersToKill(384)) - There are no sufficient resources to start guaranteed [container_1501786677410_0001_01_000002]at the moment. Opportunistic containers are in the process ofbeing killed to make room.
        ....
        

        With the patch, if the test does fail for you - it might be due to some other assertion failure, not a timeout. And you should not see the above call to pickOpportunisticContainersToKill() in the logs.

        Reason:
        During container re-initialization, the container process is killed and re-launched. This transfers control back to the ContainerScheduler, which, after YARN-6706 always checks to see if resources are available to launch the container, irrespective of whether queuing is turned on or off. Un-fortunately, when the container was killed for re-initialization, we had neglected to subtract (reclaim) the containers resources from the utilization tracker, due to which the afore mentioned check fails on re launch. This patch makes sure the resources are reclaimed.

        Show
        asuresh Arun Suresh added a comment - Without this patch, the test case would time out (the testcase timeout is 200s) and you should see the following in the logs: .... 2017-08-03 11:58:04,094 INFO container.ContainerImpl (ContainerImpl.java:transition(1382)) - Relaunching Container [container_1501786677410_0001_01_000002] for re-initialization !! 2017-08-03 11:58:04,094 INFO container.ContainerImpl (ContainerImpl.java:handle(1691)) - Container container_1501786677410_0001_01_000002 transitioned from REINITIALIZING to SCHEDULED 2017-08-03 11:58:04,094 WARN scheduler.ContainerScheduler (ContainerScheduler.java:pickOpportunisticContainersToKill(384)) - There are no sufficient resources to start guaranteed [container_1501786677410_0001_01_000002]at the moment. Opportunistic containers are in the process ofbeing killed to make room. .... With the patch, if the test does fail for you - it might be due to some other assertion failure, not a timeout. And you should not see the above call to pickOpportunisticContainersToKill() in the logs. Reason: During container re-initialization, the container process is killed and re-launched. This transfers control back to the ContainerScheduler, which, after YARN-6706 always checks to see if resources are available to launch the container, irrespective of whether queuing is turned on or off. Un-fortunately, when the container was killed for re-initialization, we had neglected to subtract (reclaim) the containers resources from the utilization tracker, due to which the afore mentioned check fails on re launch. This patch makes sure the resources are reclaimed.
        Hide
        jianhe Jian He added a comment -

        So, the CONTAINER_COMPLETED event will first decrement the resource and the try to start a container, it is possible that a different container gets started later on SCHEDULE_CONTAINER event. I think we need to make sure the same container gets started ?
        Probably we can have a new event called RESTART_CONTAINER and ContainerScheduler#handle can simply starts the container - skipping all those dec/inc resource utilizations, checking opportunistic containers etc. And for service container, user should be expected to always use Guaranteed type.

        Show
        jianhe Jian He added a comment - So, the CONTAINER_COMPLETED event will first decrement the resource and the try to start a container, it is possible that a different container gets started later on SCHEDULE_CONTAINER event. I think we need to make sure the same container gets started ? Probably we can have a new event called RESTART_CONTAINER and ContainerScheduler#handle can simply starts the container - skipping all those dec/inc resource utilizations, checking opportunistic containers etc. And for service container, user should be expected to always use Guaranteed type.
        Hide
        asuresh Arun Suresh added a comment - - edited

        Thanks for taking a look Jian He

        ..it is possible that a different container gets started later on SCHEDULE_CONTAINER event ?

        It is possible, but given the following invariant:

        1. Total Resources of Guaranteed containers ALLOCATED on a Node cannot exceed the Node capacity: The RM ensures that Guaranteed container are never over-allocated on an NM
        2. Total (Opportunistic + Guaranteed) resources of RUNNING containers cannot exceed Node capacity: The ContainerSchedulerenforces this.
        3. Running Opportunistic containers will be preempted to make room for Guaranteed containers: Also enforced by the ContainerScheduler

        We don't really have to worry about if a different container starts in the meanwhile. If the new container that was started is a Guaranteed, then the Node should have the resources to begin with.. and if Opportunistic, then, it will probably be killed when our ReInitializing container is restarted.

        And for service container, user should be expected to always use Guaranteed type.

        Yup. There is already an enforceExecutionType field in the ResourceRequest::ExecutionTypeRequest that an AM can use to ensure that container it receives against this request is of Guaranteed type.

        Show
        asuresh Arun Suresh added a comment - - edited Thanks for taking a look Jian He ..it is possible that a different container gets started later on SCHEDULE_CONTAINER event ? It is possible, but given the following invariant: Total Resources of Guaranteed containers ALLOCATED on a Node cannot exceed the Node capacity : The RM ensures that Guaranteed container are never over-allocated on an NM Total (Opportunistic + Guaranteed) resources of RUNNING containers cannot exceed Node capacity : The ContainerSchedulerenforces this. Running Opportunistic containers will be preempted to make room for Guaranteed containers : Also enforced by the ContainerScheduler We don't really have to worry about if a different container starts in the meanwhile. If the new container that was started is a Guaranteed, then the Node should have the resources to begin with.. and if Opportunistic, then, it will probably be killed when our ReInitializing container is restarted. And for service container, user should be expected to always use Guaranteed type. Yup. There is already an enforceExecutionType field in the ResourceRequest::ExecutionTypeRequest that an AM can use to ensure that container it receives against this request is of Guaranteed type.
        Hide
        jianhe Jian He added a comment -

        I see, thanks for the explanation, so, it's guaranteed the Guaranteed container will be started.
        But I was wondering if this will cause unnecessary churn. Like:
        1) CONTAINER_COMPLETED sent
        2) opportunistic container started.
        3) SCHEDULE_CONTAINER sent
        3) opportunistic killed and make room for the original upgrading container.
        If above is possible occur, we can eliminate this by skipping checking if should launch opportunistic container, and container upgrade can happen more smoothly.

        Show
        jianhe Jian He added a comment - I see, thanks for the explanation, so, it's guaranteed the Guaranteed container will be started. But I was wondering if this will cause unnecessary churn. Like: 1) CONTAINER_COMPLETED sent 2) opportunistic container started. 3) SCHEDULE_CONTAINER sent 3) opportunistic killed and make room for the original upgrading container. If above is possible occur, we can eliminate this by skipping checking if should launch opportunistic container, and container upgrade can happen more smoothly.
        Hide
        asuresh Arun Suresh added a comment -

        True.. the sequence of events you mentioned can happen. I am hoping that once YARN-5972 is completed, freezing and later thawing of the opportunistic container using cgroups freezer / docker pause - rather than simply killing it - will ensure no work is lost (we have seen good results in production on Windows). I can maybe raise a JIRA to optimize the above path and keep it open till we finish with YARN-5972. That way, I can get some data and see an optimization is required. Thoughts ?

        Show
        asuresh Arun Suresh added a comment - True.. the sequence of events you mentioned can happen. I am hoping that once YARN-5972 is completed, freezing and later thawing of the opportunistic container using cgroups freezer / docker pause - rather than simply killing it - will ensure no work is lost (we have seen good results in production on Windows). I can maybe raise a JIRA to optimize the above path and keep it open till we finish with YARN-5972 . That way, I can get some data and see an optimization is required. Thoughts ?
        Hide
        asuresh Arun Suresh added a comment -

        Ping Haibo Chen / Jian He. Do you guys want me to make the afore mentioned optimization here - or should I raise a JIRA to handle this later. I would prefer the later though, if atleast to not hold-up getting TestNMClient to pass again in trunk.

        Show
        asuresh Arun Suresh added a comment - Ping Haibo Chen / Jian He . Do you guys want me to make the afore mentioned optimization here - or should I raise a JIRA to handle this later. I would prefer the later though, if atleast to not hold-up getting TestNMClient to pass again in trunk.
        Hide
        jianhe Jian He added a comment -

        Arun Suresh, yep, I'm fine to fix it in a separate jira.
        I would think the optimization is required in addition to YARN-5972, because the container upgrade process is expected to be quick (we have been working towards this goal to make localization happen before upgrade). And pause and unpause may also introduce additional delay which can be avoided.

        Show
        jianhe Jian He added a comment - Arun Suresh , yep, I'm fine to fix it in a separate jira. I would think the optimization is required in addition to YARN-5972 , because the container upgrade process is expected to be quick (we have been working towards this goal to make localization happen before upgrade). And pause and unpause may also introduce additional delay which can be avoided.
        Hide
        asuresh Arun Suresh added a comment -

        Sure.. Raised YARN-6963 to track that.
        Let me know if this patch is fine and I'll check it in.

        Show
        asuresh Arun Suresh added a comment - Sure.. Raised YARN-6963 to track that. Let me know if this patch is fine and I'll check it in.
        Hide
        jianhe Jian He added a comment -

        yep, the patch lgtm

        Show
        jianhe Jian He added a comment - yep, the patch lgtm
        Hide
        haibochen Haibo Chen added a comment -

        Arun Suresh Seems like I am hitting another issue (due to race condition) before the one you pointed out. Your change looks good to me. But since I cannot download and verify, I'll let Jian He +1 and commit.

        Show
        haibochen Haibo Chen added a comment - Arun Suresh Seems like I am hitting another issue (due to race condition) before the one you pointed out. Your change looks good to me. But since I cannot download and verify, I'll let Jian He +1 and commit.
        Hide
        asuresh Arun Suresh added a comment - - edited

        Kicked Jenkins off again for this patch - to verify that the last successful run was not just a one off thing.
        Will commit it after, if successful - based on Jian He's lgtm

        Show
        asuresh Arun Suresh added a comment - - edited Kicked Jenkins off again for this patch - to verify that the last successful run was not just a one off thing. Will commit it after, if successful - based on Jian He 's lgtm
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 29s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 12s Maven dependency ordering for branch
        +1 mvninstall 17m 24s trunk passed
        +1 compile 13m 55s trunk passed
        +1 checkstyle 1m 14s trunk passed
        +1 mvnsite 1m 25s trunk passed
        -1 findbugs 1m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
        +1 javadoc 1m 10s trunk passed
              Patch Compile Tests
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 7s the patch passed
        +1 compile 8m 42s the patch passed
        +1 javac 8m 42s the patch passed
        +1 checkstyle 1m 8s the patch passed
        +1 mvnsite 1m 20s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 23s the patch passed
        +1 javadoc 0m 55s the patch passed
              Other Tests
        -1 unit 15m 18s hadoop-yarn-server-nodemanager in the patch failed.
        +1 unit 21m 15s hadoop-yarn-client in the patch passed.
        +1 asflicense 0m 33s The patch does not generate ASF License warnings.
        99m 20s



        Reason Tests
        Failed junit tests hadoop.yarn.server.nodemanager.containermanager.TestContainerManager



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6920
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880112/YARN-6920.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b7693b3f98b5 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / adb84f3
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16752/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16752/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16752/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16752/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 29s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 17m 24s trunk passed +1 compile 13m 55s trunk passed +1 checkstyle 1m 14s trunk passed +1 mvnsite 1m 25s trunk passed -1 findbugs 1m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 1m 10s trunk passed       Patch Compile Tests 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 7s the patch passed +1 compile 8m 42s the patch passed +1 javac 8m 42s the patch passed +1 checkstyle 1m 8s the patch passed +1 mvnsite 1m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 23s the patch passed +1 javadoc 0m 55s the patch passed       Other Tests -1 unit 15m 18s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 21m 15s hadoop-yarn-client in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 99m 20s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.TestContainerManager Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6920 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880112/YARN-6920.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b7693b3f98b5 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / adb84f3 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16752/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html unit https://builds.apache.org/job/PreCommit-YARN-Build/16752/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16752/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16752/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 13m 49s trunk passed
        +1 compile 10m 3s trunk passed
        +1 checkstyle 0m 59s trunk passed
        +1 mvnsite 1m 5s trunk passed
        -1 findbugs 0m 53s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
        +1 javadoc 0m 48s trunk passed
              Patch Compile Tests
        0 mvndep 0m 12s Maven dependency ordering for patch
        +1 mvninstall 0m 56s the patch passed
        +1 compile 5m 58s the patch passed
        +1 javac 5m 58s the patch passed
        +1 checkstyle 1m 5s the patch passed
        +1 mvnsite 1m 8s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 42s the patch passed
        +1 javadoc 0m 44s the patch passed
              Other Tests
        +1 unit 13m 32s hadoop-yarn-server-nodemanager in the patch passed.
        +1 unit 19m 45s hadoop-yarn-client in the patch passed.
        +1 asflicense 0m 27s The patch does not generate ASF License warnings.
        82m 14s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6920
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880112/YARN-6920.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e676d3b40006 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / bc20680
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16758/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16758/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16758/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 13m 49s trunk passed +1 compile 10m 3s trunk passed +1 checkstyle 0m 59s trunk passed +1 mvnsite 1m 5s trunk passed -1 findbugs 0m 53s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 0m 48s trunk passed       Patch Compile Tests 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 0m 56s the patch passed +1 compile 5m 58s the patch passed +1 javac 5m 58s the patch passed +1 checkstyle 1m 5s the patch passed +1 mvnsite 1m 8s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 42s the patch passed +1 javadoc 0m 44s the patch passed       Other Tests +1 unit 13m 32s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 19m 45s hadoop-yarn-client in the patch passed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 82m 14s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6920 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880112/YARN-6920.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e676d3b40006 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bc20680 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16758/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16758/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16758/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        Committed this to trunk and branch-2. Thanks for the reviews Jian He, Haibo Chen

        Show
        asuresh Arun Suresh added a comment - Committed this to trunk and branch-2. Thanks for the reviews Jian He , Haibo Chen

          People

          • Assignee:
            asuresh Arun Suresh
            Reporter:
            asuresh Arun Suresh
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development