Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3580

[JDK 8] TestClientRMService.testGetLabelsToNodes fails

    Details

    • Type: Test
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 2.7.2, 3.0.0-alpha1
    • Component/s: test
    • Labels:
    • Environment:

      JDK 8

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When using JDK 8, TestClientRMService.testGetLabelsToNodes fails:

      java.lang.AssertionError: null
      	at org.junit.Assert.fail(Assert.java:86)
      	at org.junit.Assert.assertTrue(Assert.java:41)
      	at org.junit.Assert.assertTrue(Assert.java:52)
      	at org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService.testGetLabelsToNodes(TestClientRMService.java:1499)
      

        Issue Links

          Activity

          Hide
          rkanter Robert Kanter added a comment -

          When setting node labels, port 0 is considered a "wildcard" port, and the CommonNodeLabelsManager applies the given label to all NMs that previously had a label on that host. Due to the iteration ordering being different between JDK 7 and JDK 8, this was changing the labeling from:

          JDK7
          z
          	host1:1
          	host3:1
          y
          	host2:0
          	host3:0
          x
          	host1:0
          

          to

          JDK8
          x
          	host1:1
          	host1:0
          y
          	host3:0
          	host2:0
          z
          	host3:1
          

          The patch fixes the problem by using different port numbers. It also cleans the test up a little.

          Show
          rkanter Robert Kanter added a comment - When setting node labels, port 0 is considered a "wildcard" port, and the CommonNodeLabelsManager applies the given label to all NMs that previously had a label on that host. Due to the iteration ordering being different between JDK 7 and JDK 8, this was changing the labeling from: JDK7 z host1:1 host3:1 y host2:0 host3:0 x host1:0 to JDK8 x host1:1 host1:0 y host3:0 host2:0 z host3:1 The patch fixes the problem by using different port numbers. It also cleans the test up a little.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 5m 14s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 29s There were no new javac warning messages.
          +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 52s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 13s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 yarn tests 52m 38s Tests passed in hadoop-yarn-server-resourcemanager.
              69m 53s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12730669/YARN-3580.001.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / 90b3845
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7723/artifact/patchprocess/whitespace.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7723/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7723/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7723/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 14s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 29s There were no new javac warning messages. +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 52s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 13s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 52m 38s Tests passed in hadoop-yarn-server-resourcemanager.     69m 53s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730669/YARN-3580.001.patch Optional Tests javac unit findbugs checkstyle git revision trunk / 90b3845 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7723/artifact/patchprocess/whitespace.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7723/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7723/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7723/console This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          Thanks Robert Kanter for reporting the issue and deliver the patch to fix it. Patch LGTM.
          +1. Will commit it shortly.

          Show
          djp Junping Du added a comment - Thanks Robert Kanter for reporting the issue and deliver the patch to fix it. Patch LGTM. +1. Will commit it shortly.
          Hide
          djp Junping Du added a comment -

          I have commit the patch to trunk and branch-2. Thanks Robert Kanter for contributing the patch!

          Show
          djp Junping Du added a comment - I have commit the patch to trunk and branch-2. Thanks Robert Kanter for contributing the patch!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7752 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7752/)
          YARN-3580. [JDK8] TestClientRMService.testGetLabelsToNodes fails. Contributed by Robert Kanter. (junping_du: rev 31b627b2a8755dec0820d01f6f7ac96e4c6194e2)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7752 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7752/ ) YARN-3580 . [JDK8] TestClientRMService.testGetLabelsToNodes fails. Contributed by Robert Kanter. (junping_du: rev 31b627b2a8755dec0820d01f6f7ac96e4c6194e2) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #187 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/187/)
          YARN-3580. [JDK8] TestClientRMService.testGetLabelsToNodes fails. Contributed by Robert Kanter. (junping_du: rev 31b627b2a8755dec0820d01f6f7ac96e4c6194e2)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #187 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/187/ ) YARN-3580 . [JDK8] TestClientRMService.testGetLabelsToNodes fails. Contributed by Robert Kanter. (junping_du: rev 31b627b2a8755dec0820d01f6f7ac96e4c6194e2) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #920 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/920/)
          YARN-3580. [JDK8] TestClientRMService.testGetLabelsToNodes fails. Contributed by Robert Kanter. (junping_du: rev 31b627b2a8755dec0820d01f6f7ac96e4c6194e2)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #920 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/920/ ) YARN-3580 . [JDK8] TestClientRMService.testGetLabelsToNodes fails. Contributed by Robert Kanter. (junping_du: rev 31b627b2a8755dec0820d01f6f7ac96e4c6194e2) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2118 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2118/)
          YARN-3580. [JDK8] TestClientRMService.testGetLabelsToNodes fails. Contributed by Robert Kanter. (junping_du: rev 31b627b2a8755dec0820d01f6f7ac96e4c6194e2)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2118 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2118/ ) YARN-3580 . [JDK8] TestClientRMService.testGetLabelsToNodes fails. Contributed by Robert Kanter. (junping_du: rev 31b627b2a8755dec0820d01f6f7ac96e4c6194e2) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #177 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/177/)
          YARN-3580. [JDK8] TestClientRMService.testGetLabelsToNodes fails. Contributed by Robert Kanter. (junping_du: rev 31b627b2a8755dec0820d01f6f7ac96e4c6194e2)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #177 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/177/ ) YARN-3580 . [JDK8] TestClientRMService.testGetLabelsToNodes fails. Contributed by Robert Kanter. (junping_du: rev 31b627b2a8755dec0820d01f6f7ac96e4c6194e2) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #187 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/187/)
          YARN-3580. [JDK8] TestClientRMService.testGetLabelsToNodes fails. Contributed by Robert Kanter. (junping_du: rev 31b627b2a8755dec0820d01f6f7ac96e4c6194e2)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #187 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/187/ ) YARN-3580 . [JDK8] TestClientRMService.testGetLabelsToNodes fails. Contributed by Robert Kanter. (junping_du: rev 31b627b2a8755dec0820d01f6f7ac96e4c6194e2) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2136 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2136/)
          YARN-3580. [JDK8] TestClientRMService.testGetLabelsToNodes fails. Contributed by Robert Kanter. (junping_du: rev 31b627b2a8755dec0820d01f6f7ac96e4c6194e2)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2136 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2136/ ) YARN-3580 . [JDK8] TestClientRMService.testGetLabelsToNodes fails. Contributed by Robert Kanter. (junping_du: rev 31b627b2a8755dec0820d01f6f7ac96e4c6194e2) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java hadoop-yarn-project/CHANGES.txt
          Show
          ozawa Tsuyoshi Ozawa added a comment - This problem is reproduced on JDK v1.7.0_79 on YARN-4312 : https://issues.apache.org/jira/browse/YARN-4312?focusedCommentId=14979968&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14979968 Backporting this to 2.7.2.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Done.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Done.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8728 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8728/)
          Move YARN-3580 in CHANGES.txt from 2.8 to 2.7.2. (ozawa: rev d2e01f4ed87c3c41156ec9a68855f923f8c0adf9)

          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8728 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8728/ ) Move YARN-3580 in CHANGES.txt from 2.8 to 2.7.2. (ozawa: rev d2e01f4ed87c3c41156ec9a68855f923f8c0adf9) hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #605 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/605/)
          Move YARN-3580 in CHANGES.txt from 2.8 to 2.7.2. (ozawa: rev d2e01f4ed87c3c41156ec9a68855f923f8c0adf9)

          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #605 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/605/ ) Move YARN-3580 in CHANGES.txt from 2.8 to 2.7.2. (ozawa: rev d2e01f4ed87c3c41156ec9a68855f923f8c0adf9) hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2547 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2547/)
          Move YARN-3580 in CHANGES.txt from 2.8 to 2.7.2. (ozawa: rev d2e01f4ed87c3c41156ec9a68855f923f8c0adf9)

          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2547 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2547/ ) Move YARN-3580 in CHANGES.txt from 2.8 to 2.7.2. (ozawa: rev d2e01f4ed87c3c41156ec9a68855f923f8c0adf9) hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1340 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1340/)
          Move YARN-3580 in CHANGES.txt from 2.8 to 2.7.2. (ozawa: rev d2e01f4ed87c3c41156ec9a68855f923f8c0adf9)

          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1340 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1340/ ) Move YARN-3580 in CHANGES.txt from 2.8 to 2.7.2. (ozawa: rev d2e01f4ed87c3c41156ec9a68855f923f8c0adf9) hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #617 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/617/)
          Move YARN-3580 in CHANGES.txt from 2.8 to 2.7.2. (ozawa: rev d2e01f4ed87c3c41156ec9a68855f923f8c0adf9)

          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #617 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/617/ ) Move YARN-3580 in CHANGES.txt from 2.8 to 2.7.2. (ozawa: rev d2e01f4ed87c3c41156ec9a68855f923f8c0adf9) hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #554 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/554/)
          Move YARN-3580 in CHANGES.txt from 2.8 to 2.7.2. (ozawa: rev d2e01f4ed87c3c41156ec9a68855f923f8c0adf9)

          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #554 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/554/ ) Move YARN-3580 in CHANGES.txt from 2.8 to 2.7.2. (ozawa: rev d2e01f4ed87c3c41156ec9a68855f923f8c0adf9) hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2492 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2492/)
          Move YARN-3580 in CHANGES.txt from 2.8 to 2.7.2. (ozawa: rev d2e01f4ed87c3c41156ec9a68855f923f8c0adf9)

          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2492 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2492/ ) Move YARN-3580 in CHANGES.txt from 2.8 to 2.7.2. (ozawa: rev d2e01f4ed87c3c41156ec9a68855f923f8c0adf9) hadoop-yarn-project/CHANGES.txt

            People

            • Assignee:
              rkanter Robert Kanter
              Reporter:
              rkanter Robert Kanter
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development