Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6165

[JDK8] TestCombineFileInputFormat failed on JDK8

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The error msg:

      testSplitPlacementForCompressedFiles(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)  Time elapsed: 2.487 sec  <<< FAILURE!
      junit.framework.AssertionFailedError: expected:<2> but was:<1>
      	at junit.framework.Assert.fail(Assert.java:57)
      	at junit.framework.Assert.failNotEquals(Assert.java:329)
      	at junit.framework.Assert.assertEquals(Assert.java:78)
      	at junit.framework.Assert.assertEquals(Assert.java:234)
      	at junit.framework.Assert.assertEquals(Assert.java:241)
      	at junit.framework.TestCase.assertEquals(TestCase.java:409)
      	at org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacementForCompressedFiles(TestCombineFileInputFormat.java:911)
      
      testSplitPlacement(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)  Time elapsed: 0.985 sec  <<< FAILURE!
      junit.framework.AssertionFailedError: expected:<2> but was:<1>
      	at junit.framework.Assert.fail(Assert.java:57)
      	at junit.framework.Assert.failNotEquals(Assert.java:329)
      	at junit.framework.Assert.assertEquals(Assert.java:78)
      	at junit.framework.Assert.assertEquals(Assert.java:234)
      	at junit.framework.Assert.assertEquals(Assert.java:241)
      	at junit.framework.TestCase.assertEquals(TestCase.java:409)
      	at org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testSplitPlacement(TestCombineFileInputFormat.java:368)
      
      1. MAPREDUCE-6165-reproduce.patch
        2 kB
        Akira Ajisaka
      2. MAPREDUCE-6165-004.patch
        65 kB
        Akira Ajisaka
      3. MAPREDUCE-6165-003.patch
        65 kB
        Akira Ajisaka
      4. MAPREDUCE-6165-003.patch
        65 kB
        Tsuyoshi Ozawa
      5. MAPREDUCE-6165-002.patch
        66 kB
        Akira Ajisaka
      6. MAPREDUCE-6165-001.patch
        34 kB
        Akira Ajisaka

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2134 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2134/)
          MAPREDUCE-6165. [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed by Akira AJISAKA. (ozawa: rev 551615fa13f65ae996bae9c1bacff189539b6557)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2134 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2134/ ) MAPREDUCE-6165 . [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed by Akira AJISAKA. (ozawa: rev 551615fa13f65ae996bae9c1bacff189539b6557) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #185 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/185/)
          MAPREDUCE-6165. [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed by Akira AJISAKA. (ozawa: rev 551615fa13f65ae996bae9c1bacff189539b6557)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #185 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/185/ ) MAPREDUCE-6165 . [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed by Akira AJISAKA. (ozawa: rev 551615fa13f65ae996bae9c1bacff189539b6557) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #175 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/175/)
          MAPREDUCE-6165. [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed by Akira AJISAKA. (ozawa: rev 551615fa13f65ae996bae9c1bacff189539b6557)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #175 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/175/ ) MAPREDUCE-6165 . [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed by Akira AJISAKA. (ozawa: rev 551615fa13f65ae996bae9c1bacff189539b6557) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2116 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2116/)
          MAPREDUCE-6165. [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed by Akira AJISAKA. (ozawa: rev 551615fa13f65ae996bae9c1bacff189539b6557)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2116 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2116/ ) MAPREDUCE-6165 . [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed by Akira AJISAKA. (ozawa: rev 551615fa13f65ae996bae9c1bacff189539b6557) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #918 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/918/)
          MAPREDUCE-6165. [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed by Akira AJISAKA. (ozawa: rev 551615fa13f65ae996bae9c1bacff189539b6557)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #918 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/918/ ) MAPREDUCE-6165 . [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed by Akira AJISAKA. (ozawa: rev 551615fa13f65ae996bae9c1bacff189539b6557) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #184 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/184/)
          MAPREDUCE-6165. [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed by Akira AJISAKA. (ozawa: rev 551615fa13f65ae996bae9c1bacff189539b6557)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #184 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/184/ ) MAPREDUCE-6165 . [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed by Akira AJISAKA. (ozawa: rev 551615fa13f65ae996bae9c1bacff189539b6557) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #7729 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7729/)
          MAPREDUCE-6165. [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed by Akira AJISAKA. (ozawa: rev 551615fa13f65ae996bae9c1bacff189539b6557)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #7729 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7729/ ) MAPREDUCE-6165 . [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed by Akira AJISAKA. (ozawa: rev 551615fa13f65ae996bae9c1bacff189539b6557) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Committed this to trunk and branch-2. Thanks Akira Ajisaka for your contribution and thanks Robert Kanter for your review!

          Show
          ozawa Tsuyoshi Ozawa added a comment - Committed this to trunk and branch-2. Thanks Akira Ajisaka for your contribution and thanks Robert Kanter for your review!
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          +1, committing this.

          Show
          ozawa Tsuyoshi Ozawa added a comment - +1, committing this.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 40s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 43s There were no new javac warning messages.
          +1 javadoc 9m 57s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 3s The applied patch generated 1 new checkstyle issues (total was 30, now 29).
          +1 whitespace 0m 5s The patch has no lines that end in whitespace.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 57s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 1m 48s Tests passed in hadoop-mapreduce-client-core.
          +1 mapreduce tests 100m 34s Tests passed in hadoop-mapreduce-client-jobclient.
              140m 34s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12729508/MAPREDUCE-6165-004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0d6aa5d
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5639/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5639/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5639/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5639/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5639/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 40s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 43s There were no new javac warning messages. +1 javadoc 9m 57s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 3s The applied patch generated 1 new checkstyle issues (total was 30, now 29). +1 whitespace 0m 5s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 57s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 1m 48s Tests passed in hadoop-mapreduce-client-core. +1 mapreduce tests 100m 34s Tests passed in hadoop-mapreduce-client-jobclient.     140m 34s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729508/MAPREDUCE-6165-004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0d6aa5d checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5639/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5639/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5639/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5639/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5639/console This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          I ran the tests in hadoop-mapreduce-client-jobclient and they all passed locally (took about 1.5 hours). I've kicked off Jenkins again just to be extra sure, but it looks like this patch is probably fine.

          Show
          rkanter Robert Kanter added a comment - I ran the tests in hadoop-mapreduce-client-jobclient and they all passed locally (took about 1.5 hours). I've kicked off Jenkins again just to be extra sure, but it looks like this patch is probably fine.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 28s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 javac 8m 59s There were no new javac warning messages.
          +1 javadoc 11m 2s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 4m 50s There were no new checkstyle issues.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
          +1 findbugs 1m 55s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 1m 34s Tests passed in hadoop-mapreduce-client-core.
          -1 mapreduce tests 62m 17s Tests failed in hadoop-mapreduce-client-jobclient.
              111m 38s  



          Reason Tests
          Timed out tests org.apache.hadoop.mapred.TestJobCounters



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12729508/MAPREDUCE-6165-004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / de9404f
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5478/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5478/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5478/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5478/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 28s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 8m 59s There were no new javac warning messages. +1 javadoc 11m 2s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 4m 50s There were no new checkstyle issues. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 1m 55s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 1m 34s Tests passed in hadoop-mapreduce-client-core. -1 mapreduce tests 62m 17s Tests failed in hadoop-mapreduce-client-jobclient.     111m 38s   Reason Tests Timed out tests org.apache.hadoop.mapred.TestJobCounters Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729508/MAPREDUCE-6165-004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / de9404f hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5478/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5478/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5478/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5478/console This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          I've kicked it off again.

          Show
          rkanter Robert Kanter added a comment - I've kicked it off again.
          Hide
          hadoopqa Hadoop QA added a comment -

          The patch artifact directory on has been removed!
          This is a fatal error for test-patch.sh. Aborting.
          Jenkins (node H3) information at https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5477/ may provide some hints.

          Show
          hadoopqa Hadoop QA added a comment - The patch artifact directory on has been removed! This is a fatal error for test-patch.sh. Aborting. Jenkins (node H3) information at https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5477/ may provide some hints.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Fixed checkstyle and whitespace issues.

          Show
          ajisakaa Akira Ajisaka added a comment - Fixed checkstyle and whitespace issues.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 30s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace.
          +1 javac 7m 32s There were no new javac warning messages.
          +1 javadoc 9m 32s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 5m 20s The applied patch generated 1 additional checkstyle issues.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 56s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 1m 36s Tests passed in hadoop-mapreduce-client-core.
          +1 mapreduce tests 105m 58s Tests passed in hadoop-mapreduce-client-jobclient.
              148m 57s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12707697/MAPREDUCE-6165-003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 5190923
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5468/artifact/patchprocess/whitespace.txt
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5468/artifact/patchprocess/checkstyle-result-diff.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5468/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5468/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5468/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5468/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 30s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 32s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 5m 20s The applied patch generated 1 additional checkstyle issues. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 56s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 1m 36s Tests passed in hadoop-mapreduce-client-core. +1 mapreduce tests 105m 58s Tests passed in hadoop-mapreduce-client-jobclient.     148m 57s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12707697/MAPREDUCE-6165-003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 5190923 whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5468/artifact/patchprocess/whitespace.txt checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5468/artifact/patchprocess/checkstyle-result-diff.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5468/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5468/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5468/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5468/console This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          LGTM. I've checked locally that it now passes on JDKs 7 and 8. I've just kicked off Jenkins to run again; hopefully it will work correctly this time.

          +1 pending Jenkins

          Show
          rkanter Robert Kanter added a comment - LGTM. I've checked locally that it now passes on JDKs 7 and 8. I've just kicked off Jenkins to run again; hopefully it will work correctly this time. +1 pending Jenkins
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12707697/MAPREDUCE-6165-003.patch
          against trunk revision af618f2.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
          org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle
          org.apache.hadoop.mapred.TestMiniMRChildTask
          org.apache.hadoop.mapred.TestMiniMRBringup
          org.apache.hadoop.conf.TestNoDefaultsJobConf
          org.apache.hadoop.mapred.TestJobSysDirWithDFS
          org.apache.hadoop.mapred.TestMiniMRClientCluster
          org.apache.hadoop.mapred.TestClusterMRNotification
          org.apache.hadoop.mapred.TestMRTimelineEventHandling
          org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
          org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
          org.apache.hadoop.mapred.TestReduceFetch
          org.apache.hadoop.mapreduce.v2.TestNonExistentJob
          org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
          org.apache.hadoop.mapred.TestNetworkedJob
          org.apache.hadoop.ipc.TestMRCJCSocketFactory
          org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
          org.apache.hadoop.mapred.TestClusterMapReduceTestCase
          org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
          org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
          org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
          org.apache.hadoop.mapreduce.TestLargeSort
          org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner
          org.apache.hadoop.mapred.TestJobName
          org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
          org.apache.hadoop.mapred.TestJobCounters
          org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
          org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
          org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
          org.apache.hadoop.mapreduce.v2.TestUberAM
          org.apache.hadoop.mapreduce.security.TestMRCredentials
          org.apache.hadoop.mapred.TestLazyOutput
          org.apache.hadoop.mapred.TestMerge
          org.apache.hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities
          org.apache.hadoop.mapreduce.v2.TestRMNMInfo
          org.apache.hadoop.mapred.TestJobCleanup
          org.apache.hadoop.mapreduce.TestChild
          org.apache.hadoop.mapred.TestMiniMRClasspath
          org.apache.hadoop.mapreduce.TestMRJobClient
          org.apache.hadoop.mapreduce.v2.TestMRJobs

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5340//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5340//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707697/MAPREDUCE-6165-003.patch against trunk revision af618f2. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapred.TestMiniMRBringup org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapred.TestClusterMRNotification org.apache.hadoop.mapred.TestMRTimelineEventHandling org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService org.apache.hadoop.mapreduce.security.TestBinaryTokenFile org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapreduce.v2.TestNonExistentJob org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapred.TestNetworkedJob org.apache.hadoop.ipc.TestMRCJCSocketFactory org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapred.TestMRIntermediateDataEncryption org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapreduce.TestLargeSort org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner org.apache.hadoop.mapred.TestJobName org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.mapreduce.security.TestMRCredentials org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapred.TestMerge org.apache.hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities org.apache.hadoop.mapreduce.v2.TestRMNMInfo org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapreduce.TestMRJobClient org.apache.hadoop.mapreduce.v2.TestMRJobs Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5340//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5340//console This message is automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          The results of tests looks strange. Attaching v3 patch again to kick Jenkins.

          Show
          ozawa Tsuyoshi Ozawa added a comment - The results of tests looks strange. Attaching v3 patch again to kick Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12707447/MAPREDUCE-6165-003.patch
          against trunk revision b4b4fe9.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapred.TestReduceFetch
          org.apache.hadoop.mapred.TestMerge
          org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
          org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
          org.apache.hadoop.mapred.TestLazyOutput
          org.apache.hadoop.mapreduce.TestLargeSort
          org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
          org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
          org.apache.hadoop.mapreduce.v2.TestRMNMInfo
          org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
          org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle
          org.apache.hadoop.mapred.TestJobSysDirWithDFS
          org.apache.hadoop.ipc.TestMRCJCSocketFactory
          org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
          org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
          org.apache.hadoop.mapred.TestJobCleanup
          org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
          org.apache.hadoop.conf.TestNoDefaultsJobConf
          org.apache.hadoop.mapred.TestClusterMRNotification
          org.apache.hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities
          org.apache.hadoop.mapreduce.v2.TestNonExistentJob
          org.apache.hadoop.mapreduce.v2.TestMRJobs
          org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
          org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner
          org.apache.hadoop.mapreduce.v2.TestUberAM
          org.apache.hadoop.mapred.TestJobName
          org.apache.hadoop.mapreduce.TestMRJobClient
          org.apache.hadoop.mapred.TestMiniMRBringup
          org.apache.hadoop.mapred.TestClusterMapReduceTestCase
          org.apache.hadoop.mapreduce.security.TestMRCredentials
          org.apache.hadoop.mapred.TestMRTimelineEventHandling
          org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
          org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
          org.apache.hadoop.mapred.TestMiniMRClasspath
          org.apache.hadoop.mapred.TestMiniMRClientCluster
          org.apache.hadoop.mapred.TestNetworkedJob
          org.apache.hadoop.mapred.TestMiniMRChildTask
          org.apache.hadoop.mapreduce.TestChild
          org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
          org.apache.hadoop.mapred.TestJobCounters

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5338//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5338//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707447/MAPREDUCE-6165-003.patch against trunk revision b4b4fe9. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestMerge org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapred.TestMRIntermediateDataEncryption org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapreduce.TestLargeSort org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler org.apache.hadoop.mapreduce.v2.TestRMNMInfo org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.ipc.TestMRCJCSocketFactory org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.mapred.TestClusterMRNotification org.apache.hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities org.apache.hadoop.mapreduce.v2.TestNonExistentJob org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.mapred.TestJobName org.apache.hadoop.mapreduce.TestMRJobClient org.apache.hadoop.mapred.TestMiniMRBringup org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapreduce.security.TestMRCredentials org.apache.hadoop.mapred.TestMRTimelineEventHandling org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapred.TestNetworkedJob org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapreduce.security.TestBinaryTokenFile org.apache.hadoop.mapred.TestJobCounters Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5338//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5338//console This message is automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Updated the patch.

          Show
          ajisakaa Akira Ajisaka added a comment - Updated the patch.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Tsuyoshi Ozawa for comments. I'll create a patch with the pattern:

          for (InputSplit split : splits) {
            fileSplit = (CombineFileSplit) split;
            if (splits.size() == 2) {
              // if split is splits.get(0), verify the split is on rack X
              // if split is splits.get(1), verify the split is on rack Y
            } else if (splits.size() == 1) {
              // verify split is on rack Z
            } else {
              fail()
            }
          }
          
          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Tsuyoshi Ozawa for comments. I'll create a patch with the pattern: for (InputSplit split : splits) { fileSplit = (CombineFileSplit) split; if (splits.size() == 2) { // if split is splits.get(0), verify the split is on rack X // if split is splits.get(1), verify the split is on rack Y } else if (splits.size() == 1) { // verify split is on rack Z } else { fail() } }
          Hide
          ozawa Tsuyoshi Ozawa added a comment -
          we need extra if statement to verify that each split is on the expected rack as follows
          

          Yes, this is one of implementations. Our first priority is preserving all tests, and second is clean code. If we can flatten these statement structure, it's better.

          Show
          ozawa Tsuyoshi Ozawa added a comment - we need extra if statement to verify that each split is on the expected rack as follows Yes, this is one of implementations. Our first priority is preserving all tests, and second is clean code. If we can flatten these statement structure, it's better.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          In addition to previous comments, I prefer to do this on separate JIRA for reviewing quickly.

          Filed MAPREDUCE-6292. I'd like to clean up the test first.

          Show
          ajisakaa Akira Ajisaka added a comment - In addition to previous comments, I prefer to do this on separate JIRA for reviewing quickly. Filed MAPREDUCE-6292 . I'd like to clean up the test first.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Clean up the test to use org.junit instead of junit.framework (TestCombineFileInputFormat used both packages)

          In addition to previous comments, I prefer to do this on separate JIRA for reviewing quickly.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Clean up the test to use org.junit instead of junit.framework (TestCombineFileInputFormat used both packages) In addition to previous comments, I prefer to do this on separate JIRA for reviewing quickly.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Sorry, maybe I confuse you. I meant we shouldn't drop any tests and should preserve tests like this:

          for (fileSplit: splits) {
            if (fileSplit.getNumPaths() == 3) {
              ...
            } else if (...) {
              ...
            } else {
              fail()
            }
          }
          
          Show
          ozawa Tsuyoshi Ozawa added a comment - Sorry, maybe I confuse you. I meant we shouldn't drop any tests and should preserve tests like this: for (fileSplit: splits) { if (fileSplit.getNumPaths() == 3) { ... } else if (...) { ... } else { fail() } }
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Tsuyoshi Ozawa for your comment.

          I think we should iterate splits to verify the values like this:

          Would you tell me the detail?

          for (CombineFileSplit fileSplit : splits) {
            if (splits.size() == 2) {
              ...
            } else if (...) {
              ...
            } else {
              fail()
            }
          }
          

          If we do as the above, we need extra if statement to verify that each split is on the expected rack as follows:

          for (InputSplit split : splits) {
            fileSplit = (CombineFileSplit) split;
            if (splits.size() == 2) {
              // if split is splits.get(0), verify the split is on rack X
              // if split is splits.get(1), verify the split is on rack Y
            } else if (splits.size() == 1) {
              // verify split is on rack Z
            } else {
              fail()
            }
          }
          

          or do you want to drop the test for each split is on the expected rack?

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Tsuyoshi Ozawa for your comment. I think we should iterate splits to verify the values like this: Would you tell me the detail? for (CombineFileSplit fileSplit : splits) { if (splits.size() == 2) { ... } else if (...) { ... } else { fail() } } If we do as the above, we need extra if statement to verify that each split is on the expected rack as follows: for (InputSplit split : splits) { fileSplit = (CombineFileSplit) split; if (splits.size() == 2) { // if split is splits.get(0), verify the split is on rack X // if split is splits.get(1), verify the split is on rack Y } else if (splits.size() == 1) { // verify split is on rack Z } else { fail() } } or do you want to drop the test for each split is on the expected rack?
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Cancelling the patch for the comment.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Cancelling the patch for the comment.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12706859/MAPREDUCE-6165-002.patch
          against trunk revision c6c396f.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 javac. The applied patch generated 1157 javac compiler warnings (more than the trunk's current 1151 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5330//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5330//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5330//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12706859/MAPREDUCE-6165-002.patch against trunk revision c6c396f. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The applied patch generated 1157 javac compiler warnings (more than the trunk's current 1151 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5330//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5330//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5330//console This message is automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Akira Ajisaka Thank you for updating. I think we should iterate splits to verify the values like this:

          for (fileSplit: splits) {
            if (fileSplit.size() == 2) {
              ...
            } else if (...) {
              ...
            } else {
              fail()
            }
          }
          
          Show
          ozawa Tsuyoshi Ozawa added a comment - Akira Ajisaka Thank you for updating. I think we should iterate splits to verify the values like this: for (fileSplit: splits) { if (fileSplit.size() == 2) { ... } else if (...) { ... } else { fail() } }
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Attaching a patch to

          • Use original tests as possible
          • Clean up the test to use org.junit instead of junit.framework (TestCombineFileInputFormat used both packages)

          I verified the test passed with JDK8.

          Show
          ajisakaa Akira Ajisaka added a comment - Attaching a patch to Use original tests as possible Clean up the test to use org.junit instead of junit.framework (TestCombineFileInputFormat used both packages) I verified the test passed with JDK8.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Make sense to me. I'll update the patch.

          Show
          ajisakaa Akira Ajisaka added a comment - Make sense to me. I'll update the patch.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          s/it's not decision/ it's not good decision/

          Show
          ozawa Tsuyoshi Ozawa added a comment - s/it's not decision/ it's not good decision/
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          OK, but it's not decision to reduce tests here. How about doing like this?

          assertEquals(2, splits.size());
          fileSplit = (CombineFileSplit) splits.get(0);
          if (fileSplit.getNumPaths() == 2) {
            assertEquals(1, fileSplit.getLocations().length);
            assertEquals(file2.getName(), fileSplit.getPath(0).getName());
            assertEquals(0, fileSplit.getOffset(0));
            assertEquals(BLOCKSIZE, fileSplit.getLength(0));
            assertEquals(file2.getName(), fileSplit.getPath(1).getName());
            assertEquals(BLOCKSIZE, fileSplit.getOffset(1));
            assertEquals(BLOCKSIZE, fileSplit.getLength(1));
            assertEquals(hosts2[0], fileSplit.getLocations()[0]); // should be on r2
          } else if (fileSplit.getNumPaths() == 1) {
            assertEquals(file1.getName(), fileSplit.getPath(0).getName());
            assertEquals(0, fileSplit.getOffset(0));
            assertEquals(BLOCKSIZE, fileSplit.getLength(0));
            assertEquals(hosts1[0], fileSplit.getLocations()[0]); // should be on r1
          }
          
          Show
          ozawa Tsuyoshi Ozawa added a comment - OK, but it's not decision to reduce tests here. How about doing like this? assertEquals(2, splits.size()); fileSplit = (CombineFileSplit) splits.get(0); if (fileSplit.getNumPaths() == 2) { assertEquals(1, fileSplit.getLocations().length); assertEquals(file2.getName(), fileSplit.getPath(0).getName()); assertEquals(0, fileSplit.getOffset(0)); assertEquals(BLOCKSIZE, fileSplit.getLength(0)); assertEquals(file2.getName(), fileSplit.getPath(1).getName()); assertEquals(BLOCKSIZE, fileSplit.getOffset(1)); assertEquals(BLOCKSIZE, fileSplit.getLength(1)); assertEquals(hosts2[0], fileSplit.getLocations()[0]); // should be on r2 } else if (fileSplit.getNumPaths() == 1) { assertEquals(file1.getName(), fileSplit.getPath(0).getName()); assertEquals(0, fileSplit.getOffset(0)); assertEquals(BLOCKSIZE, fileSplit.getLength(0)); assertEquals(hosts1[0], fileSplit.getLocations()[0]); // should be on r1 }
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Why not using assertEquals(2, splits.size())?

          split.size() can be 1 or 2. The size depends on the ordering.

          I think some applications depend on the ordering...maybe it would be good to use TreeMap<K,V> or LinkedHashMap<K, V> instead of HashMap.

          I'm thinking using TreeMap or LinkedHashMap fixes the test but it's useless for production.

          Show
          ajisakaa Akira Ajisaka added a comment - Why not using assertEquals(2, splits.size())? split.size() can be 1 or 2. The size depends on the ordering. I think some applications depend on the ordering...maybe it would be good to use TreeMap<K,V> or LinkedHashMap<K, V> instead of HashMap. I'm thinking using TreeMap or LinkedHashMap fixes the test but it's useless for production.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Akira Ajisaka Thank you for taking this issue.

          Why not using assertEquals(2, splits.size())?

          -      assertEquals(2, splits.size());
          +      assertTrue(splits.size() >= 1);
          +      assertTrue(splits.size() <= 2);
          

          I think some applications depend on the ordering...maybe it would be good to use TreeMap<K,V> or LinkedHashMap<K, V> instead of HashMap.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Akira Ajisaka Thank you for taking this issue. Why not using assertEquals(2, splits.size())? - assertEquals(2, splits.size()); + assertTrue(splits.size() >= 1); + assertTrue(splits.size() <= 2); I think some applications depend on the ordering...maybe it would be good to use TreeMap<K,V> or LinkedHashMap<K, V> instead of HashMap.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12685053/MAPREDUCE-6165-001.patch
          against trunk revision 6583ad1.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.conf.TestJobConf

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5169//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5169//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12685053/MAPREDUCE-6165-001.patch against trunk revision 6583ad1. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.conf.TestJobConf Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5169//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5169//console This message is automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Resubmitting.

          Show
          ajisakaa Akira Ajisaka added a comment - Resubmitting.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          The test asserted the location of the splits, which is defined by the iteration order of HashMap.entrySet(). However, the order of JDK8 is different from JDK7, so the location of the splits is different.

          For example, JDK7 creates two split,

          Made splits(Test1): 2
          File split(Test1): Paths:/dir2/file2:0+1024,/dir2/file2:1024+1024 Locations:host2.rack2.com:; 
          File split(Test1): Paths:/dir1/file1:0+1024 Locations:host1.rack1.com:; 
          

          on the other hand, JDK8 creates one split in the same block placement

          Made splits(Test1): 1
          File split(Test1): Paths:/dir1/file1:0+1024,/dir2/file2:0+1024,/dir2/file2:1024+1024 Locations:host1.rack1.com:; 
          

          The patch does not assert the locations. It asserts the number of the path and the total length of the splits.

          Show
          ajisakaa Akira Ajisaka added a comment - The test asserted the location of the splits, which is defined by the iteration order of HashMap.entrySet() . However, the order of JDK8 is different from JDK7, so the location of the splits is different. For example, JDK7 creates two split, Made splits(Test1): 2 File split(Test1): Paths:/dir2/file2:0+1024,/dir2/file2:1024+1024 Locations:host2.rack2.com:; File split(Test1): Paths:/dir1/file1:0+1024 Locations:host1.rack1.com:; on the other hand, JDK8 creates one split in the same block placement Made splits(Test1): 1 File split(Test1): Paths:/dir1/file1:0+1024,/dir2/file2:0+1024,/dir2/file2:1024+1024 Locations:host1.rack1.com:; The patch does not assert the locations. It asserts the number of the path and the total length of the splits.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          The test unintentionally relies the iteration order of HashMap.entrySet(). Therefore the test fails in JDK8. MAPREDUCE-6165-reproduce.patch can reproduce the failure on JDK7.

          Show
          ajisakaa Akira Ajisaka added a comment - The test unintentionally relies the iteration order of HashMap.entrySet() . Therefore the test fails in JDK8. MAPREDUCE-6165 -reproduce.patch can reproduce the failure on JDK7.

            People

            • Assignee:
              ajisakaa Akira Ajisaka
              Reporter:
              ywskycn Wei Yan
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development