Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5448

MapFileOutputFormat#getReaders bug with invisible files/folders

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      MapReduce jobs also produce some invisible files such as _SUCCESS, even when the output format is MapFileOutputFormat. MapFileOutputFormat#getReaders however reads the entire content of the job output, assming that they are MapFiles.

      Path[] names = FileUtil.stat2Paths(fs.listStatus(dir));
      

      It should use a filter to skip the files that start with "." or "_".

      1. MAPREDUCE-5448.addendum.patch
        1 kB
        Harsh J
      2. MAPREDUCE-5448.patch
        4 kB
        Harsh J
      3. MAPREDUCE-5448.patch
        3 kB
        Maysam Yabandeh

        Activity

        Hide
        maysamyabandeh Maysam Yabandeh added a comment -

        The attached path adds a filter to skip the files that start with "." or "_". It also updates the related unit test to show the problem.

        Show
        maysamyabandeh Maysam Yabandeh added a comment - The attached path adds a filter to skip the files that start with "." or "_". It also updates the related unit test to show the problem.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12596060/MAPREDUCE-5448.patch
        against trunk revision .

        -1 patch. Trunk compilation may be broken.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4006//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12596060/MAPREDUCE-5448.patch against trunk revision . -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4006//console This message is automatically generated.
        Hide
        qwertymaniac Harsh J added a comment -

        +1, I've just cleaned up the indentation and the comment typo in the test case added. Changed diff attached for ref.

        Committing shortly. Thanks Maysam!

        Show
        qwertymaniac Harsh J added a comment - +1, I've just cleaned up the indentation and the comment typo in the test case added. Changed diff attached for ref. Committing shortly. Thanks Maysam!
        Hide
        qwertymaniac Harsh J added a comment -

        Tests and compilation for core module passed locally. Committed to branch-2 and trunk.

        Show
        qwertymaniac Harsh J added a comment - Tests and compilation for core module passed locally. Committed to branch-2 and trunk.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7396 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7396/)
        MAPREDUCE-5448. MapFileOutputFormat#getReaders bug with invisible files/folders. Contributed by Maysam Yabandeh. (harsh: rev b46c2bb51ae524e6640756620f70e5925cda7592)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7396 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7396/ ) MAPREDUCE-5448 . MapFileOutputFormat#getReaders bug with invisible files/folders. Contributed by Maysam Yabandeh. (harsh: rev b46c2bb51ae524e6640756620f70e5925cda7592) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java hadoop-mapreduce-project/CHANGES.txt
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12706366/MAPREDUCE-5448.patch
        against trunk revision 4335429.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 javac. The applied patch generated 1157 javac compiler warnings (more than the trunk's current 1155 warnings).

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5325//testReport/
        Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5325//artifact/patchprocess/diffJavacWarnings.txt
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5325//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12706366/MAPREDUCE-5448.patch against trunk revision 4335429. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The applied patch generated 1157 javac compiler warnings (more than the trunk's current 1155 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5325//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5325//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5325//console This message is automatically generated.
        Hide
        qwertymaniac Harsh J added a comment -

        Ah, didn't realise junit.Assert import could cause this. Adding an addendum patch to resolve the import and fall back to using fail(…) from the TestCase inherited method.

        Show
        qwertymaniac Harsh J added a comment - Ah, didn't realise junit.Assert import could cause this. Adding an addendum patch to resolve the import and fall back to using fail(…) from the TestCase inherited method.
        Hide
        qwertymaniac Harsh J added a comment -

        Addendum should clear up the error. Changed on branch-2 and trunk. Sorry for the noise!

        Show
        qwertymaniac Harsh J added a comment - Addendum should clear up the error. Changed on branch-2 and trunk. Sorry for the noise!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7399 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7399/)
        MAPREDUCE-5448. Addendum fix to remove deprecation warning by junit.Assert import in TestFileOutputCommitter. (harsh: rev 4cd54d9a297435150ab61803284eb05603f114e2)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7399 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7399/ ) MAPREDUCE-5448 . Addendum fix to remove deprecation warning by junit.Assert import in TestFileOutputCommitter. (harsh: rev 4cd54d9a297435150ab61803284eb05603f114e2) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #140 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/140/)
        MAPREDUCE-5448. MapFileOutputFormat#getReaders bug with invisible files/folders. Contributed by Maysam Yabandeh. (harsh: rev b46c2bb51ae524e6640756620f70e5925cda7592)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
          MAPREDUCE-5448. Addendum fix to remove deprecation warning by junit.Assert import in TestFileOutputCommitter. (harsh: rev 4cd54d9a297435150ab61803284eb05603f114e2)
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #140 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/140/ ) MAPREDUCE-5448 . MapFileOutputFormat#getReaders bug with invisible files/folders. Contributed by Maysam Yabandeh. (harsh: rev b46c2bb51ae524e6640756620f70e5925cda7592) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java MAPREDUCE-5448 . Addendum fix to remove deprecation warning by junit.Assert import in TestFileOutputCommitter. (harsh: rev 4cd54d9a297435150ab61803284eb05603f114e2) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #874 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/874/)
        MAPREDUCE-5448. MapFileOutputFormat#getReaders bug with invisible files/folders. Contributed by Maysam Yabandeh. (harsh: rev b46c2bb51ae524e6640756620f70e5925cda7592)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
          MAPREDUCE-5448. Addendum fix to remove deprecation warning by junit.Assert import in TestFileOutputCommitter. (harsh: rev 4cd54d9a297435150ab61803284eb05603f114e2)
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #874 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/874/ ) MAPREDUCE-5448 . MapFileOutputFormat#getReaders bug with invisible files/folders. Contributed by Maysam Yabandeh. (harsh: rev b46c2bb51ae524e6640756620f70e5925cda7592) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java MAPREDUCE-5448 . Addendum fix to remove deprecation warning by junit.Assert import in TestFileOutputCommitter. (harsh: rev 4cd54d9a297435150ab61803284eb05603f114e2) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2072 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2072/)
        MAPREDUCE-5448. MapFileOutputFormat#getReaders bug with invisible files/folders. Contributed by Maysam Yabandeh. (harsh: rev b46c2bb51ae524e6640756620f70e5925cda7592)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java
          MAPREDUCE-5448. Addendum fix to remove deprecation warning by junit.Assert import in TestFileOutputCommitter. (harsh: rev 4cd54d9a297435150ab61803284eb05603f114e2)
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2072 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2072/ ) MAPREDUCE-5448 . MapFileOutputFormat#getReaders bug with invisible files/folders. Contributed by Maysam Yabandeh. (harsh: rev b46c2bb51ae524e6640756620f70e5925cda7592) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java MAPREDUCE-5448 . Addendum fix to remove deprecation warning by junit.Assert import in TestFileOutputCommitter. (harsh: rev 4cd54d9a297435150ab61803284eb05603f114e2) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #131 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/131/)
        MAPREDUCE-5448. MapFileOutputFormat#getReaders bug with invisible files/folders. Contributed by Maysam Yabandeh. (harsh: rev b46c2bb51ae524e6640756620f70e5925cda7592)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java
        • hadoop-mapreduce-project/CHANGES.txt
          MAPREDUCE-5448. Addendum fix to remove deprecation warning by junit.Assert import in TestFileOutputCommitter. (harsh: rev 4cd54d9a297435150ab61803284eb05603f114e2)
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #131 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/131/ ) MAPREDUCE-5448 . MapFileOutputFormat#getReaders bug with invisible files/folders. Contributed by Maysam Yabandeh. (harsh: rev b46c2bb51ae524e6640756620f70e5925cda7592) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java hadoop-mapreduce-project/CHANGES.txt MAPREDUCE-5448 . Addendum fix to remove deprecation warning by junit.Assert import in TestFileOutputCommitter. (harsh: rev 4cd54d9a297435150ab61803284eb05603f114e2) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #140 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/140/)
        MAPREDUCE-5448. MapFileOutputFormat#getReaders bug with invisible files/folders. Contributed by Maysam Yabandeh. (harsh: rev b46c2bb51ae524e6640756620f70e5925cda7592)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java
          MAPREDUCE-5448. Addendum fix to remove deprecation warning by junit.Assert import in TestFileOutputCommitter. (harsh: rev 4cd54d9a297435150ab61803284eb05603f114e2)
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #140 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/140/ ) MAPREDUCE-5448 . MapFileOutputFormat#getReaders bug with invisible files/folders. Contributed by Maysam Yabandeh. (harsh: rev b46c2bb51ae524e6640756620f70e5925cda7592) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java MAPREDUCE-5448 . Addendum fix to remove deprecation warning by junit.Assert import in TestFileOutputCommitter. (harsh: rev 4cd54d9a297435150ab61803284eb05603f114e2) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2090 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2090/)
        MAPREDUCE-5448. MapFileOutputFormat#getReaders bug with invisible files/folders. Contributed by Maysam Yabandeh. (harsh: rev b46c2bb51ae524e6640756620f70e5925cda7592)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        • hadoop-mapreduce-project/CHANGES.txt
          MAPREDUCE-5448. Addendum fix to remove deprecation warning by junit.Assert import in TestFileOutputCommitter. (harsh: rev 4cd54d9a297435150ab61803284eb05603f114e2)
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2090 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2090/ ) MAPREDUCE-5448 . MapFileOutputFormat#getReaders bug with invisible files/folders. Contributed by Maysam Yabandeh. (harsh: rev b46c2bb51ae524e6640756620f70e5925cda7592) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java hadoop-mapreduce-project/CHANGES.txt MAPREDUCE-5448 . Addendum fix to remove deprecation warning by junit.Assert import in TestFileOutputCommitter. (harsh: rev 4cd54d9a297435150ab61803284eb05603f114e2) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java

          People

          • Assignee:
            maysamyabandeh Maysam Yabandeh
            Reporter:
            maysamyabandeh Maysam Yabandeh
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 1h
              1h
              Remaining:
              Remaining Estimate - 1h
              1h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development