Hadoop Common
  1. Hadoop Common
  2. HADOOP-10425

Incompatible behavior of LocalFileSystem:getContentSummary

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.4.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Unlike in Hadoop1, FilterFileSystem overrides getContentSummary, which causes content summary to be called on rawLocalFileSystem in Local mode.

      This impacts the computations of Stats in Hive with getting back FileSizes that include the size of the crc files.

      1. c10425_20140324.patch
        2 kB
        Tsz Wo Nicholas Sze
      2. c10425_20140324b.patch
        3 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          Tsz Wo Nicholas Sze added a comment -

          FilterFileSystem should not override getContentSummary(..) since it may use wrong versions of getFileStatus(..) and listStatus(..).

          Show
          Tsz Wo Nicholas Sze added a comment - FilterFileSystem should not override getContentSummary(..) since it may use wrong versions of getFileStatus(..) and listStatus(..).
          Hide
          Tsz Wo Nicholas Sze added a comment -

          c10425_20140324.patch: remove getContentSummary from FilterFileSystem. If necessary, subclasses of FilterFileSystem should override it.

          Show
          Tsz Wo Nicholas Sze added a comment - c10425_20140324.patch: remove getContentSummary from FilterFileSystem. If necessary, subclasses of FilterFileSystem should override it.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Without the patch, the new test will failed since getContentSummary will count also the CRC files created by ChecksumFileSystem.

          Show
          Tsz Wo Nicholas Sze added a comment - Without the patch, the new test will failed since getContentSummary will count also the CRC files created by ChecksumFileSystem.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Oops, also need to update ChRootedFileSystem. Thank Brandon for catching it.

          c10425_20140324b.patch

          Show
          Tsz Wo Nicholas Sze added a comment - Oops, also need to update ChRootedFileSystem. Thank Brandon for catching it. c10425_20140324b.patch
          Hide
          Brandon Li added a comment -

          +1. Pending Jenkins.

          Show
          Brandon Li added a comment - +1. Pending Jenkins.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12636475/c10425_20140324b.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3703//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3703//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12636475/c10425_20140324b.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3703//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3703//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Thanks Brandon for reviewing the patch.

          I have committed this.

          Show
          Tsz Wo Nicholas Sze added a comment - Thanks Brandon for reviewing the patch. I have committed this.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5396 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5396/)
          HADOOP-10425. LocalFileSystem.getContentSummary should not count crc files. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1581183)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5396 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5396/ ) HADOOP-10425 . LocalFileSystem.getContentSummary should not count crc files. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1581183 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #520 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/520/)
          HADOOP-10425. LocalFileSystem.getContentSummary should not count crc files. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1581183)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #520 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/520/ ) HADOOP-10425 . LocalFileSystem.getContentSummary should not count crc files. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1581183 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1737 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1737/)
          HADOOP-10425. LocalFileSystem.getContentSummary should not count crc files. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1581183)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1737 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1737/ ) HADOOP-10425 . LocalFileSystem.getContentSummary should not count crc files. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1581183 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1712 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1712/)
          HADOOP-10425. LocalFileSystem.getContentSummary should not count crc files. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1581183)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1712 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1712/ ) HADOOP-10425 . LocalFileSystem.getContentSummary should not count crc files. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1581183 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Brandon Li
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development