Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4090

getFileChecksum() result incompatible when called against zero-byte files.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.4, 2.0.2-alpha
    • Fix Version/s: 3.0.0, 2.0.3-alpha, 0.23.5
    • Component/s: hdfs-client
    • Labels:
      None

      Description

      When getFileChecksum() is called against a zero-byte file, the branch-1 client returns MD5MD5CRC32FileChecksum with crcPerBlock=0, bytePerCrc=0 and md5=70bc8f4b72a86921468bf8e8441dce51, whereas a null is returned in trunk.

      The null makes sense since there is no actual block checksums, but this breaks the compatibility when doing distCp and calling getFileChecksum() via webhdfs or hftp.

      This JIRA is to make the client to return the same 'magic' value that the branch-1 and earlier clients return.

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1235 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1235/)
        HDFS-4090. getFileChecksum() result incompatible when called against zero-byte files. (Kihwal Lee via daryn) (Revision 1401309)

        Result = SUCCESS
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401309
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1235 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1235/ ) HDFS-4090 . getFileChecksum() result incompatible when called against zero-byte files. (Kihwal Lee via daryn) (Revision 1401309) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401309 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1205 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1205/)
        HDFS-4090. getFileChecksum() result incompatible when called against zero-byte files. (Kihwal Lee via daryn) (Revision 1401309)

        Result = FAILURE
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401309
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1205 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1205/ ) HDFS-4090 . getFileChecksum() result incompatible when called against zero-byte files. (Kihwal Lee via daryn) (Revision 1401309) Result = FAILURE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401309 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #414 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/414/)
        svn merge -c 1401309 FIXES: HDFS-4090. getFileChecksum() result incompatible when called against zero-byte files. (Kihwal Lee via daryn) (Revision 1401322)

        Result = SUCCESS
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401322
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #414 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/414/ ) svn merge -c 1401309 FIXES: HDFS-4090 . getFileChecksum() result incompatible when called against zero-byte files. (Kihwal Lee via daryn) (Revision 1401322) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401322 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #13 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/13/)
        HDFS-4090. getFileChecksum() result incompatible when called against zero-byte files. (Kihwal Lee via daryn) (Revision 1401309)

        Result = FAILURE
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401309
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #13 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/13/ ) HDFS-4090 . getFileChecksum() result incompatible when called against zero-byte files. (Kihwal Lee via daryn) (Revision 1401309) Result = FAILURE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401309 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
        Hide
        Daryn Sharp added a comment -

        I've committed to trunk, branch-2, branch-23. Thanks Kihwal!

        Show
        Daryn Sharp added a comment - I've committed to trunk, branch-2, branch-23. Thanks Kihwal!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #2911 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2911/)
        HDFS-4090. getFileChecksum() result incompatible when called against zero-byte files. (Kihwal Lee via daryn) (Revision 1401309)

        Result = SUCCESS
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401309
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #2911 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2911/ ) HDFS-4090 . getFileChecksum() result incompatible when called against zero-byte files. (Kihwal Lee via daryn) (Revision 1401309) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401309 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
        Hide
        Daryn Sharp added a comment -

        +1 Will commit shortly.

        Show
        Daryn Sharp added a comment - +1 Will commit shortly.
        Hide
        Ravi Prakash added a comment -

        The only nit (and I don't care if you don't fix it), is that the comments could have taken 2 lines instead of 3.
        Also, when I removed the src/main part of the patch, the test fails with a NPE because zeroChecksum == null. Maybe check for null? But I'm not going to be a stickler for that either.

        Please feel free to check it in. +1 lgtm

        Show
        Ravi Prakash added a comment - The only nit (and I don't care if you don't fix it), is that the comments could have taken 2 lines instead of 3. Also, when I removed the src/main part of the patch, the test fails with a NPE because zeroChecksum == null. Maybe check for null? But I'm not going to be a stickler for that either. Please feel free to check it in. +1 lgtm
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12550084/hdfs-4090.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3372//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3372//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550084/hdfs-4090.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3372//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3372//console This message is automatically generated.

          People

          • Assignee:
            Kihwal Lee
            Reporter:
            Kihwal Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development