Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1524

Image loader should make sure to read every byte in image file

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.22.0, 0.23.0
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When I work on HDFS-1070, I come across a very strange bug. Occasionally when loading a compressed image, NameNode throws an exception complaining that the image file is corrupt. However, the result returned by the md5sum utility matches the checksum value stored in the VERSION file.

      It turns out the image loader leaves 4 bytes unread after loading all the real data of an image. Those unread bytes may be some compression-related meta-info. The image loader should make sure to read to the end of file in order for an correct checksum.

      1. eofImageFile.patch
        0.6 kB
        Hairong Kuang

        Activity

        Hairong Kuang created issue -
        Hairong Kuang made changes -
        Field Original Value New Value
        Attachment eofImageFile.patch [ 12465053 ]
        Hairong Kuang made changes -
        Priority Major [ 3 ] Blocker [ 1 ]
        Hairong Kuang made changes -
        Attachment eofImageFile.patch [ 12465053 ]
        Hairong Kuang made changes -
        Attachment eofImageFile.patch [ 12465056 ]
        Hide
        Konstantin Boudnik added a comment -

        +1 patch looks good.

        Show
        Konstantin Boudnik added a comment - +1 patch looks good.
        Hide
        Hairong Kuang added a comment -

        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] -1 tests included. The patch doesn't appe
        [exec] ar to include any new or modified tests.
        [exec] Please justify why no new tests are needed for this patch.
        [exec] Also please list what manual steps were performed to verify this patch.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        [exec]
        [exec] +1 system test framework. The patch passed system test framework compile.
        [exec]

        Failed unit tests are all known ones:
        TestBlockRecovery
        TestBlockTokenWithDFS
        TestStorageRestore
        TestBackupNode
        TestHDFSTrash (timeout)

        Since the change is trivial and it is hard to come up with the test case, I will not provide a unit test and commit the patch.

        Show
        Hairong Kuang added a comment - [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appe [exec] ar to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] Failed unit tests are all known ones: TestBlockRecovery TestBlockTokenWithDFS TestStorageRestore TestBackupNode TestHDFSTrash (timeout) Since the change is trivial and it is hard to come up with the test case, I will not provide a unit test and commit the patch.
        Hide
        Hairong Kuang added a comment -

        I've just committed this.

        Show
        Hairong Kuang added a comment - I've just committed this.
        Hairong Kuang made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-22-branch #35 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/35/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #35 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/35/ )
        Konstantin Shvachko made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Hairong Kuang
            Reporter:
            Hairong Kuang
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development