Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8031 Follow-on work for erasure coding phase I (striping layout)
  3. HDFS-9833

Erasure coding: recomputing block checksum on the fly by reconstructing the missed/corrupt block data

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0-alpha1
    • None

    Description

      As discussed in HDFS-8430 and HDFS-9694, to compute striped file checksum even some of striped blocks are missed, we need to consider recomputing block checksum on the fly for the missed/corrupt blocks. To recompute the block checksum, the block data needs to be reconstructed by erasure decoding, and the main needed codes for the block reconstruction could be borrowed from HDFS-9719, the refactoring of the existing ErasureCodingWorker. In EC worker, reconstructed blocks need to be written out to target datanodes, but here in this case, the remote writing isn't necessary, as the reconstructed block data is only used to recompute the checksum.

      Attachments

        1. HDFS-9833-00-draft.patch
          47 kB
          Rakesh Radhakrishnan
        2. HDFS-9833-01.patch
          39 kB
          Rakesh Radhakrishnan
        3. HDFS-9833-02.patch
          39 kB
          Rakesh Radhakrishnan
        4. HDFS-9833-03.patch
          41 kB
          Rakesh Radhakrishnan
        5. HDFS-9833-04.patch
          41 kB
          Rakesh Radhakrishnan
        6. HDFS-9833-05.patch
          47 kB
          Rakesh Radhakrishnan
        7. HDFS-9833-06.patch
          47 kB
          Rakesh Radhakrishnan
        8. HDFS-9833-07.patch
          47 kB
          Rakesh Radhakrishnan
        9. HDFS-9833-08.patch
          52 kB
          Rakesh Radhakrishnan

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            rakeshr Rakesh Radhakrishnan
            drankye Kai Zheng
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment