Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7285 Erasure Coding Support inside HDFS
  3. HDFS-7672

Erasure Coding: consolidate streamer coordination logic and handle failure when writing striped blocks

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • HDFS-7285
    • hdfs-client
    • None
    • Reviewed

    Description

      In stripping case, for (6, 3)-Reed-Solomon, a client writes to 6 data blocks and 3 parity blocks concurrently. We need to handle datanode or network failures when writing a EC BlockGroup.

      We also refactor the existing code in DFSStripedOutputStream and StripedDataStreamer.

      Attachments

        1. h7672_20150504.patch
          53 kB
          Tsz-wo Sze
        2. h7672_20150504b.patch
          56 kB
          Tsz-wo Sze
        3. h7672_20150504c.patch
          57 kB
          Tsz-wo Sze
        4. h7672_20150505.patch
          57 kB
          Tsz-wo Sze
        5. h7672_20150505b.patch
          57 kB
          Tsz-wo Sze

        Issue Links

          incorporates

          Sub-task - The sub-task of the issue HDFS-8288 Refactor DFSStripedOutputStream and StripedDataStreamer

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Sub-task - The sub-task of the issue HDFS-8172 Failure handling: DFSStripedOutputStream continues writing with enough remaining datanodes

          • Major - Major loss of function.
          • Resolved
          is part of

          Sub-task - The sub-task of the issue HDFS-7545 [umbrella] Data striping support in HDFS client

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HDFS-8339 Erasure Coding: Badly treated when createBlockOutputStream failed in DataStreamer

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HDFS-8254 Avoid assigning a leading streamer in StripedDataStreamer to tolerate datanode failure

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              szetszwo Tsz-wo Sze
              szetszwo Tsz-wo Sze
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: