Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-265 Revisit append
  3. HDFS-642

Support pipeline close and close recovery

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Append Branch
    • None
    • None
    • Reviewed

    Description

      This jira intends to implement the third stage of block construction: close. The main ideas are:
      1. A client notifies the datanodes in the pipeline to finalize a replica and then shuts down the pipeline by sending a special end-of-block message.
      2. The end-of-block message does not get sent until the client finishes sending all packets belong to the block and have received all their acks.
      3. If close fails, the client recovers the failures by setting up a new pipeline with the remaining datanodes and a newer generation stamp.
      4. I plan to use an empty packet as the end-of-block message.

      Attachments

        1. closePipeline4.patch
          40 kB
          Hairong Kuang
        2. closePipeline3.patch
          40 kB
          Hairong Kuang
        3. closePipeline2.patch
          38 kB
          Hairong Kuang
        4. closePipeline1.patch
          39 kB
          Hairong Kuang
        5. closePipeline.patch
          37 kB
          Hairong Kuang

        Issue Links

          relates to

          Test - A new unit, integration or system test. HDFS-714 Create fault injection test for the new pipeline close

          • Major - Major loss of function.
          • Closed

          Activity

            People

              hairong Hairong Kuang
              hairong Hairong Kuang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: