Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7793

Refactor DFSOutputStream separating DataStreamer out

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: hdfs-client
    • Labels:
      None

      Description

      As proposed by Jing Zhao, Tsz Wo Nicholas Sze and discussed with Zhe Zhang in HDFS-7729, it would be great to refactor DFSOutputStream first and separate DataStreamer out of it, before we enhance it to support the stripping layout. Uma Maheswara Rao G suggested we have this issue to handle the refactoring. Thanks.

      1. HDFS-7793.000.patch
        37 kB
        Jing Zhao
      2. HDFS-7793-001.patch
        138 kB
        Li Bo
      3. HDFS-7793-002.patch
        139 kB
        Li Bo

        Issue Links

          Activity

          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks for the work!

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks for the work!
          Hide
          libo-intel Li Bo added a comment -

          Because the patch is very large, I split this task into three sub tasks.

          Show
          libo-intel Li Bo added a comment - Because the patch is very large, I split this task into three sub tasks.
          Hide
          libo-intel Li Bo added a comment -

          patch 002: fix bugs of appending files

          Show
          libo-intel Li Bo added a comment - patch 002: fix bugs of appending files
          Hide
          libo-intel Li Bo added a comment -

          For patch 001: I split DFSOutputStream into 3 classes, DFSOutputStream, DataStreamer, Packet. Several file append cases still fail. I have just incorporated a part of changes in patch 000 due to the time limited.
          Because the spring festival is coming, I will take leave for more than a week. Happy spring festival, everybody, it's a great pleasure to work with you!

          Show
          libo-intel Li Bo added a comment - For patch 001: I split DFSOutputStream into 3 classes, DFSOutputStream , DataStreamer , Packet . Several file append cases still fail. I have just incorporated a part of changes in patch 000 due to the time limited. Because the spring festival is coming, I will take leave for more than a week. Happy spring festival, everybody, it's a great pleasure to work with you!
          Hide
          jingzhao Jing Zhao added a comment -

          is the final plan still to move DataStreamer out as a separate standalone class

          Yes. I just took the first step to make it a static class. The target of this jira is still to separate it out.

          Show
          jingzhao Jing Zhao added a comment - is the final plan still to move DataStreamer out as a separate standalone class Yes. I just took the first step to make it a static class. The target of this jira is still to separate it out.
          Hide
          zhz Zhe Zhang added a comment -

          Jing Zhao That was fast Just to clarify, is the final plan still to move DataStreamer out as a separate standalone class?

          Show
          zhz Zhe Zhang added a comment - Jing Zhao That was fast Just to clarify, is the final plan still to move DataStreamer out as a separate standalone class?
          Hide
          libo-intel Li Bo added a comment -

          Great! Thanks for helping me and I will go on working on your patch.

          Show
          libo-intel Li Bo added a comment - Great! Thanks for helping me and I will go on working on your patch.
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks for working on this, Bo!

          I also spent some time on this and got a very initial patch. The patch currently converts DataStreamer to a static class, but still needs more code cleanup and extra check on various synchronization scenarios. Please see if this patch can help and save you some time.

          Show
          jingzhao Jing Zhao added a comment - Thanks for working on this, Bo! I also spent some time on this and got a very initial patch. The patch currently converts DataStreamer to a static class, but still needs more code cleanup and extra check on various synchronization scenarios. Please see if this patch can help and save you some time.

            People

            • Assignee:
              libo-intel Li Bo
              Reporter:
              drankye Kai Zheng
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development