Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1702

Reduce buffer copies when data is written to DFS

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.14.0
    • 0.18.0
    • None
    • None
    • Incompatible change, Reviewed
    • Reduced buffer copies as data is written to HDFS. The order of sending data bytes and control information has changed, but this will not be observed by client applications.

    Description

      HADOOP-1649 adds extra buffering to improve write performance. The following diagram shows buffers as pointed by (numbers). Each eatra buffer adds an extra copy since most of our read()/write()s match the io.bytes.per.checksum, which is much smaller than buffer size.

             (1)                 (2)          (3)                 (5)
         +---||----[ CLIENT ]---||----<>-----||---[ DATANODE ]---||--<>-> to Mirror  
         | (buffer)                  (socket)           |  (4)
         |                                              +--||--+
       =====                                                    |
       =====                                                  =====
       (disk)                                                 =====
      

      Currently loops that read and write block data, handle one checksum chunk at a time. By reading multiple chunks at a time, we can remove buffers (1), (2), (3), and (5).

      Similarly some copies can be reduced when clients read data from the DFS.

      Attachments

        1. HADOOP-1702.patch
          34 kB
          Raghu Angadi
        2. HADOOP-1702.patch
          39 kB
          Raghu Angadi
        3. HADOOP-1702.patch
          39 kB
          Raghu Angadi
        4. HADOOP-1702.patch
          39 kB
          Raghu Angadi
        5. HADOOP-1702.patch
          39 kB
          Raghu Angadi
        6. HADOOP-1702.patch
          41 kB
          Raghu Angadi
        7. HADOOP-1702.patch
          41 kB
          Raghu Angadi
        8. HADOOP-1702.patch
          42 kB
          Raghu Angadi
        9. HADOOP-1702.patch
          42 kB
          Raghu Angadi

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            rangadi Raghu Angadi
            rangadi Raghu Angadi
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment