Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1702

Reduce buffer copies when data is written to DFS

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Reduced buffer copies as data is written to HDFS. The order of sending data bytes and control information has changed, but this will not be observed by client applications.

      Description

      HADOOP-1649 adds extra buffering to improve write performance. The following diagram shows buffers as pointed by (numbers). Each eatra buffer adds an extra copy since most of our read()/write()s match the io.bytes.per.checksum, which is much smaller than buffer size.

             (1)                 (2)          (3)                 (5)
         +---||----[ CLIENT ]---||----<>-----||---[ DATANODE ]---||--<>-> to Mirror  
         | (buffer)                  (socket)           |  (4)
         |                                              +--||--+
       =====                                                    |
       =====                                                  =====
       (disk)                                                 =====
      

      Currently loops that read and write block data, handle one checksum chunk at a time. By reading multiple chunks at a time, we can remove buffers (1), (2), (3), and (5).

      Similarly some copies can be reduced when clients read data from the DFS.

        Attachments

        1. HADOOP-1702.patch
          42 kB
          Raghu Angadi
        2. HADOOP-1702.patch
          42 kB
          Raghu Angadi
        3. HADOOP-1702.patch
          41 kB
          Raghu Angadi
        4. HADOOP-1702.patch
          41 kB
          Raghu Angadi
        5. HADOOP-1702.patch
          39 kB
          Raghu Angadi
        6. HADOOP-1702.patch
          39 kB
          Raghu Angadi
        7. HADOOP-1702.patch
          39 kB
          Raghu Angadi
        8. HADOOP-1702.patch
          39 kB
          Raghu Angadi
        9. HADOOP-1702.patch
          34 kB
          Raghu Angadi

          Issue Links

            Activity

              People

              • Assignee:
                rangadi Raghu Angadi
                Reporter:
                rangadi Raghu Angadi
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: