Hadoop Common
  1. Hadoop Common
  2. HADOOP-1702

Reduce buffer copies when data is written to DFS

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Reduced buffer copies as data is written to HDFS. The order of sending data bytes and control information has changed, but this will not be observed by client applications.

      Description

      HADOOP-1649 adds extra buffering to improve write performance. The following diagram shows buffers as pointed by (numbers). Each eatra buffer adds an extra copy since most of our read()/write()s match the io.bytes.per.checksum, which is much smaller than buffer size.

             (1)                 (2)          (3)                 (5)
         +---||----[ CLIENT ]---||----<>-----||---[ DATANODE ]---||--<>-> to Mirror  
         | (buffer)                  (socket)           |  (4)
         |                                              +--||--+
       =====                                                    |
       =====                                                  =====
       (disk)                                                 =====
      

      Currently loops that read and write block data, handle one checksum chunk at a time. By reading multiple chunks at a time, we can remove buffers (1), (2), (3), and (5).

      Similarly some copies can be reduced when clients read data from the DFS.

      1. HADOOP-1702.patch
        42 kB
        Raghu Angadi
      2. HADOOP-1702.patch
        42 kB
        Raghu Angadi
      3. HADOOP-1702.patch
        41 kB
        Raghu Angadi
      4. HADOOP-1702.patch
        41 kB
        Raghu Angadi
      5. HADOOP-1702.patch
        39 kB
        Raghu Angadi
      6. HADOOP-1702.patch
        39 kB
        Raghu Angadi
      7. HADOOP-1702.patch
        39 kB
        Raghu Angadi
      8. HADOOP-1702.patch
        39 kB
        Raghu Angadi
      9. HADOOP-1702.patch
        34 kB
        Raghu Angadi

        Issue Links

          Activity

            People

            • Assignee:
              Raghu Angadi
              Reporter:
              Raghu Angadi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development