Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1702

Reduce buffer copies when data is written to DFS

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.14.0
    • 0.18.0
    • None
    • None
    • Incompatible change, Reviewed
    • Reduced buffer copies as data is written to HDFS. The order of sending data bytes and control information has changed, but this will not be observed by client applications.

    Description

      HADOOP-1649 adds extra buffering to improve write performance. The following diagram shows buffers as pointed by (numbers). Each eatra buffer adds an extra copy since most of our read()/write()s match the io.bytes.per.checksum, which is much smaller than buffer size.

             (1)                 (2)          (3)                 (5)
         +---||----[ CLIENT ]---||----<>-----||---[ DATANODE ]---||--<>-> to Mirror  
         | (buffer)                  (socket)           |  (4)
         |                                              +--||--+
       =====                                                    |
       =====                                                  =====
       (disk)                                                 =====
      

      Currently loops that read and write block data, handle one checksum chunk at a time. By reading multiple chunks at a time, we can remove buffers (1), (2), (3), and (5).

      Similarly some copies can be reduced when clients read data from the DFS.

      Attachments

        1. HADOOP-1702.patch
          34 kB
          Raghu Angadi
        2. HADOOP-1702.patch
          39 kB
          Raghu Angadi
        3. HADOOP-1702.patch
          39 kB
          Raghu Angadi
        4. HADOOP-1702.patch
          39 kB
          Raghu Angadi
        5. HADOOP-1702.patch
          39 kB
          Raghu Angadi
        6. HADOOP-1702.patch
          41 kB
          Raghu Angadi
        7. HADOOP-1702.patch
          41 kB
          Raghu Angadi
        8. HADOOP-1702.patch
          42 kB
          Raghu Angadi
        9. HADOOP-1702.patch
          42 kB
          Raghu Angadi

        Issue Links

          Activity

            People

              rangadi Raghu Angadi
              rangadi Raghu Angadi
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: