Hadoop Common
  1. Hadoop Common
  2. HADOOP-1702

Reduce buffer copies when data is written to DFS

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Reduced buffer copies as data is written to HDFS. The order of sending data bytes and control information has changed, but this will not be observed by client applications.

      Description

      HADOOP-1649 adds extra buffering to improve write performance. The following diagram shows buffers as pointed by (numbers). Each eatra buffer adds an extra copy since most of our read()/write()s match the io.bytes.per.checksum, which is much smaller than buffer size.

             (1)                 (2)          (3)                 (5)
         +---||----[ CLIENT ]---||----<>-----||---[ DATANODE ]---||--<>-> to Mirror  
         | (buffer)                  (socket)           |  (4)
         |                                              +--||--+
       =====                                                    |
       =====                                                  =====
       (disk)                                                 =====
      

      Currently loops that read and write block data, handle one checksum chunk at a time. By reading multiple chunks at a time, we can remove buffers (1), (2), (3), and (5).

      Similarly some copies can be reduced when clients read data from the DFS.

      1. HADOOP-1702.patch
        42 kB
        Raghu Angadi
      2. HADOOP-1702.patch
        42 kB
        Raghu Angadi
      3. HADOOP-1702.patch
        41 kB
        Raghu Angadi
      4. HADOOP-1702.patch
        41 kB
        Raghu Angadi
      5. HADOOP-1702.patch
        39 kB
        Raghu Angadi
      6. HADOOP-1702.patch
        39 kB
        Raghu Angadi
      7. HADOOP-1702.patch
        39 kB
        Raghu Angadi
      8. HADOOP-1702.patch
        39 kB
        Raghu Angadi
      9. HADOOP-1702.patch
        34 kB
        Raghu Angadi

        Issue Links

          Activity

          Raghu Angadi created issue -
          Doug Cutting made changes -
          Field Original Value New Value
          Fix Version/s 0.15.0 [ 12312565 ]
          Konstantin Shvachko made changes -
          Link This issue relates to HADOOP-2154 [ HADOOP-2154 ]
          Raghu Angadi made changes -
          Link This issue depends upon HADOOP-2758 [ HADOOP-2758 ]
          Raghu Angadi made changes -
          Fix Version/s 0.17.0 [ 12312913 ]
          Description
          HADOOP-1649 adds extra buffering to improve write performance. The following diagram shows buffers as pointed by (numbers). Each eatra buffer adds an extra copy since most of our read()/write()s match the io.bytes.per.checksum, which is much smaller than buffer size.

          {noformat}
                 (1) (2) (3) (5)
             +---||----[ CLIENT ]---||----<>-----||---[ DATANODE ]---||--<>-> to Mirror
             | (buffer) (socket) | (4)
             | +--||--+
           ===== |
           ===== =====
           (disk) =====
          {noformat}

          Currently loops that read and write block data, handle one checksum chunk at a time. By reading multiple chunks at a time, we can remove buffers (1), (2), (3), and (5).

          Similarly some copies can be reduced when clients read data from the DFS.
          HADOOP-1649 adds extra buffering to improve write performance. The following diagram shows buffers as pointed by (numbers). Each eatra buffer adds an extra copy since most of our read()/write()s match the io.bytes.per.checksum, which is much smaller than buffer size.

          {noformat}
                 (1) (2) (3) (5)
             +---||----[ CLIENT ]---||----<>-----||---[ DATANODE ]---||--<>-> to Mirror
             | (buffer) (socket) | (4)
             | +--||--+
           ===== |
           ===== =====
           (disk) =====
          {noformat}

          Currently loops that read and write block data, handle one checksum chunk at a time. By reading multiple chunks at a time, we can remove buffers (1), (2), (3), and (5).

          Similarly some copies can be reduced when clients read data from the DFS.
          Raghu Angadi made changes -
          Attachment HADOOP-1702.patch [ 12376288 ]
          Robert Chansler made changes -
          Fix Version/s 0.17.0 [ 12312913 ]
          Raghu Angadi made changes -
          Fix Version/s 0.18.0 [ 12312972 ]
          Raghu Angadi made changes -
          Attachment HADOOP-1702.patch [ 12380339 ]
          Raghu Angadi made changes -
          Attachment HADOOP-1702.patch [ 12380397 ]
          Raghu Angadi made changes -
          Attachment HADOOP-1702.patch [ 12380401 ]
          Raghu Angadi made changes -
          Attachment HADOOP-1702.patch [ 12380885 ]
          Raghu Angadi made changes -
          Attachment HADOOP-1702.patch [ 12381544 ]
          Raghu Angadi made changes -
          Attachment HADOOP-1702.patch [ 12381708 ]
          Raghu Angadi made changes -
          Hadoop Flags [Reviewed]
          Status Open [ 1 ] Patch Available [ 10002 ]
          Raghu Angadi made changes -
          Attachment HADOOP-1702.patch [ 12381710 ]
          Raghu Angadi made changes -
          Hadoop Flags [Reviewed] [Incompatible change, Reviewed]
          Release Note Reduce buffer copies when data is written to DFS. DataNode takes 30% less CPU. As a result, the format of data DFSClient sends changed and is incompatible with previous clients.
          Raghu Angadi made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Raghu Angadi made changes -
          Attachment HADOOP-1702.patch [ 12381909 ]
          Raghu Angadi made changes -
          Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
          Status Open [ 1 ] Patch Available [ 10002 ]
          Raghu Angadi made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
          Resolution Fixed [ 1 ]
          Robert Chansler made changes -
          Release Note Reduce buffer copies when data is written to DFS. DataNode takes 30% less CPU. As a result, the format of data DFSClient sends changed and is incompatible with previous clients. Reduced buffer copies as data is written to HDFS. The order of sending data bytes and control information has changed, but this will not be observed by client applications.
          Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Owen O'Malley made changes -
          Component/s dfs [ 12310710 ]

            People

            • Assignee:
              Raghu Angadi
              Reporter:
              Raghu Angadi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development