Reduce buffer copies when data is written to DFS

• Key: HADOOP-1702
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Raghu Angadi
• Reporter: Raghu Angadi
• Affects Version/s: 0.14.0
• Fix Version/s: 0.18.0

Description

HADOOP-1649 adds extra buffering to improve write performance. The following diagram shows buffers as pointed by (numbers). Each eatra buffer adds an extra copy since most of our read()/write()s match the io.bytes.per.checksum, which is much smaller than buffer size.

       (1)                 (2)          (3)                 (5)
   +---||----[ CLIENT ]---||----<>-----||---[ DATANODE ]---||--<>-> to Mirror  
   | (buffer)                  (socket)           |  (4)
   |                                              +--||--+
 =====                                                    |
 =====                                                  =====
 (disk)                                                 =====

Currently loops that read and write block data, handle one checksum chunk at a time. By reading multiple chunks at a time, we can remove buffers (1), (2), (3), and (5).

Similarly some copies can be reduced when clients read data from the DFS.