[HADOOP-15292] Distcp's use of pread is slowing it down.

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.5.0
• Fix Version/s: 3.1.0
• Component/s: tools/distcp
• Target Version/s: 2.9.1
• Assignee: Virajith Jalaparti
• Reporter: Virajith Jalaparti
• Created: 06/Mar/18 02:26
• Updated: 17/Jan/19 14:28
• Resolved: 08/Mar/18 11:19

Description

Distcp currently uses positioned-reads (in RetriableFileCopyCommand#copyBytes) when the source offset is > 0. This results in unnecessary overheads (new BlockReader being created on the client-side, multiple readBlock() calls to the Datanodes, each of which requires the creation of a BlockSender and an inputstream to the ReplicaInfo).

Issue Links

• is related to: HDFS-9146 HDFS forward seek() within a block shouldn't spawn new TCP Peer/RemoteBlockReader
• is required by: HADOOP-16049 DistCp result has data and checksum mismatch when blocks per chunk > 0
• relates to: HADOOP-15209 DistCp to eliminate needless deletion of files under already-deleted directories
• relates to: MAPREDUCE-5899 Support incremental data copy in DistCp