[HDFS-14574] [distcp] Add ability to increase the replication factor for fileList.seq

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: distcp
• Assignee: Unassigned
• Reporter: Wei-Chiu Chuang
• Created: 15/Jun/19 03:56
• Updated: 02/Oct/19 17:15

Description

distcp creates fileList.seq with default replication factor = 3.

For large clusters runing distcp job with thousands of mappers, that 3-replica for the file listing file is not good enough, because DataNodes easily run out of max number of xceivers.

 

It looks like we can pass in a distcp option, update replication factor in when creating the sequence file writer: https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java#L517-L521

 

Like this:

return SequenceFile.createWriter(getConf(),
        SequenceFile.Writer.file(pathToListFile),
        SequenceFile.Writer.keyClass(Text.class),
        SequenceFile.Writer.valueClass(CopyListingFileStatus.class),
        SequenceFile.Writer.compression(SequenceFile.CompressionType.NONE),
        SequenceFile.Writer.replication((short)100)); <-- this line