[HADOOP-17531] DistCp: Reduce memory usage on copying huge directories

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.3.1, 3.4.0
• Fix Version/s: 3.3.1, 3.4.0
• Component/s: tools/distcp
• Labels: pull-request-available
• Target Version/s: 3.3.1, 3.4.0
• Hadoop Flags: Reviewed
• Release Note: Added a -useiterator option in distcp which uses listStatusIterator for building the listing. Primarily to reduce memory usage at client for building listing.
• Assignee: Ayush Saxena
• Reporter: Ayush Saxena
• Created: 16/Feb/21 13:04
• Updated: 11/Feb/24 08:51
• Resolved: 27/Mar/21 04:01

Description

Presently distCp, uses the producer-consumer kind of setup while building the listing, the input queue and output queue are both unbounded, thus the listStatus grows quite huge.

Rel Code Part :

https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java#L635

This goes on bredth-first traversal kind of stuff(uses queue instead of earlier stack), so if you have files at lower depth, it will like open up the entire tree and the start processing....

Issue Links

• causes: HADOOP-17628 Distcp contract test is really slow with ABFS and S3A; timing out
• causes: HBASE-25900 HBoss tests compile/failure against Hadoop 3.3.1
• is related to: HADOOP-11827 Speed-up distcp buildListing() using threadpool
• relates to: HADOOP-17558 DistCp: Reduce memory usage using a fixed size ThreadPoolExecutor
• relates to: HIVE-24852 Add support for Snapshots during external table replication
• links to: GitHub Pull Request #2732
• links to: GitHub Pull Request #2808
• links to: GitHub Pull Request #2820