[HADOOP-11827] Speed-up distcp buildListing() using threadpool

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.0, 2.7.1
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: tools/distcp
• Target Version/s: 2.8.0
• Assignee: Zoran Dimitrijevic
• Reporter: Zoran Dimitrijevic
• Due: 17/Apr/15
• Created: 11/Apr/15 19:37
• Updated: 16/Feb/21 13:13
• Resolved: 21/Apr/15 23:52

Description

For very large source trees on s3 distcp is taking long time to build file listing (client code, before starting mappers). For a dataset I used (1.5M files, 50K dirs) it was taking 65 minutes before my fix in HADOOP-11785 and 36 minutes after the fix).

Issue Links

• breaks: HADOOP-12087 [JDK8] Fix javadoc errors caused by incorrect or illegal tags
• breaks: HDFS-9612 DistCp worker threads are not terminated after jobs are done.
• is depended upon by: HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
• relates to: HADOOP-17531 DistCp: Reduce memory usage on copying huge directories