[HADOOP-15208] DistCp to offer -xtrack <path> option to save src/dest filesets as alternative to delete()

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.9.0
• Component/s: tools/distcp
• Target Version/s: 3.1.0, 2.10.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 05/Feb/18 18:26
• Updated: 22/Mar/18 05:29
• Resolved: 22/Mar/18 05:29

Description

There are opportunities to improve distcp delete performance and scalability with object stores, but you need to test with production datasets to determine if the optimizations work, don't run out of memory, etc.

By adding the option to save the sequence files of source, dest listings, people (myself included) can experiment with different strategies before trying to commit one which doesn't scale

Issue Links

• Is contained by: HADOOP-15209 DistCp to eliminate needless deletion of files under already-deleted directories
• is duplicated by: HADOOP-15209 DistCp to eliminate needless deletion of files under already-deleted directories
• relates to: HADOOP-15191 Add Private/Unstable BulkDelete operations to supporting object stores for DistCP