[HADOOP-15209] DistCp to eliminate needless deletion of files under already-deleted directories

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.0
• Fix Version/s: 3.1.0
• Component/s: tools/distcp
• Target Version/s: 3.2.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 05/Feb/18 18:32
• Updated: 22/Mar/18 05:29
• Resolved: 15/Mar/18 18:07

Description

DistCP issues a delete(file) request even if is underneath an already deleted directory. This generates needless load on filesystems/object stores, and, if the store throttles delete, can dramatically slow down the delete operation.

If the distcp delete operation can build a history of deleted directories, then it will know when it does not need to issue those deletes.

Care is needed here to make sure that whatever structure is created does not overload the heap of the process.

Issue Links

• contains: HADOOP-15208 DistCp to offer -xtrack <path> option to save src/dest filesets as alternative to delete()
• duplicates: HADOOP-15208 DistCp to offer -xtrack <path> option to save src/dest filesets as alternative to delete()
• is related to: HADOOP-15292 Distcp's use of pread is slowing it down.
• is related to: HADOOP-15281 Distcp to add no-rename copy option
• supercedes: HADOOP-15191 Add Private/Unstable BulkDelete operations to supporting object stores for DistCP