[HADOOP-15191] Add Private/Unstable BulkDelete operations to supporting object stores for DistCP

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.9.0
• Component/s: fs/s3, tools/distcp
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 25/Jan/18 06:10
• Updated: 21/Mar/18 23:58
• Resolved: 21/Feb/18 16:35

Description

Large scale DistCP with the -delete option doesn't finish in a viable time because of the final CopyCommitter doing a 1 by 1 delete of all missing files. This isn't randomized (the list is sorted), and it's throttled by AWS.

If bulk deletion of files was exposed as an API, distCP would do 1/1000 of the REST calls, so not get throttled.

Proposed: add an initially private/unstable interface for stores, BulkDelete which declares a page size and offers a bulkDelete(List<Path>) operation for the bulk deletion.

Issue Links

• is related to: HADOOP-13936 S3Guard: DynamoDB can go out of sync with S3AFileSystem.delete()
• is related to: HADOOP-15208 DistCp to offer -xtrack <path> option to save src/dest filesets as alternative to delete()
• is superceded by: HADOOP-15209 DistCp to eliminate needless deletion of files under already-deleted directories