[HADOOP-18679] Add API for bulk/paged delete of files and objects

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.5
• Fix Version/s: 3.4.1
• Component/s: fs/s3
• Labels: pull-request-available
• Assignee: Mukund Thakur
• Reporter: Steve Loughran
• Created: 24/Mar/23 14:49
• Updated: 5 days ago 10:44
• Resolved: 28/May/24 21:22

Description

iceberg and hbase could benefit from being able to give a list of individual files to delete -files which may be scattered round the bucket for better read peformance.

Add some new optional interface for an object store which allows a caller to submit a list of paths to files to delete, where
the expectation is

• if a path is a file: delete
• if a path is a dir, outcome undefined
  For s3 that'd let us build these into DeleteRequest objects, and submit, without any probes first.

Cherrypicking

when cherrypicking, you must include

• followup commit #6854
• https://issues.apache.org/jira/browse/HADOOP-19196
• test fixes HADOOP-19814 and HADOOP-19188

Issue Links

• causes: HADOOP-19184 TestStagingCommitter.testJobCommitFailure failing
• causes: HADOOP-19188 TestHarFileSystem and TestFilterFileSystem failing after bulk delete API added
• causes: HADOOP-19196 Bulk delete api doesn't take the path to delete as the base path
• is depended upon by: HADOOP-19131 WrappedIO to export modern filesystem/statistics APIs in a reflection friendly form
• is duplicated by: HADOOP-19191 Batch APIs for delete
• is related to: HADOOP-18722 Optimise S3A delete objects when multiObjectDelete is disabled
• is related to: HADOOP-19203 WrappedIO BulkDelete API to raise IOEs as UncheckedIOExceptions
• relates to: HADOOP-18420 Optimise S3A’s recursive delete to drop successful S3 keys on retry of S3 DeleteObjects
• links to: GitHub Pull Request #5993
• links to: GitHub Pull Request #6494
• links to: GitHub Pull Request #6726
• links to: GitHub Pull Request #6738
• links to: GitHub Pull Request #6854
• links to: GitHub Pull Request #10233