Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18477 Über-jira: S3A Hadoop 3.3.9 features
  3. HADOOP-18420

Optimise S3A’s recursive delete to drop successful S3 keys on retry of S3 DeleteObjects

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • fs/s3
    • None

    Description

      S3A users with large filesystems performing renames or deletes can run into throttling when S3A performs a bulk delete on keys. These are currently batches of 250 (https://github.com/apache/hadoop/blob/c1d82cd95e375410cb0dffc2931063d48687386f/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L319-L323).

      When the bulk delete (S3 DeleteObjects) fails, it provides a list of keys that failed and why. Today, S3A recovers from throttles by sending the DeleteObjects request again with no change. This can result in additional deletes and counts towards throttling limits.

      Instead, S3A should retry only the keys that failed, limiting the number of mutations against the S3 bucket, and hopefully mitigate errors when deleting a large number of objects.

      Attachments

        Issue Links

          is related to

          Sub-task - The sub-task of the issue HADOOP-18679 Add API for bulk/paged object deletion

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue HADOOP-17063 S3A deleteObjects hanging/retrying forever

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            People

              Unassigned Unassigned
              dannycjones Daniel Carl Jones
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: