Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15619 Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features
  3. HADOOP-13936

S3Guard: DynamoDB can go out of sync with S3AFileSystem.delete()

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Duplicate
    • Affects Version/s: 3.0.0-beta1, 3.1.0, 3.1.1
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      As a part of S3AFileSystem.delete operation innerDelete is invoked, which deletes keys from S3 in batches (default is 1000). But DynamoDB is updated only at the end of this operation. This can cause issues when deleting large number of keys.

      E.g, it is possible to get exception after deleting 1000 keys and in such cases dynamoDB would not be updated. This can cause DynamoDB to go out of sync.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                rajesh.balamohan Rajesh Balamohan
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: