Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15619 Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features
  3. HADOOP-16429

DynamoDBMetaStore deleteSubtree to delete leaf nodes first

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 3.3.0
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      in deleteSubtree(path), the DynamoDB metastore walks down the tree, returning elements to delete. But it will delete parent entries before children, so if an operation fails partway through, there will be orphans

      Better: DescendantsIterator to return all the leaf nodes before their parents so the deletion is done bottom up

      Also: push the deletions off into their own async queue/pool so that they don't become the bottleneck on the process

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: