Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13345 S3Guard: Improved Consistency for S3A
  3. HADOOP-14226

S3Guard: DynamoDBMetadataStore::move() should populate ancestor directories of destination paths

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: HADOOP-13345
    • Fix Version/s: HADOOP-13345
    • Component/s: fs/s3
    • Labels:
      None

      Description

      UPDATE: Instead of changing the test, in this JIRA we make sure for each path to put, DDBMS::move() has records of each directory up to the root. See Aaron Fabbri's comment in this JIRA for more detail.

      After running ITestDynamoDBMetadataStoreScale, the test data is not cleaned up. There is a call to clearMetadataStore(ms, count); in the finally clause though. The reason is that, the internally called method DynamoDBMetadataStore::deleteSubtree() is assuming there should be an item for the parent dest path:

      parent=/fake-bucket, child=moved-here, is_dir=true
      

      In DynamoDBMetadataStore implementation, we assume that if a path exists, all its ancestors will also exist in the table. We need to pre-create dest path to maintain this invariant so that test data can be cleaned up successfully.

      I think there may be other tests with the same problem. Let's identify/address them separately.

        Attachments

        1. HADOOP-14226-HADOOP-13345.000.patch
          2 kB
          Mingliang Liu
        2. HADOOP-14226-HADOOP-13345.001.patch
          7 kB
          Mingliang Liu
        3. HADOOP-14226-HADOOP-13345.002.patch
          7 kB
          Aaron Fabbri

          Activity

            People

            • Assignee:
              liuml07 Mingliang Liu
              Reporter:
              liuml07 Mingliang Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: