Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13345 S3Guard: Improved Consistency for S3A
  3. HADOOP-13914

s3guard: improve S3AFileStatus#isEmptyDirectory handling

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: HADOOP-13345
    • Fix Version/s: HADOOP-13345
    • Component/s: fs/s3
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      As discussed in HADOOP-13449, proper support for the isEmptyDirectory() flag stored in S3AFileStatus is missing from DynamoDBMetadataStore.

      The approach taken by LocalMetadataStore is not suitable for the DynamoDB implementation, and also sacrifices good code separation to minimize S3AFileSystem changes pre-merge to trunk.

      I will attach a design doc that attempts to clearly explain the problem and preferred solution. I suggest we do this work after merging the HADOOP-13345 branch to trunk, but am open to suggestions.

      I can also attach a patch of a integration test that exercises the missing case and demonstrates a failure with DynamoDBMetadataStore.

        Attachments

        1. test-only-HADOOP-13914.patch
          5 kB
          Aaron Fabbri
        2. s3guard-empty-dirs.md
          8 kB
          Aaron Fabbri
        3. HADOOP-13914-HADOOP-13345.008.patch
          69 kB
          Aaron Fabbri
        4. HADOOP-13914-HADOOP-13345.007.patch
          69 kB
          Aaron Fabbri
        5. HADOOP-13914-HADOOP-13345.006.patch
          69 kB
          Aaron Fabbri
        6. HADOOP-13914-HADOOP-13345.005.patch
          68 kB
          Aaron Fabbri
        7. HADOOP-13914-HADOOP-13345.004.patch
          67 kB
          Aaron Fabbri
        8. HADOOP-13914-HADOOP-13345.003.patch
          63 kB
          Aaron Fabbri
        9. HADOOP-13914-HADOOP-13345.002.patch
          62 kB
          Aaron Fabbri
        10. HADOOP-13914-HADOOP-13345.000.patch
          14 kB
          Mingliang Liu

          Issue Links

            Activity

              People

              • Assignee:
                fabbri Aaron Fabbri
                Reporter:
                fabbri Aaron Fabbri
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: