Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13345 S3Guard: Improved Consistency for S3A
  3. HADOOP-14457

create() does not notify metadataStore of parent directories or ensure they're not existing files

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: HADOOP-13345
    • Component/s: fs/s3
    • Labels:
      None

      Description

      Not a great test yet, but it at least reliably demonstrates the issue. LocalMetadataStore will sometimes erroneously report that a directory is empty with isAuthoritative = true when it definitely has children the metadatastore should know about. It doesn't appear to happen if the children are just directory. The fact that it's returning an empty listing is concerning, but the fact that it says it's authoritative might be a second bug.

      diff --git a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
      index 78b3970..1821d19 100644
      --- a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
      +++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
      @@ -965,7 +965,7 @@ public boolean hasMetadataStore() {
         }
       
         @VisibleForTesting
      -  MetadataStore getMetadataStore() {
      +  public MetadataStore getMetadataStore() {
           return metadataStore;
         }
       
      diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
      index 4339649..881bdc9 100644
      --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
      +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
      @@ -23,6 +23,11 @@
       import org.apache.hadoop.fs.contract.AbstractFSContract;
       import org.apache.hadoop.fs.FileSystem;
       import org.apache.hadoop.fs.Path;
      +import org.apache.hadoop.fs.s3a.S3AFileSystem;
      +import org.apache.hadoop.fs.s3a.Tristate;
      +import org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata;
      +import org.apache.hadoop.fs.s3a.s3guard.MetadataStore;
      +import org.junit.Test;
       
       import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
       import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
      @@ -72,4 +77,24 @@ public void testRenameDirIntoExistingDir() throws Throwable {
           boolean rename = fs.rename(srcDir, destDir);
           assertFalse("s3a doesn't support rename to non-empty directory", rename);
         }
      +
      +  @Test
      +  public void testMkdirPopulatesFileAncestors() throws Exception {
      +    final FileSystem fs = getFileSystem();
      +    final MetadataStore ms = ((S3AFileSystem) fs).getMetadataStore();
      +    final Path parent = path("testMkdirPopulatesFileAncestors/source");
      +    try {
      +      fs.mkdirs(parent);
      +      final Path nestedFile = new Path(parent, "dir1/dir2/dir3/file4");
      +      byte[] srcDataset = dataset(256, 'a', 'z');
      +      writeDataset(fs, nestedFile, srcDataset, srcDataset.length,
      +          1024, false);
      +
      +      DirListingMetadata list = ms.listChildren(parent);
      +      assertTrue("MetadataStore falsely reports authoritative empty list",
      +          list.isEmpty() == Tristate.FALSE || !list.isAuthoritative());
      +    } finally {
      +      fs.delete(parent, true);
      +    }
      +  }
       }
      

        Attachments

        1. HADOOP-14457-HADOOP-13345.001.patch
          7 kB
          Sean Mackrory
        2. HADOOP-14457-HADOOP-13345.002.patch
          11 kB
          Sean Mackrory
        3. HADOOP-14457-HADOOP-13345.003.patch
          16 kB
          Sean Mackrory
        4. HADOOP-14457-HADOOP-13345.004.patch
          16 kB
          Sean Mackrory
        5. HADOOP-14457-HADOOP-13345.005.patch
          16 kB
          Sean Mackrory
        6. HADOOP-14457-HADOOP-13345.006.patch
          9 kB
          Sean Mackrory
        7. HADOOP-14457-HADOOP-13345.007.patch
          9 kB
          Sean Mackrory
        8. HADOOP-14457-HADOOP-13345.008.patch
          9 kB
          Sean Mackrory
        9. HADOOP-14457-HADOOP-13345.009.patch
          9 kB
          Sean Mackrory
        10. HADOOP-14457-HADOOP-13345.010.patch
          13 kB
          Sean Mackrory

          Issue Links

            Activity

              People

              • Assignee:
                mackrorysd Sean Mackrory
                Reporter:
                mackrorysd Sean Mackrory
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: