Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13345 S3Guard: Improved Consistency for S3A
  3. HADOOP-14172

S3Guard: import does not import empty directory

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • HADOOP-13345
    • fs/s3
    • None
    • Reviewed

    Description

      It imports everything comes up from listFiles, which includes only files (and their parent directories as a side-effect). My first thought on doing this would be to override S3AFileSystem to add an optional parameter to use AcceptAllButSelfAndS3nDirs instead of AcceptFilesOnly. But we could also manually traverse the tree to get all FileStatus objects directory by directory like we do for diff. That's far slower but doesn't add surface area to S3AFileSystem. But there's also the impact to other S3 clients to worry about - I could go either way on that.

      Attachments

        Issue Links

          is related to

          Sub-task - The sub-task of the issue HADOOP-14266 S3Guard: S3AFileSystem::listFiles() to employ MetadataStore

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              mackrorysd Sean Mackrory
              mackrorysd Sean Mackrory
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: