Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16829 Über-jira: S3A Hadoop 3.3.1 features
  3. HADOOP-14000

S3guard metadata stores to support millions of entries

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 3.0.0-beta1
    • None
    • fs/s3
    • None

    Description

      S3 repos can have millions of child entries

      Currently DirListingMetaData can't and MetadataStore.listChildren(Path path) won't be able to handle directories that big, for listing, deleting or naming.

      We will need a paged response from the listing operation, something which can be iterated over.

      Attachments

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue HADOOP-13926 S3Guard: S3AFileSystem::listLocatedStatus() to employ MetadataStore

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Sub-task - The sub-task of the issue HADOOP-14936 S3Guard: remove "experimental" from documentation

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-16184 S3Guard: Handle OOB deletions and creation of a file which has a tombstone marker

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HADOOP-14159 Add some Java-8 friendly way to work with RemoteIterable, especially listings

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: