Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16829 Über-jira: S3A Hadoop 3.4 features
  3. HADOOP-14000

S3guard metadata stores to support millions of entries

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-beta1
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      S3 repos can have millions of child entries

      Currently DirListingMetaData can't and MetadataStore.listChildren(Path path) won't be able to handle directories that big, for listing, deleting or naming.

      We will need a paged response from the listing operation, something which can be iterated over.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated: