Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18477 Über-jira: S3A Hadoop 3.3.9 features
  3. HADOOP-17961

s3 and abfs incremental listing: use SAX parsers to stream results to list iterators

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.3.2
    • None
    • fs/azure, fs/s3
    • None

    Description

      With code gradually adopting listStatusIncremental(), asking for a smaller initial batch could permit faster ramp of result processing.

      probably most significant on an s3 versioned bucket, as there the need to skip tombstones can result in significantly slower listings -but could benefit ABFS too

      Attachments

        Activity

          People

            Unassigned Unassigned
            stevel@apache.org Steve Loughran
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: