Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17566 Über-jira: S3A Hadoop 3.4 features
  3. HADOOP-17415

Use S3 content-range header to update length of an object during reads

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.3.0
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      As part of all the openFile work, knowing full length of an object allows for a HEAD to be skipped. But: code knowing only the splits don't know the final length of the file.

      If the content-range header is used, then as soon as a single GET is initiated against an object, if the field is returned then we can update the length of the S3A stream to its real/final length

      Also: when any input stream fails with an EOF exception, we can distinguish stream-interrupted from "no, too far"

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: