Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11571 Über-jira: S3a stabilisation phase I
  3. HADOOP-11570

S3AInputStream.close() downloads the remaining bytes of the object from S3

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.7.0
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      Currently, S3AInputStream.close() calls S3Object.close(). But, S3Object.close() will read the remaining bytes of the S3 object, potentially transferring a lot of bytes from S3 that are discarded. Instead, the wrapped stream should be aborted to avoid transferring discarded bytes (unless the preceding read() finished at contentLength). For example, reading only the first byte of a 1 GB object and then closing the stream will result in all 1 GB transferred from S3.

        Attachments

        1. HADOOP-11570-002.patch
          2 kB
          Daniel Hecht
        2. HADOOP-11570-001.patch
          2 kB
          Daniel Hecht

          Issue Links

            Activity

              People

              • Assignee:
                dhecht Daniel Hecht
                Reporter:
                dhecht Daniel Hecht
              • Votes:
                1 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: