[HADOOP-11570] S3AInputStream.close() downloads the remaining bytes of the object from S3

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.7.0
• Component/s: fs/s3
• Target Version/s: 2.7.0
• Assignee: Daniel Hecht
• Reporter: Daniel Hecht
• Created: 10/Feb/15 06:54
• Updated: 03/Sep/15 14:19
• Resolved: 17/Feb/15 16:37

Description

Currently, S3AInputStream.close() calls S3Object.close(). But, S3Object.close() will read the remaining bytes of the S3 object, potentially transferring a lot of bytes from S3 that are discarded. Instead, the wrapped stream should be aborted to avoid transferring discarded bytes (unless the preceding read() finished at contentLength). For example, reading only the first byte of a 1 GB object and then closing the stream will result in all 1 GB transferred from S3.

Issue Links

• is related to: HADOOP-11874 s3a can throw spurious IOEs on close()
• is related to: HADOOP-12376 S3NInputStream.close() downloads the remaining bytes of the object from S3
• relates to: HADOOP-11851 s3n to swallow IOEs on inner stream close