[HADOOP-11571] Über-jira: S3a stabilisation phase I

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.7.0
• Component/s: fs/s3
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 10/Feb/15 08:48
• Updated: 05/May/15 16:55
• Resolved: 10/Mar/15 00:12

Description

s3a shipped in 2.6; now its out various corner cases, scale and error handling issues are surfacing.

fix them before 2.7 ships

Issue Links

• depends upon: HADOOP-10714 AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call
• depends upon: HADOOP-11446 S3AOutputStream should use shared thread pool to avoid OutOfMemoryError
• depends upon: HADOOP-11488 Difference in default connection timeout for S3A FS
• depends upon: HADOOP-11171 Enable using a proxy server to connect to S3a.
• depends upon: HADOOP-11261 Set custom endpoint for S3A
• depends upon: HADOOP-11520 Clean incomplete multi-part uploads in S3A tests
• depends upon: HADOOP-11607 Reduce log spew in S3AFileSystem
• depends upon: HADOOP-11463 Replace method-local TransferManager object with S3AFileSystem#transfers
• is depended upon by: HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
• relates to: HADOOP-10400 Incorporate new S3A FileSystem implementation

Sub-Tasks

• S3AInputStream.close() downloads the remaining bytes of the object from S3: Closed
• Make connection timeout configurable in s3a: Closed
• Update S3A Documentation: Closed
• Memory-based S3AOutputstream: Closed
• s3a file block size set to 0 in getFileStatus: Closed
• Regression: s3a auth setup broken: Closed