[HADOOP-14831] Über-jira: S3a phase IV: Hadoop 3.1 features

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: fs/s3
• Target Version/s: 3.1.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 04/Sep/17 15:20
• Updated: 02/Apr/18 14:56
• Resolved: 02/Apr/18 14:56

Description

All the S3/S3A features targeting Hadoop 3.1

Issue Links

• depends upon: HADOOP-15273 distcp can't handle remote stores with different checksum algorithms
• depends upon: HADOOP-14620 S3A authentication failure for regions other than us-east-1
• depends upon: HADOOP-13786 Add S3A committers for zero-rename commits to S3 endpoints
• depends upon: HADOOP-14531 [Umbrella] Improve S3A error handling & reporting
• depends upon: HADOOP-13204 Über-jira: S3a phase III: scale and tuning
• depends upon: HADOOP-14825 Über-JIRA: S3Guard Phase II: Hadoop 3.1 features
• depends upon: HADOOP-14325 [Umbrella] Stabilise S3A Server Side Encryption
• is superceded by: HADOOP-15220 Über-jira: S3a phase V: Hadoop 3.2 features

Sub-Tasks

• Cut S3AOutputStream: Resolved
• S3A globber to use bulk listObject call over recursive directory scan: Resolved
• S3A delete and rename may fail to preserve parent directory.: Resolved
• s3a home directory to be "/": Resolved
• ITestS3AContractRootDir.testRmEmptyRootDirNonRecursive failing intermittently: Resolved
• S3AInputStream to implement an efficient skip() call through seeking: Resolved
• Intermittent failure of TestS3AContractRootDir.testListEmptyRootDirectory: Resolved
• extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key: Resolved
• s3a input stream "normal" fadvise mode to be adaptive: Resolved
• S3AInputStream/OutputStream statistics aren't getting into StorageStatistics: Resolved
• [s3a] Log StorageStatistics: Resolved
• error message in S3AUtils.getServerSideEncryptionKey() needs to expand property constant: Resolved
• reinstate URI parameter in AWSCredentialProvider constructors: Resolved
• S3 blob etags to be made visible in S3A status/getFileChecksum() calls: Resolved
• Metrics of S3A don't print out when enable it in Hadoop metrics property file: Resolved
• s3a troubleshooting docs to add a couple more failure modes: Resolved
• Enhance S3A troubleshooting documents and add a performance document: Resolved
• S3Guard CLI to support list/purge of pending multipart commits: Resolved
• NPE in S3A getFileStatus: null instrumentation on using closed instance: Resolved
• Support IAM Assumed roles in S3A: Resolved
• ITestS3AFileOperationCost#testFakeDirectoryDeletion failing after OutputCommitter patch: Resolved
• s3a: Stream and common statistics missing from metrics: Resolved
• Enhance IAM Assumed Role support in S3A client: Resolved
• add mapreduce.outputcommitter.factory.scheme.s3a to core-default: Resolved
• Upgrade AWS SDK to 1.11.271: NPE in 1.11.199 bug spams logs w/ Yarn Log Aggregation: Resolved
• Translate AWS SSE-KMS missing key exception to something: Resolved
• S3a "Bad Request" message to include diagnostics: Resolved
• transient failure of ITestCommitOperations.testCommitEmptyFile; fault injection related: Resolved
• Listing s3a bucket without credentials gives Interrupted error: Resolved
• S3a committer factory to warn when default FileOutputFormat committer is created: Resolved
• S3AInputStream to handle reconnect on read() failure better: Resolved
• AWS "shaded" SDK 1.11.271 is pulling in netty 4.1.17: Resolved
• S3A multipart upload fails when SSE-C encryption is enabled: Resolved
• log s3a at info: Resolved
• remove .FluentPropertyBeanIntrospector from CLI operation log output: Resolved
• Make S3A etag => checksum feature optional: Resolved