[HADOOP-17566] Über-jira: S3A Hadoop 3.3.2 features

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.3.1
• Fix Version/s: 3.3.2
• Component/s: fs/s3
• Target Version/s: 3.3.2
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 04/Mar/21 17:29
• Updated: 05/Jan/22 17:05
• Resolved: 05/Jan/22 17:05

Issue Links

• is superceded by: HADOOP-18067 Über-jira: S3A Hadoop 3.3.5 features
• supercedes: HADOOP-16829 Über-jira: S3A Hadoop 3.3.1 features

Sub-Tasks

• Add an Audit plugin point for S3A auditing/context: Resolved
• S3A to treat "SdkClientException: Data read has a different length than the expected" as EOFException: Resolved
• Add a MkdirOperation for chained S3 operations during mkdir: Resolved
• Magic committer to downgrade abort in cleanup if list uploads fails with access denied: Resolved
• S3Guard import can OOM on large imports: Resolved
• Support S3 Access Points: Resolved
• Failure of ITestAssumeRole.testRestrictedCommitActions: Resolved
• Re-enable optimized copyFromLocal implementation in S3AFileSystem: Resolved
• S3A AWS Credential provider loading gets confused with isolated classloaders: Resolved
• S3A mkdirs to indicate which parent path element refers to a file: Resolved
• transient ITestS3AFileContextStatistics failure -read buffer not filled: Resolved
• s3a rename failed during copy, "Unable to copy part" + 200 error code: Resolved
• S3a: Failed to reset the request input stream/make S3A uploadPart() retriable: Resolved
• S3AFileSystem copyFile to propagate etag/version from getObjectMetadata to copy request: Resolved
• ITestS3AContractGetFileStatusV1List may have consistency issues: Resolved
• Distcp is unable to determine region with S3 PrivateLink endpoints: Resolved
• s3a to set fake directory marker contentType to application/x-directory: Resolved
• S3A copy/rename of large files to be parallelized as a multipart operation: Resolved
• S3A add histogram metrics types for latency, etc.: Resolved
• s3guard to provide better diags on ddb init failures: Resolved
• s3a doesn't consider blobs with trailing / and content-length >0 as directories: Resolved
• Encrypt S3A data client-side with AWS SDK (S3-CSE): Resolved
• declare that fs.s3a.ext. is a prefix for arbitrary extensions: Resolved
• improve setting of max connections in AWS client: Resolved
• AWS Data read stack trace in S3a putObjectDirect: Resolved
• increase performance of s3guard import command: Resolved
• S3A to add option fs.s3a.endpoint.region to set AWS region: Resolved
• Upgrade aws-java-sdk to 1.11.1026: Resolved
• S3A NetworkBinding has a runtime class dependency on a third-party shaded class: Resolved
• S3AFS creation fails "Unable to find a region via the region provider chain.": Resolved
• bytesRead FS statistic showing twice the correct value in S3A: Resolved
• S3A ITestPartialRenamesDeletes.testRenameDirFailsInDelete failure: missing directory marker: Resolved
• ITestS3AContractDistCp failing IllegalStateException: WFOPENSSL0030 Running handshake with buffered unwrapped data: Resolved
• fs.s3a.acl.default not working after S3A Audit feature added: Resolved
• S3A Tests to skip if S3Guard and S3-CSE are enabled.: Resolved
• S3A CSE: minor tuning: Resolved
• CredentialProviderFactory.getProviders() recursion loading JCEKS file from s3a: Resolved
• s3a: set fs.s3a.downgrade.syncable.exceptions = true by default: Resolved
• AccessPoint verifyBucketExistsV2 always returns false: Resolved