[HADOOP-18067] Über-jira: S3A Hadoop 3.3.5 features

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.2
• Fix Version/s: 3.3.5
• Component/s: fs/s3
• Target Version/s: 3.3.5
• Assignee: Mukund Thakur
• Reporter: Steve Loughran
• Created: 05/Jan/22 15:32
• Updated: 21/Dec/22 12:18
• Resolved: 21/Dec/22 12:18

Description

Everything for s3a connector in hadoop 3.3.5.

followed up by HADOOP-18477

Issue Links

• is superceded by: HADOOP-18477 Über-jira: S3A Hadoop 3.3.9 features
• supercedes: HADOOP-17566 Über-jira: S3A Hadoop 3.3.2 features

Sub-Tasks

• ITestS3AContractSeek.teardown closes FS before superclass does its cleanup: Resolved
• fs.s3a.buffer.dir to be under Yarn container path on yarn applications: Resolved
• remove misleading fs.s3a.delegation.tokens.enabled prompt: Resolved
• Improve isolation of FS instances in S3A committer tests: Resolved
• Abfs and S3A FileContext bindings to close wrapped filesystems in finalizer: Resolved
• S3A to use a thread pool for async path operations: Resolved
• Distcp to set S3 Storage Class: Resolved
• Add S3A support for Async Scatter/Gather IO: Resolved
• S3 SSEC tests to downgrade when running against a mandatory encryption object store: Resolved
• S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use: Resolved
• Support configuration of different S3 storage classes: Resolved
• S3A to support Requester Pays Buckets: Resolved
• Upgrade AWS SDK to 1.12.132: Resolved
• S3Object must be closed/ref set to null in S3AInputStream.closeStream(): Resolved
• S3A to downgraded to unguarded if the DDB table cannot be found: Resolved
• Remove S3Guard - no longer needed: Resolved
• S3A auditing leaks memory through ThreadLocal references: Resolved
• Implement paging during S3 multi object delete.: Resolved
• ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged: Resolved
• distcp to use openFile() with sequential IO; ranges of reads: Resolved
• S3A storage class option only picked up when buffering writes to disk: Resolved
• AWS SDK update to 1.12.262 to address jackson CVE-2018-7489 and AWS CVE-2022-31159: Resolved
• HEAD OBJECT returns only 400 BAD REQUEST when token is expired: Resolved
• ITestS3Select.testSelectSeekFullLandsat is timing out: Resolved
• ITestCustomSigner fails when access point name has '-': Resolved
• s3a FS init logs at warn if fs.s3a.create.storage.class is unset: Resolved
• ILoadTestS3ABulkDeleteThrottling failing: Resolved
• Fix FileSystem leak in ITestS3AAWSCredentialsProvider: Resolved
• S3AInputStream.unbuffer() async drain not releasing http connections: Resolved
• ITestS3AIOStatisticsContext failure: Resolved
• s3a endpoint per bucket configuration in pyspark is ignored: Resolved
• S3A FS to add "fs.s3a.create.performance" to the builder file creation option set: Resolved
• upgrade AWS SDK to 1.12.316: Resolved
• Improve S3A committers documentation clarity: Resolved
• Add some S3A-specific create file options: Resolved
• AvroFSInput opens a stream twice and discards the second one without closing: Resolved
• s3a magic committer may commit more data: Resolved
• S3A to support user-specified content encoding: Resolved
• S3A server-side encryption tests fail before checking encryption tests should skip: Resolved
• Leak of S3AInstrumentation instances via hadoop Metrics references: Resolved