[HADOOP-19353] Über-jira: S3A Hadoop 3.4.2 features

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.1
• Component/s: fs/s3
• Target Version/s: 3.4.2
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 03/Dec/24 11:29
• Updated: 03/Dec/24 13:49

Description

Über-jira for stuff we want into 3.4.2 for s3a connector

Issue Links

• supercedes: HADOOP-18477 Über-jira: S3A Hadoop 3.3.9-3.4.1 features

Sub-Tasks

• Support multipart download in S3AFileSystem: Open
• S3A connector to improve support for all AWS partitions: Open
• S3AFilesystem trash handling should respect the current UGI: Open
• Some S3A tests leak filesystem instances: Open
• S3A AssumedRole credentials provider should use Instance Role credentials in chain for assuming role: Open
• multipart/huge file upload tests to look at checksums returned: Open
• ITestS3A select tests fail if user kinited in: Open
• S3AInputStream.seek should throw EOFException if seeking past the end of file: Open
• s3guard bucket-info command to add a verify-property <key>=<value> <bucket>: Open
• S3a DelegationToken bindings to to support a "correlation ID" for the UA header: Open
• S3a operations keep retrying if the password is wrong: Open
• ITestS3AContractSeek teardown closes test FS before superclass can do its cleanup: Open
• Impersonate hosts in s3a for better data locality handling: Open
• S3A: Support S3 Conditional Writes: Open
• Understand status of S3 access point alias support in S3A: Open
• Use S3 content-range header to update length of an object during reads: Open
• Speed up S3A test runs: Open
• S3A Secret access to fall back to XML if credential provider raises IOE.: Open
• Add common getFileBlockLocations() emulation for object stores, including S3A: Patch Available
• S3A deleteObjects hanging/retrying forever: Open
• shell rm command to not rename to ~/.Trash in object stores: Open
• Clarify committers.md around v2 failure handling: Open
• ITestS3ARemoteFileChanged doesn't overwrite test data creation: Open
• S3A: Test failures with CSE enabled: Resolved
• Add s3a tool to convert S3 server logs to avro/csv files: In Progress
• S3AInputStream logging to make it easier to debug file leakage: Open
• Add new store vendor config option: Open
• S3A: Set thread names with more specific information about the call.: Open
• S3A can support short user-friendly aliases for configuration of credential providers.: Open
• s3guard uploads command to list date and initiator of outstanding uploads: Open
• TestS3AGetFileStatus:testNotFound() to use intercept(): Open
• log accepted/rejected fs.s3a.authoritative.path paths @ debug: Open
• cherry pick s3 ehancements from PrestoS3FileSystem: Open
• S3A: support custom S3 and STS headers: Open
• GCS to support per-bucket configuration: Open
• Support AWS IAM Identity Centre (prev. AWS SSO) for providing credentials to S3A: Open
• strip s3.amazonaws.com off hostnames before making s3a calls: Open
• S3A DeleteOperation to parallelize POSTing of bulk deletes: Open
• S3AInputStream.remainingInFile should use nextReadPos: Reopened
• ITestS3AMiniYarnCluster fails on sequential runs with Kerberos error: Open
• Add a way for an FS instance to say "really, no trash interval at all": Open
• improve s3a committer stats collected: Open
• S3A: add option to disable probe for dir marker recreation on delete/rename.: Open
• test and document use of fs.s3a.signing-algorithm: Open
• Review S3A documentation to make sure it is consistent with the current codebase: Open
• ITestS3AAWSCredentialsProvider tests fail if a bucket has DTs enabled: Open
• S3a auth exception to link to a wiki page on the problem: Open
• review S3A translateException translation matches IBM CORS spec: Open
• S3A input stream to support ByteBufferReadable: Open
• S3A Authentication to support WebIdentity: Open
• S3A to implement rename(final Path src, final Path dst, final Rename... options): Open
• S3ARetryPolicy to handle AWS 500 responses/error code TooBusyException with the throttle backoff policy: Open
• S3A Xattr headers need hdfs-compatible prefix: Open
• Amazon S3 disabling ACLs on all new buckets: Open
• Encrypt S3A buffered data on disk: Open
• S3 Select Exceptions are not being converted to IOEs: Open
• S3AFileStatus to add a serialVersionUID; review & test serialization: Open
• Add S3AWriteOpContext for write ops; pass in statistics and other settings: Open
• S3A DT support to warn when loading expired token: Open
• S3A (async) ObjectListingIterator to block in hasNext() for results: Open
• hadoop-aws tests to take a configurable subdir in the test bucket: Open
• S3A DT marshalling to include nested error text in wrapped message: Open
• clean up ITestS3AFileSystemContract: Patch Available
• S3A Xattr/getXAttr to handle directories without markers: Open
• Add custom InstanceProfileCredentialsProvider with more resilience to throttling: Open
• Support Overwrite Directory On Commit For S3A Committers: Open
• s3 and abfs incremental listing: use SAX parsers to stream results to list iterators: Open
• make s3a read fault injection configurable including "off": Open
• S3A: fs.s3a.connection.request.timeout too low for large uploads over slow links: Reopened
• remove/deprecate fs.s3a.multipart.purge: Open
• s3guard bucket-info command to include default bucket encryption info: Open
• Optimise S3A’s recursive delete to drop successful S3 keys on retry of S3 DeleteObjects: Open
• S3A client retries on SSL Auth exceptions triggered by "." bucket names: Open
• S3A: Allow SSE configurations per object path: Open
• Test MR split optimisation with recursive listing: Open
• s3a rm on the CLI generates deprecation warning on io.bytes.per.checksum: Open
• ITestS3AFileSystemStatistic failure on mvn verify: Open
• add a special 0 byte input stream for empty blobs: Open
• S3A to support configuring various AWS S3 client extended options: Open
• FileSystem/s3a processDeleteOnExit to skip the exists() check: Open
• ITestS3AInputStreamPerformance#testDecompressionSequential128K NPE if no CSV file available: Open
• s3a to improve diags on s3a bad request message: Open
• Handle S3A "glacier" data: Open
• remove filtering of directory markers in s3a RenameOperation: Open
• Filesystem discovery to stop loading implementation classes: Open
• s3a new getdefaultblocksize be called in getFileStatus which has not been implemented in s3afilesystem yet: Open
• s3a listing IOStatistics to count #of entries returned per LIST call: Open
• ITestS3AConfiguration.testProxyConnection failing when s3a bucket probe disabled: Open
• ITestS3ABlockOutputArray failure with IO File name too long: Open
• Add a way to get the IOStatistics of active filesystems in long-lived processes: Open
• NPE in S3AInputStream.read() in ITestS3AInconsistency.testOpenFailOnRead: Open
• Public dataset class for S3A integration tests: Open
• S3AInputStream read(bytes[]) to not retry on read failure: pass action up: Open
• Report problems w/ local S3A buffer directory meaningfully: Open
• test YARN log collection works to s3a: Open
• AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade: Open
• Add "versions" tool to s3a command line entry point: Open
• Tune hadoop-aws parallel test surefire/failsafe settings: Open
• builld up md5 checksum as blocks are built in S3ABlockOutputStream; validate upload: Open
• S3aDelegationTokens to add accessor for tests to get at the token binding: Open
• S3A: ITestS3AFileContextURI: MultiObjectDeleteException bulk delete of odd filenames: Open
• ITestS3ACopyFromLocalFile: AuditFailureException: Open
• s3a client SSLException is raised after very long timeout "Unsupported or unrecognized SSL message": Open
• S3A doesn't calculate Content-MD5 on uploads: Open
• Warn when no region is configured: Open
• Test hadoop fs shell against s3a; fix problems: Open
• Use lighter-weight alternatives to innerGetFileStatus where possible: Open
• support git-secrets commit hook to keep AWS secrets out of git: Patch Available
• increase the default number of threads and http connections in S3A: Open
• S3AInputStream.skip() to use lazy seek: Open
• Possible inconsistent state of AbstractDelegationTokenSecretManager: Patch Available
• S3A Filesystem does not check return from AmazonS3Client deleteObjects: Open
• define s3a encryption behaviour on copy: Open
• Remove fs.s3a.executor.capacity: Open
• ITestCustomSigner uses absolute paths off the bucket root rather than fork-relative: Open
• AWS AssumedRoleCredentialProvider needs ExternalId add: Open
• Remove transient dependency on hadoop-hdfs-client: Open
• Add AWS S3 Transfer acceleration support: Open
• S3A openFile() options to allow etag/version to be set: Reopened
• S3 Express: document use: Open
• S3A third party: document "Certificate doesn't match": Open
• S3A: handle alternative forms of connection failure: Open
• NoSuchMethodError in aws sdk third party logger in hadoop aws 3.4: Open
• S3A: IAMCredentialsProvider throttling results in AWS auth failures: Open
• S3A: ITestCustomSigner failing against S3Express Buckets: Open
• S3A: transfer manager not wired up to s3a executor pool: Open
• Unknown S3Express bucket raises UnknownHostException rather than NoSuchBucketException; will block for retries: Open
• remove head bucket request from calls which can be made without audit header: Open
• S3A Assume role tests failing against S3Express stores: Open
• S3A: Support dynamic region resolution: Open
• S3A HeaderProcessing to process all metadata entries of HEAD response: Open
• S3A : ITestS3AConcurrentOps#testParallelRename intermittent timeout failure: Open
• s3a file rename does double HEAD or LIST on source file/dir: Open
• S3A: retry on credential expiry: Open
• S3A: TestIAMInstanceCredentialsProvider.testIAMInstanceCredentialsInstantiate failure: Open
• AWS SDK V2 - Add socket factory to Netty Client: Open
• AWS SDK V2 - Move to S3 Java async client: Open
• S3A: detect and recover from SSL ConnectionReset exceptions: Open
• S3A: S3A: ITestS3AConfiguration failing with region problems: Open
• S3A: remove @deprecated tags where no longer needed: Open
• AWS SDK V2 - Refactor getS3Region & other follow up items: Open
• coalesce AWS S3 client proxy settings: Open
• S3A: Add LeakReporter; use in S3AInputStream: Resolved
• Move LeakReporter to utils, use more: Open
• Support S3A cross region access when S3 region/endpoint is set: Resolved
• S3A: Add config option to skip test with performance mode: Open
• S3A: terasort tests fail with CSE-kMS enabled and london region With Delegation Token Secrets: Open