[HADOOP-18477] Über-jira: S3A Hadoop 3.3.9 features

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.5
• Component/s: fs/s3
• Target Version/s: 3.3.9
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 04/Oct/22 16:21
• Updated: 22/Nov/23 14:03

Description

Changes related to s3a in the next branch-3.3 release.
Presence in this list != any commitment to implement, unless there's active dev

Issue Links

• depends upon: HADOOP-18028 High performance S3A input stream with prefetching & caching
• depends upon: HADOOP-18179 Boost S3A Stream Read Performance
• depends upon: HADOOP-18073 S3A: Upgrade AWS SDK to V2
• supercedes: HADOOP-18067 Über-jira: S3A Hadoop 3.3.5 features

Sub-Tasks

• ITestCustomSigner uses absolute paths off the bucket root rather than fork-relative: Open
• S3A input stream to support ByteBufferReadable: Open
• Tune hadoop-aws parallel test surefire/failsafe settings: Open
• Impersonate hosts in s3a for better data locality handling: Open
• ITestS3AContractRootDir failure on non-S3Guarded bucket: Resolved
• Document `dynamodb:TagResource` an explicit client-side permission for S3Guard: Resolved
• Public dataset class for S3A integration tests: Open
• Some S3A tests leak filesystem instances: Open
• s3 and abfs incremental listing: use SAX parsers to stream results to list iterators: Open
• builld up md5 checksum as blocks are built in S3ABlockOutputStream; validate upload: Open
• Use S3 content-range header to update length of an object during reads: Open
• S3a operations keep retrying if the password is wrong: Open
• S3A: Set thread names with more specific information about the call.: Open
• NPE in S3AInputStream.read() in ITestS3AInconsistency.testOpenFailOnRead: Open
• ITestS3ARemoteFileChanged doesn't overwrite test data creation: Open
• test and document use of fs.s3a.signing-algorithm: Open
• s3a listing IOStatistics to count #of entries returned per LIST call: Open
• S3A delegation token binding to support secondary binding list: Resolved
• S3A DT marshalling to include nested error text in wrapped message: Open
• increase the default number of threads and http connections in S3A: Open
• clean up ITestS3AFileSystemContract: Patch Available
• S3A to implement rename(final Path src, final Path dst, final Rename... options): Open
• ITestS3AInputStreamPerformance#testDecompressionSequential128K NPE if no CSV file available: Open
• hadoop-cloud-storage transient dependencies need review: Resolved
• Possible inconsistent state of AbstractDelegationTokenSecretManager: Patch Available
• S3A DT support to warn when loading expired token: Open
• s3a rm on the CLI generates deprecation warning on io.bytes.per.checksum: Open
• Clarify committers.md around v2 failure handling: Open
• Handle S3A "glacier" data: Open
• Test MR split optimisation with recursive listing: Open
• s3a to improve diags on s3a bad request message: Open
• Support AWS IAM Identity Centre (prev. AWS SSO) for providing credentials to S3A: Open
• NPE in s3a byte buffer block upload: Resolved
• S3AInputStream logging to make it easier to debug file leakage: Open
• ITestS3AConfiguration.testProxyConnection failing when s3a bucket probe disabled: Open
• support git-secrets commit hook to keep AWS secrets out of git: Patch Available
• S3A doesn't calculate Content-MD5 on uploads: Open
• cherry pick s3 ehancements from PrestoS3FileSystem: Open
• S3A deleteObjects hanging/retrying forever: Open
• Understand status of S3 access point alias support in S3A: Open
• Add custom InstanceProfileCredentialsProvider with more resilience to throttling: Open
• S3A connector to improve support for all AWS partitions: Open
• GCS to support per-bucket configuration: Open
• S3aDelegationTokens to add accessor for tests to get at the token binding: Open
• ITestDynamoDBMetadataStore.testTableVersioning failure -DDB deleteItem consistency?: Resolved
• S3 Select Exceptions are not being converted to IOEs: Open
• compatibility table in directory_markers.md doesn't render right: Resolved
• s3a new getdefaultblocksize be called in getFileStatus which has not been implemented in s3afilesystem yet: Open
• S3A Xattr/getXAttr to handle directories without markers: Open
• Support multipart download in S3AFileSystem: Open
• review S3A translateException translation matches IBM CORS spec: Open
• add a special 0 byte input stream for empty blobs: Open
• Remove fs.s3a.executor.capacity: Open
• Optimise S3A’s recursive delete to drop successful S3 keys on retry of S3 DeleteObjects: Open
• S3A (async) ObjectListingIterator to block in hasNext() for results: Open
• S3A: Allow SSE configurations per object path: Open
• strip s3.amazonaws.com off hostnames before making s3a calls: Open
• Filesystem discovery to stop loading implementation classes: Open
• shell rm command to not rename to ~/.Trash in object stores: Open
• Use lighter-weight alternatives to innerGetFileStatus where possible: Open
• Speed up S3A test runs: Open
• FileSystem/s3a processDeleteOnExit to skip the exists() check: Open
• Remove transient dependency on hadoop-hdfs-client: Open
• s3guard bucket-info command to add a verify-property <key>=<value> <bucket>: Open
• S3A Secret access to fall back to XML if credential provider raises IOE.: Open
• ITestS3A select tests fail if user kinited in: Open
• ITestS3AContractSeek teardown closes test FS before superclass can do its cleanup: Open
• Add s3a tool to convert S3 server logs to avro/csv files: Open
• abfs & s3a FS instantiate triggers warning about deprecated io.bytes.per.checksum: Resolved
• remove filtering of directory markers in s3a RenameOperation: Open
• S3AFilesystem trash handling should respect the current UGI: Open
• S3A can support short user-friendly aliases for configuration of credential providers.: Open
• S3AFileStatus to add a serialVersionUID; review & test serialization: Open
• Add common getFileBlockLocations() emulation for object stores, including S3A: Patch Available
• S3A Filesystem does not check return from AmazonS3Client deleteObjects: Open
• S3A client retries on SSL Auth exceptions triggered by "." bucket names: Open
• s3guard uploads command to list date and initiator of outstanding uploads: Open
• Address JavaDoc warnings in classes like MarkerTool, S3ObjectAttributes, etc.: Resolved
• S3ARetryPolicy to handle AWS 500 responses/error code TooBusyException with the throttle backoff policy: Open
• S3AInputStream.skip() to use lazy seek: Open
• s3guard bucket-info command to include default bucket encryption info: Open
• S3A DeleteOperation to parallelize POSTing of bulk deletes: Open
• ITestCustomSigner fails with NPE against private endpoint: Resolved
• S3a DelegationToken bindings to to support a "correlation ID" for the UA header: Open
• Review S3A documentation to make sure it is consistent with the current codebase: Open
• S3AInputStream.remainingInFile should use nextReadPos: Reopened
• Add S3AWriteOpContext for write ops; pass in statistics and other settings: Open
• log accepted/rejected fs.s3a.authoritative.path paths @ debug: Open
• typo in TestNeworkBinding: Resolved
• S3A to support configuring various AWS S3 client extended options: Open
• Add AWS S3 Transfer acceleration support: Open
• Add a way for an FS instance to say "really, no trash interval at all": Open
• AWS AssumedRoleCredentialProvider needs ExternalId add: Open
• Report problems w/ local S3A buffer directory meaningfully: Open
• ITestS3AMiniYarnCluster fails on sequential runs with Kerberos error: Open
• test YARN log collection works to s3a: Open
• Encrypt S3A buffered data on disk: Open
• S3AInputStream read(bytes[]) to not retry on read failure: pass action up: Open
• make s3a read fault injection configurable including "off": Open
• S3a auth exception to link to a wiki page on the problem: Open
• Improve S3A delegations token documentation: Resolved
• Test hadoop fs shell against s3a; fix problems: Open
• AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade: Open
• S3AInputStream.seek should throw EOFException if seeking past the end of file: Open
• multipart/huge file upload tests to look at checksums returned: Open
• TestS3AGetFileStatus:testNotFound() to use intercept(): Open
• ITestS3AAWSCredentialsProvider tests fail if a bucket has DTs enabled: Open
• ITestS3APrefetchingInputStream does not skip if no CSV test file available: Resolved
• hadoop-aws tests to take a configurable subdir in the test bucket: Open
• support multiple s3a integration test runs on same bucket in parallel: In Progress
• S3A: add option to disable probe for dir marker recreation on delete/rename.: Open
• New configuration for static headers to be added to all S3 requests: Open
• Warn when no region is configured: Open
• S3A AssumedRole credentials provider should use Instance Role credentials in chain for assuming role: Open
• define s3a encryption behaviour on copy: Open
• S3A openFile() options to allow etag/version to be set: Reopened
• improve s3a committer stats collected: Open
• Add "versions" tool to s3a command line entry point: Open
• Add API for bulk/paged object deletion: Open
• Add new store vendor config option: Open
• Amazon S3 disabling ACLs on all new buckets: Open
• S3A audit header to include count of items in delete ops: Resolved
• S3A ITestS3ABucketExistence access point test failure: Resolved
• S3ABlockOutputStream doesn't collect stats on multipart block uploads: Resolved
• S3A Authentication to support WebIdentity: Open
• ITestS3ABlockOutputArray failure with IO File name too long: Open
• Change fs.s3a.directory.marker.retention to "keep": Resolved
• Upgrade aws-java-sdk to 1.12.367+: Resolved
• Test failures with CSE enabled: Open
• ITestS3AFileSystemStatistic failure on mvn verify: Open
• Make it possible to disable ITestS3ATemporaryCredentials: Resolved
• ITestS3ACopyFromLocalFile: AuditFailureException: Open
• Upgrade aws-java-sdk to 1.12.499+: Resolved
• Add a way to get the IOStatistics of active filesystems in long-lived processes: Open
• Add ability to configure ConnectionTTL of http connections while creating S3 Client.: Resolved
• s3a client SSLException is raised after very long timeout "Unsupported or unrecognized SSL message": Open
• Support Overwrite Directory On Commit For S3A Committers: Open
• remove/deprecate fs.s3a.multipart.purge: Open
• ITestS3GuardTool fails if SSE/DSSE encryption is used: Resolved
• S3A: Upgrade AWS SDK version to 2.21.33 for Amazon S3 Express One Zone support: Resolved
• fs.getXattrs(path) for S3FS doesn't have x-amz-server-side-encryption-aws-kms-key-id header.: Open
• S3A: ITestS3AFileContextURI: MultiObjectDeleteException bulk delete of odd filenames: Open
• Assist reflection IO with WrappedOperations class: Open