[HADOOP-16829] Über-jira: S3A Hadoop 3.4 features

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: fs/s3
• Target Version/s: 3.4.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 27/Jan/20 20:09
• Updated: 18/Jun/20 17:01

Description

Über-jira: S3A features/fixes for Hadoop 3.4

As usual, this will clutter up with everything which hasn't gone in: don't interpret presence on this list as a commitment to implement.

And for anyone wanting to add patches

MUST

1. reviews via github PRs
2. no declaration of AWS S3 endpoint (or other S3 impl) -no review

SHOULD

1. have a setup for testing SSE-KMS, DDB/S3Guard
2. including an assumed role we can use for AssumedRole Delegation Tokens

If you are going near those bits of code, they uprate from SHOULD to MUST.

Issue Links

• depends upon: HADOOP-15387 Produce a shaded hadoop-cloud-storage JAR for applications to use
• depends upon: MAPREDUCE-7182 MapReduce input format/record readers to support S3 select queries
• depends upon: HADOOP-15788 Improve Distcp for long-haul/cloud deployments
• depends upon: HADOOP-16604 Provide copy functionality for cloud native applications
• depends upon: MAPREDUCE-6760 LocatedFileStatusFetcher to use listFiles(recursive)
• depends upon: HADOOP-13278 S3AFileSystem mkdirs does not need to validate parent path components
• depends upon: HADOOP-16844 FS/FC createFile() builder to add a lazy-check option for async probes of the store
• depends upon: HADOOP-14630 Contract Tests to verify create, mkdirs and rename under a file is forbidden
• depends upon: MAPREDUCE-5907 Improve getSplits() performance for fs implementations that can utilize performance gains from recursive listing
• is blocked by: HADOOP-16111 FileSystem.closeAllForUGI closes() filesystem in Sync block
• is blocked by: HADOOP-15606 FSDataInputStream/FSInputStream to implement getFileLength()
• supercedes: HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features

Sub-Tasks

• S3AInputStream logging to make it easier to debug file leakage: Open
• Filesystem discovery to stop loading implementation classes: Open
• s3a rename failed during copy, "Unable to copy part" + 200 error code: Open
• make sure staging committers collect DTs for the staging FS: Open
• Remove transient dependency on hadoop-hdfs-client: Open
• s3a directory housekeeping operations to be done in async thread: Open
• S3A input stream to support ByteBufferReadable: Open
• Encrypt S3A buffered data on disk: Open
• S3a: Failed to reset the request input stream/make S3A uploadPart() retriable: Open
• support git-secrets commit hook to keep AWS secrets out of git: Patch Available
• Tune hadoop-aws parallel test surefire/failsafe settings: Open
• S3A Secret access to fall back to XML if credential provider raises IOE.: Open
• Use error code detail in AWS server responses for finer grained exceptions: Open
• Test MR split optimisation with recursive listing: Open
• initial part uploads seem to block unnecessarily in S3ABlockOutputStream: Open
• Speed up S3A test runs: Open
• S3A copy/rename of large files to be parallelized as a multipart operation: Open
• Test hadoop fs shell against s3a; fix problems: Open
• s3guard bucket-info command to add a verify-property <key>=<value> <bucket>: Open
• S3A to implement rename(final Path src, final Path dst, final Rename... options): Open
• Collect AwsSdkMetrics in S3A FileSystem IOStatistics: In Progress
• Add some tests about S3 timestamp tracking: Open
• S3a operations keep retrying if the password is wrong: Open
• Add some S3A-specific create file options: Open
• s3a to set fake directory marker contentType to application/x-directory: Open
• Report problems w/ local S3A buffer directory meaningfully: Open
• S3A Filesystem does not check return from AmazonS3Client deleteObjects: Open
• Optimize uses of FS operations in the ASF analysis frameworks and libraries: Open
• Possible NPE in S3A MultiObjectDeleteSupport error handling: Open
• shell rm command to not rename to ~/.Trash in object stores: Open
• Add S3A support for Async Scatter/Gather IO: Open
• Impersonate hosts in s3a for better data locality handling: Open
• export s3a BlockingThreadPoolExecutorService pool info (size, load) as gauges: Open
• increase the default number of threads and http connections in S3A: Open
• Support AWS S3 reduced redundancy storage class: Open
• s3a doesn't consider blobs with trailing / and content-length >0 as directories: Open
• make s3a read fault injection configurable including "off": Open
• strip s3.amazonaws.com off hostnames before making s3a calls: Open
• add a special 0 byte input stream for empty blobs: Open
• improve setting of max connections in AWS client: Open
• multipart/huge file upload tests to look at checksums returned: Open
• Some S3A tests leak filesystem instances: Open
• s3guard to provide better diags on ddb init failures: Open
• Improve isolation of FS instances in S3A committer tests: Open
• s3a rm on the CLI generates deprecation warning on io.bytes.per.checksum: Open
• S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use: Open
• review S3A translateException translation matches IBM CORS spec: Open
• Add fs.s3a.rename.raises.exceptions to raise exceptions on rename failures: Open
• add tests/docs for HAR files on s3a: Open
• S3A to use a thread pool for async path operations: Open
• builld up md5 checksum as blocks are built in S3ABlockOutputStream; validate upload: Open
• FileSystem/s3a processDeleteOnExit to skip the exists() check: Open
• Add custom InstanceProfileCredentialsProvider with more resilience to throttling: Open
• ITestS3AMiniYarnCluster fails on sequential runs with Kerberos error: Open
• S3a DelegationToken bindings to to support a "correlation ID" for the UA header: Open
• Stabilize openFile() and adopt internally: In Progress
• S3AInputStream.skip() to use lazy seek: Open
• Document `dynamodb:TagResource` an explicit client-side permission for S3Guard: Open
• ITestS3ARemoteFileChanged doesn't overwrite test data creation: Open
• AWS Data read stack trace in S3a putObjectDirect: Open
• AmazonClient 30x exceptions to include redirect URL in message: Open
• S3A FS to add "s3a:no-existence-checks" to the builder file creation option set: Open
• s3a new getdefaultblocksize be called in getFileStatus which has not been implemented in s3afilesystem yet: Open
• Use lighter-weight alternatives to innerGetFileStatus where possible: Open
• intermittent failure of ITestS3GuardListConsistency.testInconsistentS3ClientDeletes in parallel runs: Open
• ITestDynamoDBMetadataStore.testTableVersioning failure -DDB deleteItem consistency?: Open
• Add S3AWriteOpContext for write ops; pass in statistics and other settings: Open
• S3A: Set thread names with more specific information about the call.: Open
• Add AWS S3 Transfer acceleration support: Open
• Add a way for an FS instance to say "really, no trash interval at all": Open
• s3guard uploads command to list date and initiator of outstanding uploads: Open
• test and document use of fs.s3a.signing-algorithm: Open
• S3AInputStream.seek should throw EOFException if seeking past the end of file: Open
• S3a auth exception to link to a wiki page on the problem: Open
• s3a to improve diags on s3a bad request message: Open
• AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade: Open
• ITestS3AContractRootDir failure on non-S3Guarded bucket: Open
• s3guard bucket-info command to include default bucket encryption info: Open
• S3 SSEC tests to downgrade when running against a mandatory encryption object store: Open
• S3AInputStream read(bytes[]) to not retry on read failure: pass action up: Open
• typo in TestNeworkBinding: Open
• ITestS3A select tests fail if user kinited in: Open
• S3A add histogram metrics types for latency, etc.: Open
• cherry pick s3 ehancements from PrestoS3FileSystem: Open
• S3aDelegationTokens to add accessor for tests to get at the token binding: Open
• s3guard LimitExceededException -too many tables: Open
• Support multipart download in S3AFileSystem: Open
• NPE in S3AInputStream.read() in ITestS3AInconsistency.testOpenFailOnRead: Open
• S3A DT marshalling to include nested error text in wrapped message: Open
• S3 Select Exceptions are not being converted to IOEs: Open
• remove misleading fs.s3a.delegation.tokens.enabled prompt: Open
• S3A to support configuring various AWS S3 client extended options: Open
• Review S3A documentation to make sure it is consistent with the current codebase: Open
• S3A DT support to warn when loading expired token: Open
• ITestS3AAWSCredentialsProvider tests fail if a bucket has DTs enabled: Open
• Clarify committers.md around v2 failure handling: Open
• S3AFileStatus to add a serialVersionUID; review & test serialization: Open
• test YARN log collection works to s3a: Open
• Encrypt S3A data client-side with AWS SDK (S3-CSE): Patch Available
• S3A can support short user-friendly aliases for configuration of credential providers.: Open
• Handle S3A "glacier" data: Open
• Add common getFileBlockLocations() emulation for object stores, including S3A: Patch Available
• S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes: Patch Available
• S3A to support Requester Pays Buckets: Patch Available
• clean up ITestS3AFileSystemContract: Patch Available
• ITestS3AContractGetFileStatusV1List may have consistency issues: Open
• S3ARetryPolicy to handle AWS 500 responses/error code TooBusyException with the throttle backoff policy: Open
• Add some Abortable.abort() interface for streams etc which can be terminated: Open
• S3A mkdirs to indicate which parent path element refers to a file: Open
• NPE in s3a byte buffer block upload: Open
• declare that fs.s3a.ext. is a prefix for arbitrary extensions: Open
• AWS AssumedRoleCredentialProvider needs ExternalId add: Open
• IAM role created by S3A DT doesn't include DynamoDB scan: Open
• S3AFilesystem trash handling should respect the current UGI: Open
• S3AInputStream.remainingInFile should use nextReadPos: Reopened
• s3guard can't init table if caller doesn't have tag permissions: Open
• ITestS3AContractSeek teardown closes test FS before superclass can do its cleanup: Open
• Possible inconsistent state of AbstractDelegationTokenSecretManager: Patch Available
• ITestCustomSigner uses absolute paths off the bucket root rather than fork-relative: Open
• log accepted/rejected fs.s3a.authoritative.path paths @ debug: Open
• Add option for a prefix to put in front of every s3guard table: Open
• Add more s3guard metrics: Open
• Improve DynamoDB schema update story: Open
• S3Guard: Optimize performance of handling OOB operations in non-authoritative mode: Open
• increase performance of s3guard import command: Open
• intermittent failure of ITestCommitOperations: too many s3guard writes: Open
• Possible for modified configuration to leak into metadatastore in S3GuardTool: Open
• S3AFileSystem copyFile to propagate etag/version from getObjectMetadata to copy request: Open
• Intermittent failure of ITestS3GuardConcurrentOps#testConcurrentTableCreations: Open
• S3Guard init command uses global settings, not those of target bucket: Reopened
• S3guard: add inconsistency detection metrics: Open
• S3Guard prune to only remove auth dir marker if files (not tombstones) are removed: Open
• Fix ITestS3GuardToolLocal#testInitNegativeRead test failure: Open
• Ensure controls in-place to prevent clients with significant clock skews pruning aggressively: Open
• Scheme assertion in S3Guard DynamoDBMetadataStore::checkPath is unnecessarily restrictive: Open
• improvements to S3GuardTool destroy command: Open
• ITestS3GuardToolDynamoDB.testDynamoDBInitDestroyCycle fails if test bucket isn't on demand: Open
• S3Guard to self update on directory listings of S3: Open
• S3guard mistakes root URI without / as non-absolute path: Open
• S3A deleteObjects hanging/retrying forever: Open
• S3A delegation token binding to support secondary binding list: In Progress
• whitespace not allowed in paths when saving files to s3a via committer: Open
• Re-enable optimized copyFromLocal implementation in S3AFileSystem: Open
• ITestS3GuardOutOfBandOperations testListingDelete[auth=false] fails on unversioned bucket: Open
• S3A client retries on SSL Auth exceptions triggered by "." bucket names: Open
• Support S3 Access Points: Open
• ITestS3AConfiguration.testProxyConnection failing when s3a bucket probe disabled: Open
• Failure of ITestAssumeRole.testRestrictedCommitActions: Open
• S3A (async) ObjectListingIterator to block in hasNext() for results: Open
• Distcp to set S3 Storage Class: Open
• transient ITestS3AFileContextStatistics failure -read buffer not filled: Open
• hadoop-cloud-storage transient dependencies need review: Open
• S3A AWS Credential provider loading gets confused with isolated classloaders: Open
• ITestS3AContractSeek.teardown closes FS before superclass does its cleanup: Open
• fs.s3a.buffer.dir to be under Yarn container path on yarn applications: Open
• S3A ITestPartialRenamesDeletes.testRenameDirFailsInDelete failure: missing directory marker: Reopened
• enable s3a magic committer by default: Open
• GCS to support per-bucket configuration: Open
• Use S3 content-range header to update length of an object during reads: Open
• S3A to treat "SdkClientException: Data read has a different length than the expected" as EOFException: Open
• ITestAssumeRole.testAssumeRoleBadInnerAuth failure: Open
• magic committer to be enabled for all S3 buckets: Open
• Add a MkdirOperation for chained S3 operations during mkdir: Open