[HADOOP-16829] Über-jira: S3A Hadoop 3.3.1 features

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.1
• Component/s: fs/s3
• Target Version/s: 3.3.1
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 27/Jan/20 20:09
• Updated: 01/Jun/21 16:46
• Resolved: 13/May/21 04:25

Description

Über-jira: S3A features/fixes for Hadoop 3.4

As usual, this will clutter up with everything which hasn't gone in: don't interpret presence on this list as a commitment to implement.

And for anyone wanting to add patches

MUST

1. reviews via github PRs
2. no declaration of AWS S3 endpoint (or other S3 impl) -no review

SHOULD

1. have a setup for testing SSE-KMS, DDB/S3Guard
2. including an assumed role we can use for AssumedRole Delegation Tokens

If you are going near those bits of code, they uprate from SHOULD to MUST.

Issue Links

• depends upon: HADOOP-15387 Produce a shaded hadoop-cloud-storage JAR for applications to use
• depends upon: MAPREDUCE-7182 MapReduce input format/record readers to support S3 select queries
• depends upon: HADOOP-15788 Improve Distcp for long-haul/cloud deployments
• depends upon: HADOOP-16604 Provide copy functionality for cloud native applications
• depends upon: MAPREDUCE-6760 LocatedFileStatusFetcher to use listFiles(recursive)
• depends upon: HADOOP-13278 S3AFileSystem mkdirs does not need to validate parent path components
• depends upon: HADOOP-16844 FS/FC createFile() builder to add a lazy-check option for async probes of the store
• depends upon: HADOOP-14630 Contract Tests to verify create, mkdirs and rename under a file is forbidden
• depends upon: MAPREDUCE-5907 Improve getSplits() performance for fs implementations that can utilize performance gains from recursive listing
• is blocked by: HADOOP-16111 FileSystem.closeAllForUGI closes() filesystem in Sync block
• is blocked by: HADOOP-15606 FSDataInputStream/FSInputStream to implement getFileLength()
• is blocked by: HADOOP-17519 Decommission S3Guard
• is superceded by: HADOOP-17566 Über-jira: S3A Hadoop 3.4 features
• supercedes: HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features

Sub-Tasks

• make sure staging committers collect DTs for the staging FS: Resolved
• s3a test can hang in teardown with network problems: Resolved
• distcp -update to S3A; abfs, etc always overwrites due to block size mismatch: Resolved
• S3AFileSystem.getContentSummary() to use listFiles(recursive): Resolved
• add extra S3A MPU test to see what happens if a file is created during the MPU: Resolved
• S3A doesn't actually verify paths have the correct authority: Resolved
• Use error code detail in AWS server responses for finer grained exceptions: Resolved
• initial part uploads seem to block unnecessarily in S3ABlockOutputStream: Resolved
• S3AFilesystem.initiateRename() can skip check on dest.parent status if src has same parent: Resolved
• Collect AwsSdkMetrics in S3A FileSystem IOStatistics: Resolved
• Add some tests about S3 timestamp tracking: Resolved
• Optimize uses of FS operations in the ASF analysis frameworks and libraries: Resolved
• log DNS addresses on s3a init: Resolved
• export s3a BlockingThreadPoolExecutorService pool info (size, load) as gauges: Resolved
• S3A authenticators to log origin of .secret.key options: Resolved
• Retrive modtime of PUT file from store, via response or HEAD: Resolved
• s3a create() doesn't check for an ancestor path being a file: Resolved
• Improve S3A rename resilience: Resolved
• add tests/docs for HAR files on s3a: Resolved
• add experimental optimization of s3a directory marker handling: Resolved
• Report on S3A cached 404 recovery better: Resolved
• Stabilize openFile() and adopt internally: In Progress
• s3a to instrument duration of HTTP calls: Resolved
• catch and downgrade all exceptions trying to load openssl native libs through wildfly: Resolved
• Optimize getFileStatus in S3A: Resolved
• s3a mkdir path/ can add 404 to S3 load balancers: Resolved
• AmazonClient 30x exceptions to include redirect URL in message: Resolved
• S3A Input Stream bytes read counter isn't getting through to StorageStatistics/instrumentation properly: Resolved
• job commit failure in S3A MR magic committer test: Resolved
• intermittent failure of ITestS3GuardListConsistency.testInconsistentS3ClientDeletes in parallel runs: Resolved
• S3A Client to add explicit support for versioned stores: Resolved
• S3A statistic collection underrecords bytes written in helper threads: Resolved
• s3a: auto-detect region for bucket and use right endpoint: Resolved
• s3guard LimitExceededException -too many tables: Resolved
• s3a directory housekeeping operations to be done in async thread: Resolved
• ITestS3ARemoteFileChanged tests fail if you set the bucket to versionid tracking: Resolved
• Add some Java-8 friendly way to work with RemoteIterable, especially listings: Resolved
• S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes: Resolved
• s3a mkdirs() to not check dest for a dir marker: Resolved
• S3A FS deleteOnExit to skip the exists check: Resolved
• ITestAssumeRole.testAssumeRoleBadInnerAuth failure: Resolved
• S3A to optionally retain directory markers: Resolved
• Add some Abortable.abort() interface for streams etc which can be terminated: Resolved
• IAM role created by S3A DT doesn't include DynamoDB scan: Resolved
• S3A FullCredentialsTokenBinding fails if local credentials are unset: Resolved
• s3guard can't init table if caller doesn't have tag permissions: Resolved
• ITestS3GuardOutOfBandOperations.testListingDelete failing on versioned bucket: Resolved
• Add option for a prefix to put in front of every s3guard table: Resolved
• Add more s3guard metrics: Resolved
• Improve DynamoDB schema update story: Resolved
• S3Guard: Optimize performance of handling OOB operations in non-authoritative mode: Resolved
• getFileChecksum() needs to adopt S3Guard: Resolved
• reduce/tune read failure fault injection on inconsistent client: Resolved
• intermittent failure of ITestCommitOperations: too many s3guard writes: Resolved
• S3a getFileStatus to update DDB if an S3 query returns etag/versionID: Resolved
• Possible for modified configuration to leak into metadatastore in S3GuardTool: Resolved
• S3Guard instrumentation to include cost of DynamoDB ops as metric: Resolved
• Intermittent failure of ITestS3GuardConcurrentOps#testConcurrentTableCreations: Resolved
• Intermittent failure of ITestS3GuardToolDynamoDB#testDynamoDBInitDestroyCycle: Resolved
• S3Guard init command uses global settings, not those of target bucket: Resolved
• Improve throttling on S3Guard DDB batch retries: Resolved
• S3guard: add inconsistency detection metrics: Resolved
• S3Guard prune to only remove auth dir marker if files (not tombstones) are removed: Resolved
• ZookeeperMetadataStore: Use Zookeeper as S3Guard backend store: Resolved
• tag S3GuardTool entry points as limitedPrivate("management-tools")/evolving: Resolved
• Fix ITestS3GuardToolLocal#testInitNegativeRead test failure: Resolved
• Ensure controls in-place to prevent clients with significant clock skews pruning aggressively: Resolved
• Scheme assertion in S3Guard DynamoDBMetadataStore::checkPath is unnecessarily restrictive: Resolved
• improvements to S3GuardTool destroy command: Resolved
• Clock skew can cause S3Guard to think object metadata is out of date: Resolved
• S3guard metadata stores to support millions of entries: Resolved
• ITestS3GuardToolDynamoDB.testDynamoDBInitDestroyCycle fails if test bucket isn't on demand: Resolved
• S3Guard to self update on directory listings of S3: Resolved
• S3guard mistakes root URI without / as non-absolute path: Resolved
• HADOOP-16953. tune s3guard disabled warnings: Resolved
• mkdir on s3a should not be sensitive to trailing '/': Resolved
• s3a to not need wildfly on the classpath: Resolved
• ITestS3AConfiguration proxy tests fail when bucket probes == 0: Resolved
• S3A to support additional token issuers: Resolved
• S3A staging committer committing duplicate files: Resolved
• ITestS3GuardOutOfBandOperations testListingDelete[auth=false] fails on unversioned bucket: Resolved
• S3AFileSystem silently deletes "fake" directories when writing a file.: Resolved
• ITestS3AEncryptionWithDefaultS3Settings fails if default bucket encryption != KMS: Resolved
• Handle transient stream read failures in FileSystem contract tests: Resolved
• add way for s3a to recognise buckets with "." in name and switch to path access: Resolved
• Decrease size of s3a dependencies: Resolved
• Backport HADOOP-13230 list/getFileStatus changes for preserved directory markers: Resolved
• Renaming a file under a sibling empty directory doesn't delete dest dir's marker: Resolved
• improve s3guard markers command line tool: Resolved
• Backport HADOOP-13230 listing changes for preserved directory markers to 3.1.x: Resolved
• HADOOP-17244. S3A directory delete tombstones dir markers prematurely.: Resolved
• s3a rename() now requires s3:deleteObjectVersion permission: Resolved
• S3A to always probe S3 in S3A getFileStatus on non-auth paths: Resolved
• ITestCustomSigner fails with gcs s3 compatible endpoint.: Resolved
• S3AInputStream to be resilient to faiures in abort(); translate AWS Exceptions: Resolved
• FileSystem.get to support slow-to-instantiate FS clients: Resolved
• S3A committer to support concurrent jobs with same app attempt ID & dest dir: Resolved
• S3A marker tool mixes up -min and -max: Resolved
• ITestS3AContractRename failing against stricter tests: Resolved
• AbstractS3ATokenIdentifier to set issue date == now: Resolved
• Upgrade aws-java-sdk to 1.11.901: Resolved
• ITestS3ADeleteCost.testDirMarkersFileCreation failure: Resolved
• AbstractS3ATokenIdentifier to issue date in UTC: Resolved
• enable s3a magic committer by default: Resolved
• Magic committer files don't have the count of bytes written collected by spark: Resolved
• Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc: Resolved
• S3A docs to state s3 is consistent, deprecate S3Guard: Resolved
• magic committer to be enabled for all S3 buckets: Resolved
• typo in MagicCommitTracker: Resolved
• Add option to downgrade S3A rejection of Syncable to warning: Resolved
• whitespace not allowed in paths when saving files to s3a via committer: Resolved
• Possible NPE in S3A MultiObjectDeleteSupport error handling: Resolved