[HADOOP-15620] Über-jira: S3A phase VI: Hadoop 3.3 features

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Component/s: fs/s3
• Target Version/s: 3.3.0

Release Note

Lots of enhancements to the S3A code, including
* Delegation Token support
* better handling of 404 caching
* S3guard performance, resilience improvements

• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 19/Jul/18 21:28
• Updated: 31/Mar/20 16:49
• Resolved: 23/Mar/20 13:12

Issue Links

• depends upon: HADOOP-16248 MutableQuantiles leak memory under heavy load
• depends upon: HADOOP-16665 Filesystems to be closed if they failed during initialize()
• depends upon: HADOOP-15281 Distcp to add no-rename copy option
• depends upon: MAPREDUCE-7092 MR examples to work better against cloud stores
• is blocked by: HADOOP-16759 Filesystem openFile() builder to take a FileStatus param
• is related to: HADOOP-15619 Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features
• is related to: HADOOP-16646 Backport S3A enhancements and fixes from trunk to branch-3.2
• is superceded by: HADOOP-16829 Über-jira: S3A Hadoop 3.3.1 features
• supercedes: HADOOP-15220 Über-jira: S3a phase V: Hadoop 3.2 features

Sub-Tasks

• s3a create(overwrite=true) to only look for dir/ and list entries, not file: Resolved
• S3A init hangs if you try to connect while the system is offline: Resolved
• S3AInputStream to implement CanUnbuffer: Resolved
• S3A: Consider using TransferManager.download for copyToLocalFile: Resolved
• Parallelize S3A directory deletes: Resolved
• Add support for S3 Select to S3A: Resolved
• S3A to support Delegation Tokens: Resolved
• Stabilise/formalise the JSON _SUCCESS format used in the S3A committers: Resolved
• Add S3A implementation of FSMainOperationsBaseTest: Resolved
• S3a to support get/set permissions through S3 object tags: Resolved
• S3a rename() to copy files in a directory in parallel: Resolved
• Add HTrace to the s3a connector: Resolved
• S3A should allow renaming to a pre-existing destination directory to move the source path under that directory, similar to HDFS.: Resolved
• fs -expunge to take a filesystem: Resolved
• S3A Retry policy to retry on NoResponseException: Resolved
• S3A client raising ConnectionPoolTimeoutException in test: Resolved
• Bulk commits of S3A MPUs place needless excessive load on S3 & S3Guard: Resolved
• S3A log message on rm s3a://bucket/ not intuitive: Resolved
• S3aUtils.getEncryptionAlgorithm() always logs@Debug "Using SSE-C": Resolved
• S3A warning of obsolete encryption key which is never used: Resolved
• add s3guard CLI command to generate session keys for an assumed role: Resolved
• S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2: Resolved
• FileSystemMultipartUploader should verify that UploadHandle has non-0 length: Resolved
• Memory leak in S3AOutputStream: Resolved
• [s3a] stop treat fs.s3a.max.threads as the long-term minimum: Resolved
• S3 listing inconsistency can raise NPE in globber: Resolved
• remove obsolete S3A test ITestS3ACredentialsInURL: Resolved
• S3A input stream to use etags/version number to detect changed source files: Resolved
• Move ITestS3AMiniYarnCluster to S3A committers: Resolved
• @Retries annotation of putObject() call & uses wrong: Resolved
• Review + update cloud store sensitive keys in hadoop.security.sensitive-config-keys: Resolved
• ITestS3AContractMultipartUploader#testMultipartUploadEmptyPart test error: Resolved
• Some S3A committer tests don't match ITest* pattern; don't run in maven: Resolved
• get patch for S3a nextReadPos(), through Yetus: Resolved
• Hadoop aws does not use shaded jars: Resolved
• Oozie unable to create sharelib in s3a filesystem: Resolved
• S3A committers: make sure there's regular progress() calls: Resolved
• S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2: Resolved
• Add bouncycastle jars to hadoop-aws as test dependencies: Resolved
• [DOC] Effective use of FS instances during S3A integration tests: Resolved
• hamcrest-library declaration in hadoop-aws to be scoped test: Resolved
• S3A SSL connections should use OpenSSL: Resolved
• S3A tests to include Terasort: Resolved
• Token.toString faulting if any token listed can't load.: Resolved
• Move DurationInfo from hadoop-aws to hadoop-common org.apache.hadoop.util: Resolved
• Parquet reading S3AFileSystem causes EOF: Resolved
• Update AWS SDK to 1.11.563: Resolved
• Extend documentation in testing.md about endpoint constants: Resolved
• regression: ITestS3AMiniYarnCluster failing on branch-3.2: Resolved
• S3Guard to add DynamoDBLocal Support: Resolved
• S3A copyFile operation to include source versionID or etag in the copy request: Resolved
• S3A MarshalledCredentials.toString() doesn't print full date/time of expiry: Resolved
• S3AUtils.translateException to map CredentialInitializationException to AccessDeniedException: Resolved
• S3AFileSystem#innerMkdirs builds needless lists: Resolved
• warning about user:pass in URI to explicitly call out Hadoop 3.2 as removal: Resolved
• Improved S3A MR tests: Resolved
• S3AFileStatus to declare that isEncrypted() is always true: Resolved
• S3A delegation tests fail if you set fs.s3a.secret.key: Resolved
• S3A Etag tests fail with default encryption enabled on bucket: Resolved
• S3A Delegation Token code to spell "Marshalled" as Marshaled: Resolved
• s3a test docs to mention non-auth; or s3a tests to default to non-auth: Resolved
• ClassCastException in S3GuardTool.checkMetadataStoreUri: Resolved
• Regression: TestStagingPartitionedJobCommit failing with empty etag list: Resolved
• Remove S3A's depedency on http core: Resolved
• Test Hang in S3A S3guard test MetadataStoreTestBase.testListChildren: Resolved
• Stabilize S3A OpenSSL support: Resolved
• MapReduce job tasks fails on S3A ssl3_get_server_certificate:certificate verify: Resolved
• TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists: Resolved
• S3A NullPointerException: null uri host. This can be caused by unencoded / in the password string: Resolved
• Option to disable GCM for SSL connections when running on Java 8: Resolved
• S3AInputStream#unbuffer should merge input stream stats into fs-wide stats: Resolved
• S3A returns 400 "bad request" on a single path within an S3 bucket: Resolved
• AbstractITCommitMRJob.testMRJob test failures: Resolved
• Downgrade INFO message on rm s3a root dir to DEBUG: Resolved
• ITestS3ACommitterFactory failing, S3 client is not inconsistent: Resolved
• Typo in s3a committers.md doc: Resolved
• Make last AWS credential provider in default auth chain EC2ContainerCredentialsProviderWrapper: Resolved
• Restore (documented) fs.s3a.SharedInstanceProfileCredentialsProvider: Resolved
• S3A delegation token tests fail if fs.s3a.encryption.key set: Resolved
• S3Guard bucket-info fails if the bucket location is denied to the caller: Resolved
• S3A retry policy to be exponential: Resolved
• S3ADelegationTokens to only log at debug on startup: Resolved
• S3A committers leak threads/raises OOM on job/task commit at scale: Resolved
• s3a attempts to look up password/encryption fail if JCEKS file unreadable: Resolved
• S3A ITestRestrictedReadAccess fails: Resolved
• Speculating & Partitioned S3A magic committers can leave pending files under __magic: Resolved
• S3A ITest*MRjob failures: Resolved
• S3A ITest failures without S3Guard: Resolved
• S3A innerGetFileStatus s"directories only" scan still does a HEAD: Resolved
• S3A Delegation Token extension point to use StoreContext: Resolved
• ITestS3AClosedFS failing -junit test thread: Resolved
• S3 getBucketLocation() can return "US" for us-east: Resolved
• S3Guard DDB overreacts to no tag access: Resolved
• HadoopExecutors cleanup to only log at debug: Resolved
• S3Guard: Make authoritative mode exclusive for metadata - don't check for expiry for authoritative paths: Resolved
• S3A bucket existence checks to support v2 API and "no checks at all": Resolved
• S3GuardTool to support FilterFileSystem: Resolved
• s3guard prune can delete directories -leaving orphan children.: Resolved
• S3Guard to support encrypted DynamoDB table: Resolved
• S3A empty dir markers are not created in s3guard as authoritative: Resolved
• DurationInfo text parsing/formatting should be moved out of hotpath: Resolved
• Increase timeout unit test rule for MetadataStoreTestBase: Resolved
• Refine testing.md to tell user better how to use auth-keys.xml: Resolved
• fs.s3a.authoritative.path should support multiple FS URIs: Resolved
• Filesystem openFile() builder to take a FileStatus param: Resolved
• S3AInputStream reopening does not handle non IO exceptions properly: Resolved
• Let s3 clients configure request timeout: Resolved
• S3Guard listFiles will not query S3 if all listings are authoritative: Resolved
• Large DeleteObject requests are their own Thundering Herd: Resolved
• TestHarFileSystem.testInheritedMethodsImplemented broken: Resolved
• ITestS3GuardOutOfBandOperations failing on versioned S3 buckets: Resolved
• S3A reverts KMS encryption to the bucket's default KMS key in rename/copy: Resolved