[HADOOP-15220] Über-jira: S3a phase V: Hadoop 3.2 features

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.2.0
• Target Version/s: 3.2.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 12/Feb/18 12:50
• Updated: 17/Sep/18 21:41
• Resolved: 17/Sep/18 21:41

Description

Über-jira for S3A work for Hadoop 3.2.x

The items from HADOOP-14831 which didn't get into Hadoop-3.1, and anything else

Issue Links

• depends upon: HADOOP-15541 AWS SDK can mistake stream timeouts for EOF and throw SdkClientExceptions
• depends upon: HADOOP-15226 Über-JIRA: S3Guard Phase III: Hadoop 3.2 features
• depends upon: HADOOP-15263 hadoop cloud-storage module to mark hadoop-common as provided; add azure-datalake
• depends upon: MAPREDUCE-6974 Add standard configuration keys for HTrace values, propagate across to MR committers if set
• is superceded by: HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
• supercedes: HADOOP-14831 Über-jira: S3a phase IV: Hadoop 3.1 features

Sub-Tasks

• add htrace context to HTTP requests as ? parameter: Resolved
• S3ADataBlocks.DiskBlock to lazy create dest file for faster 0-byte puts: Resolved
• use s3 tags/headers to record permissions on objects, so preserving them through distcp round trips: Resolved
• Automatically determine region & hence S3 endpoint of buckets: Resolved
• S3A shell entry point to support commands specific to S3A.: Resolved
• Hive doesn't let s3a patch the credential provider path: Resolved
• s3a encryption tests fail when dest bucket has SSE-KMS enabled: Resolved
• AWSCredentialProviderList to throw custom NoCredentialsException; retry logic to handle: Resolved
• Add Private/Unstable BulkDelete operations to supporting object stores for DistCP: Resolved
• S3AInputStream: Handle http stream skip(n) skipping < n bytes in a forward seek: Resolved
• S3ABlockOutputStream.flush() be no-op when stream closed: Resolved
• ITestS3AEmptyDirectory.testParallelJobsToAdjacentPaths failure, FileAlreadyExistsException: Resolved
• regression: secret stripping from S3x URIs breaks some downstream code: Resolved
• S3A ITestS3GuardWriteBack failing if bucket explicitly set to s3guard+DDB: Resolved
• S3 returning 400 on the directory /test/ GET of getFileStatus: Resolved
• S3ARetryPolicy for 400/BadArgument to be "fail": Resolved
• Stabilize/tune S3A committers; review correctness & docs: Resolved
• distcp numListstatusThreads option doesn't get to -delete scan: Resolved
• S3A Metrics in S3AInstrumentation Cause Memory Leaks in HBase Export: Resolved
• S3A directory committer commit job fails if _temporary directory created under dest: Resolved
• S3A staging committers to not log FNFEs on job abort listings: Resolved
• S3AFileStatus to add a serialVersionUID; review & test serialization: Resolved
• Expand S3A Assumed Role docs: Resolved
• S3A Multipart Uploader to work with S3Guard and encryption: Resolved
• AssumedRoleCredentialProvider to propagate connection settings of S3A FS: Resolved
• Stabilize S3A Assumed Role support: Resolved
• Update aws-sdk version to 1.11.375: Resolved
• s3guard uploads command to add a way to complete outstanding uploads: Resolved
• FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails on filesystems without append.: Resolved
• increase default value of fs.s3a.multipart.size to 128M: Resolved
• Remove s3a user:secret authentication: Resolved