[HADOOP-13204] Über-jira: S3a phase III: scale and tuning

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 3.0.0
• Component/s: fs/s3
• Target Version/s: 2.9.0, 3.0.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 26/May/16 13:58
• Updated: 13/Dec/17 11:19
• Resolved: 13/Dec/17 11:19

Description

S3A Phase III work; post 2.8.

Areas could include

• customisation
• performance enhancement
• management and metrics
• error/failure handling

And of course any bugs which surface

Issue Links

• depends upon: HADOOP-13050 Upgrade to AWS SDK 1.11.45
• depends upon: HADOOP-14475 Metrics of S3A don't print out when enable it in Hadoop metrics property file
• depends upon: HADOOP-14283 Upgrade AWS SDK to 1.11.134
• depends upon: HADOOP-13022 S3 MD5 check fails on Server Side Encryption-KMS with AWS and default key is used
• depends upon: HADOOP-14114 S3A can no longer handle unencoded + in URIs
• depends upon: HADOOP-13345 S3Guard: Improved Consistency for S3A
• depends upon: HADOOP-13912 S3a Multipart Committer (avoid rename)
• depends upon: HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
• depends upon: HADOOP-12981 Remove/deprecate s3native properties from S3NativeFileSystemConfigKeys and core-default.xml
• incorporates: HADOOP-14439 regression: secret stripping from S3x URIs breaks some downstream code
• is broken by: SPARK-15849 FileNotFoundException on _temporary while doing saveAsTable to S3
• is depended upon by: HADOOP-14831 Über-jira: S3a phase IV: Hadoop 3.1 features
• is related to: HIVE-14269 Performance optimizations for data on S3
• is related to: HADOOP-13321 Deprecate FileSystem APIs that promote inefficient call patterns.

Sub-Tasks

• harden S3AFileSystem.copyFromLocalFile(): Resolved
• save partition split size on multipart uploads: Resolved
• Switch to v2 of the S3 List Objects API in S3A: Resolved
• hook up AwsSdkMetrics to hadoop metrics: Resolved
• S3AFastOutputStream to implement flush(): Resolved
• split up AWS index.md: Resolved
• S3A create() to support asynchronous check of dest & parent paths: Resolved
• s3a delete() operation fails during a concurrent delete of child entries: Resolved
• s3a.mkdirs() to delete empty fake parent directories: Resolved
• S3A open to avoid needless HEAD on the successful execution path: Resolved
• Remove verifyBuckets overhead in S3AFileSystem::initialize(): Resolved
• s3a to skip s3 bucket existence check in initialize() for faster creation: Resolved
• S3A getContentSummary to use flat listFiles instead of treewalk: Resolved
• Add support for SSE-KMS and SSE-C in s3a filesystem: Resolved
• use JUnit categories for s3a scale tests and others: Resolved
• S3A create(path, overwrite=true) need only check for path being a dir, not a file: Resolved
• Use shaded aws-sdk uber-JAR 1.11.86: Resolved
• Split S3 testing documentation out into its own file: Resolved
• Sort out hadoop-aws contract-test-options.xml: Resolved
• Relax error message assertion in S3A test ITestS3AEncryptionSSEC: Resolved
• cut some obsolete, ignored s3 tests in TestS3Credentials: Resolved
• move jets3t into a dependency on hadoop-aws JAR: Resolved
• needless S3AFileSystem.setOptionalPutRequestParameters in S3ABlockOutputStream putObject(): Resolved
• Remove S3A ref from META-INF service discovery, rely on existing core-default entry: Resolved
• add more on s3 regions to the s3a documentation: Resolved
• Remove URI parameter in AWSCredentialProvider constructors: Resolved
• S3A to delete unnecessary fake directory objects in mkdirs(): Resolved
• Deprecate SharedInstanceProfileCredentialsProvider in branch-2.: Resolved
• Explicitly exclude S3A root dir ITests from parallel runs: Resolved
• S3A to support S3 reduced redundancy storage: Resolved
• S3A copyFromLocalFile to be robust, tested: Resolved
• remove jackson, joda and other transient aws SDK dependencies from hadoop-aws: Resolved
• Uninitialized S3A instance NPEs on toString(): Resolved
• Extend and enhance S3A exception handling: Resolved
• AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error: Resolved
• transient list consistency failure in ITestS3AContractRootDir: Resolved
• Remove S3N and obsolete bits of S3A; rework docs: Resolved
• S3A case "testRandomReadOverBuffer" failed due to improper range parameter: Resolved
• review S3 docs prior to 3.0-beta-1: Resolved
• Remove original S3A output stream: Resolved
• Move up to AWS SDK 1.11.199: Resolved