[HADOOP-11694] Über-jira: S3a phase II: robustness, scale and performance

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.8.0
• Component/s: fs/s3
• Target Version/s: 2.8.1
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 09/Mar/15 21:28
• Updated: 05/Apr/17 12:30
• Resolved: 05/Apr/17 12:30

Description

HADOOP-11571 covered the core s3a bugs surfacing in Hadoop-2.6 & other enhancements to improve S3 (performance, proxy, custom endpoints)

This JIRA covers post-2.7 issues and enhancements.

Issue Links

• depends upon: HADOOP-12269 Update aws-sdk dependency to 1.10.6; move to aws-sdk-s3
• depends upon: HADOOP-13826 S3A Deadlock in multipart copy due to thread pool limits.
• depends upon: HADOOP-14028 S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures
• depends upon: HADOOP-12346 Increase some default timeouts / retries for S3a connector
• depends upon: HADOOP-12667 s3a: Support createNonRecursive API
• depends upon: HADOOP-12851 S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders
• depends upon: HADOOP-12970 Intermittent signature match failures in S3AFileSystem due connection closure
• depends upon: HADOOP-12979 IOE in S3a: ${hadoop.tmp.dir}/s3a not configured
• depends upon: HADOOP-13389 TestS3ATemporaryCredentials.testSTS error when using IAM credentials
• depends upon: HADOOP-3733 "s3:" URLs break when Secret Key contains a slash, even if encoded
• depends upon: HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories
• depends upon: HADOOP-13406 S3AFileSystem: Consider reusing filestatus in delete() and mkdirs()
• depends upon: HADOOP-13516 Listing an empty s3a NON root directory throws FileNotFound.
• depends upon: HADOOP-13993 transient failure of ITestS3AInputStreamPerformance.testDecompressionSequential128K
• depends upon: HADOOP-11571 Über-jira: S3a stabilisation phase I
• depends upon: HADOOP-12810 FileSystem#listLocatedStatus causes unnecessary RPC calls
• depends upon: HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider
• depends upon: HADOOP-12537 S3A to support Amazon STS temporary credentials
• depends upon: HADOOP-11687 Ignore x-* and response headers when copying an Amazon S3 object
• depends upon: HADOOP-11827 Speed-up distcp buildListing() using threadpool
• depends upon: HADOOP-12891 S3AFileSystem should configure Multipart Copy threshold and chunk size
• depends upon: HADOOP-13017 Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0
• depends upon: HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata.
• depends upon: HADOOP-13239 Deprecate s3:// in branch-2
• depends upon: HADOOP-12807 S3AFileSystem should read AWS credentials from environment variables
• depends upon: HADOOP-12963 Allow using path style addressing for accessing the s3 endpoint
• depends upon: HADOOP-12982 Document missing S3A and S3 properties
• depends upon: HADOOP-13166 add getFileStatus("/") test to AbstractContractGetFileStatusTest
• depends upon: HADOOP-13169 Randomize file list in SimpleCopyListing
• depends upon: HADOOP-13323 Downgrade stack trace on FS load from Warn to debug
• depends upon: HADOOP-13139 Branch-2: S3a to use thread pool that blocks clients
• is blocked by: HADOOP-14204 S3A multipart commit failing, "UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort"
• is depended upon by: HIVE-14269 Performance optimizations for data on S3
• is depended upon by: HADOOP-13204 Über-jira: S3a phase III: scale and tuning
• relates to: HADOOP-13113 Enable parallel test execution for hadoop-aws.
• relates to: HADOOP-13212 Provide an option to set the socket buffers in S3AFileSystem
• relates to: HADOOP-12548 Read s3a creds from a Credential Provider

Sub-Tasks

• Enable YARN to use S3A: Resolved
• Add NativeS3Fs that delegates calls from FileContext apis to native s3 fs implementation: Resolved
• S3a to use thread pool that blocks clients: Resolved
• verify that s3a handles / in secret key: Resolved
• s3a can throw spurious IOEs on close(): Resolved
• ListStatus on empty dir in S3A lists itself instead of returning an empty list: Resolved
• Make use of DeleteObjects optional: Resolved
• Listing an empty s3a root directory throws FileNotFound.: Resolved
• Support lazy seek in S3AInputStream: Resolved
• Update documentation to cover fs.s3.buffer.dir enhancements: Resolved
• S3A JUnit tests failing if using HTTP proxy: Resolved
• s3a should use UGI.getCurrentUser.getShortname() for username: Resolved
• Recover when S3A fails on IOException in read(): Resolved
• s3a toString to be meaningful in logs: Resolved
• s3a to handle delete("/", true) robustly: Resolved
• move s3a to slf4j logging: Resolved
• s3a to pass PositionedReadable contract tests, improve readFully perf.: Resolved
• add option for lazy open() on s3a: Resolved
• add low level counter metrics for S3A; use in read performance tests: Resolved
• S3a Forward seek in stream length to be configurable: Resolved
• S3A FS fails during init against a read-only FS if multipart purge is enabled: Resolved
• switch hadoop-aws back to using the (heavy) amazon-sdk JAR: Resolved
• if fs.s3a.block.size option == 0, use partition size option for blocksize: Resolved
• S3A Introspect to invoke incompatible AWS TransferManagerConfiguration methods: Resolved
• Customize User-Agent header sent in HTTP requests by S3A.: Resolved
• Add tests to verify that S3A supports SSE-S3 encryption: Resolved
• s3a failures can surface as RTEs, not IOEs: Resolved
• S3AFileSystem printAmazonServiceException/printAmazonClientException appear copy & paste of AWS examples: Closed
• S3AFileSystem#toString might throw NullPointerException due to null cannedACL.: Resolved
• Enable parallel test execution for hadoop-aws.: Resolved
• Add StorageStatistics to S3A; instrument some more operations: Resolved
• S3A file-create should throw error rather than overwrite directories: Resolved
• S3A: Support fadvise "random" mode for high performance readPositioned() reads: Resolved
• S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories: Resolved
• Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs: Resolved
• Isolate test path used by a few S3A tests for more reliable parallel execution.: Resolved
• s3a initialization against public bucket fails if caller lacks any credentials: Resolved
• document s3a better: Resolved
• Tune S3A provider plugin mechanism: Resolved
• Provide an option to set the socket buffers in S3AFileSystem: Resolved
• Read Proxy Password from Credential Providers in S3 FileSystem: Resolved
• mvn fs/s3 test runs to set DNS TTL to 20s: Resolved
• set multipart delete timeout to 5 * 60s in S3ATestUtils.createTestFileSystem: Resolved
• hadoop fs command path doesn't include translation of amazon client exceptions: Resolved
• add a S3A scale test to do gunzip and linecount: Resolved
• S3A TemporaryAWSCredentialsProvider to support Hadoop Credential providers for secrets: Resolved
• S3A to list InstanceProfileCredentialsProvider after EnvironmentVariableCredentialsProvider: Resolved
• s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints): Resolved
• doc for “fs.s3a.acl.default” indicates incorrect values: Resolved
• improve section on troubleshooting s3a auth problems: Resolved
• explicitly declare the Joda time version S3A depends on: Resolved
• NPE in S3AFastOutputStream.write: Resolved
• S3AFileSystem to override getStorageStatistics() and so serve up its statistics: Resolved
• s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown: Resolved
• s3:// should have been fully cut off from trunk: Resolved
• document object store use with fs shell and distcp: Resolved
• S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint.: Resolved
• fs.s3a.readahead.range to use getLongBytes: Resolved
• hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts.: Resolved
• S3A reporting of file group as empty is harmful to compatibility for the shell.: Resolved
• Document S3A known limitations in file ownership and permission model.: Resolved
• S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider.: Resolved
• switch to Configuration.getLongBytes for byte options: Resolved
• S3AFastOutputStream to take ProgressListener in file create(): Resolved
• S3ABlockOutputStream to pass Yetus & Jenkins: Resolved
• S3ABlockOutputStream to support huge (many GB) file writes: Resolved
• Purge some superfluous/obsolete S3 FS tests that are slowing test runs down: Resolved
• ITestS3AContractRootDir still playing up, bug in eventually() retry logic?: Resolved
• regression: ITestS3AMiniYarnCluster failing: Resolved
• Upgrade to AWS SDK 1.11.45: Resolved
• Purge superfluous/obsolete S3A Tests: Resolved
• ITestS3AFileContextStatistics.testStatistics() failing: Resolved
• s3a rename: fail if dest file exists: Resolved
• Fix a couple of the s3a statistic names to be consistent with the rest: Resolved
• ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful: Resolved
• S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs: Resolved
• S3A to track multipart upload count, size, duration: Resolved
• Improve S3AFastOutputStream memory management: Resolved
• S3A output streams to share a single LocalDirAllocator for round-robin drive use: Resolved
• S3A to support per-bucket configuration: Resolved
• fix some typos in the s3a docs: Resolved
• S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock): Resolved
• GCS to support per-bucket configuration: Resolved