Bulk Operation

HADOOP-11694 GCS to support per-bucket configuration

HADOOP-11694 S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock)

HADOOP-11694 fix some typos in the s3a docs

HADOOP-11694 ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful

HADOOP-11694 S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs

HADOOP-11694 Fix a couple of the s3a statistic names to be consistent with the rest

HADOOP-11694 s3a rename: fail if dest file exists

HADOOP-11694 regression: ITestS3AMiniYarnCluster failing

HADOOP-11694 ITestS3AContractRootDir still playing up, bug in eventually() retry logic?

HADOOP-11694 Purge superfluous/obsolete S3A Tests

HADOOP-11694 ITestS3AFileContextStatistics.testStatistics() failing

HADOOP-11694 switch to Configuration.getLongBytes for byte options

HADOOP-11694 S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider.

HADOOP-11694 S3ABlockOutputStream to pass Yetus & Jenkins

HADOOP-11694 hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts.

HADOOP-11694 fs.s3a.readahead.range to use getLongBytes

HADOOP-11694 S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint.

HADOOP-11694 document object store use with fs shell and distcp

HADOOP-11694 s3:// should have been fully cut off from trunk

HADOOP-11694 Purge some superfluous/obsolete S3 FS tests that are slowing test runs down

HADOOP-11694 s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown

HADOOP-11694 S3AFastOutputStream to take ProgressListener in file create()

HADOOP-11694 S3AFileSystem to override getStorageStatistics() and so serve up its statistics

HADOOP-11694 NPE in S3AFastOutputStream.write

HADOOP-11694 S3ABlockOutputStream to support huge (many GB) file writes

HADOOP-11694 explicitly declare the Joda time version S3A depends on

HADOOP-11694 improve section on troubleshooting s3a auth problems

HADOOP-11694 S3A output streams to share a single LocalDirAllocator for round-robin drive use

HADOOP-11694 doc for “fs.s3a.acl.default” indicates incorrect values

HADOOP-11694 S3A to support per-bucket configuration

HADOOP-11694 s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints)

HADOOP-11694 S3A to list InstanceProfileCredentialsProvider after EnvironmentVariableCredentialsProvider

HADOOP-11694 S3A TemporaryAWSCredentialsProvider to support Hadoop Credential providers for secrets

HADOOP-11694 S3A reporting of file group as empty is harmful to compatibility for the shell.

HADOOP-11694 Document S3A known limitations in file ownership and permission model.

HADOOP-11694 add a S3A scale test to do gunzip and linecount

HADOOP-11694 S3A to track multipart upload count, size, duration

HADOOP-11694 hadoop fs command path doesn't include translation of amazon client exceptions

HADOOP-11694 set multipart delete timeout to 5 * 60s in S3ATestUtils.createTestFileSystem

HADOOP-11694 mvn fs/s3 test runs to set DNS TTL to 20s

HADOOP-11694 Tune S3A provider plugin mechanism

HADOOP-11694 document s3a better

HADOOP-11694 s3a initialization against public bucket fails if caller lacks any credentials

HADOOP-11694 Isolate test path used by a few S3A tests for more reliable parallel execution.

HADOOP-11694 Provide an option to set the socket buffers in S3AFileSystem

HADOOP-11694 S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories

HADOOP-11694 S3A: Support fadvise "random" mode for high performance readPositioned() reads

HADOOP-11694 S3A file-create should throw error rather than overwrite directories

HADOOP-11694 Add StorageStatistics to S3A; instrument some more operations

HADOOP-11694 Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs

HADOOP-11694 S3AFileSystem#toString might throw NullPointerException due to null cannedACL.

HADOOP-11694 S3AFileSystem printAmazonServiceException/printAmazonClientException appear copy & paste of AWS examples

HADOOP-11694 Add tests to verify that S3A supports SSE-S3 encryption

HADOOP-11694 s3a failures can surface as RTEs, not IOEs

HADOOP-11694 Customize User-Agent header sent in HTTP requests by S3A.

HADOOP-11694 Enable parallel test execution for hadoop-aws.

HADOOP-11694 if fs.s3a.block.size option == 0, use partition size option for blocksize

HADOOP-11694 switch hadoop-aws back to using the (heavy) amazon-sdk JAR

HADOOP-11694 S3A Introspect to invoke incompatible AWS TransferManagerConfiguration methods

HADOOP-11694 S3A FS fails during init against a read-only FS if multipart purge is enabled

HADOOP-11694 Upgrade to AWS SDK 1.11.45

HADOOP-11694 S3a Forward seek in stream length to be configurable

HADOOP-11694 add low level counter metrics for S3A; use in read performance tests

HADOOP-11694 add option for lazy open() on s3a

HADOOP-11694 s3a to pass PositionedReadable contract tests, improve readFully perf.

HADOOP-11694 move s3a to slf4j logging

HADOOP-11694 s3a to handle delete("/", true) robustly

HADOOP-11694 s3a toString to be meaningful in logs

HADOOP-11694 Recover when S3A fails on IOException in read()

HADOOP-11694 Read Proxy Password from Credential Providers in S3 FileSystem

HADOOP-11694 s3a should use UGI.getCurrentUser.getShortname() for username

HADOOP-11694 S3A JUnit tests failing if using HTTP proxy

HADOOP-11694 Update documentation to cover fs.s3.buffer.dir enhancements

HADOOP-11694 Support lazy seek in S3AInputStream

HADOOP-11694 Improve S3AFastOutputStream memory management

HADOOP-11694 Make use of DeleteObjects optional

HADOOP-11694 ListStatus on empty dir in S3A lists itself instead of returning an empty list

HADOOP-11694 Listing an empty s3a root directory throws FileNotFound.

HADOOP-11694 s3a can throw spurious IOEs on close()

HADOOP-11694 S3a to use thread pool that blocks clients

HADOOP-11694 verify that s3a handles / in secret key

HADOOP-11694 Enable YARN to use S3A

HADOOP-11694 Add NativeS3Fs that delegates calls from FileContext apis to native s3 fs implementation