Bulk Operation

HADOOP-13345 backport S3guard to branch-2

HADOOP-13345 s3guard usage calls function incorrectly

HADOOP-13345 S3Guard: handle provisioning failure through backoff & retry (& metrics)

HADOOP-13345 hadoop-aws shell profile not being built

HADOOP-13345 s3guard diff demand creates a new table

HADOOP-13345 S3Guard premerge changes: java 7 build & test tuning

HADOOP-13345 Review S3guard docs & code prior to merge

HADOOP-13345 ITestS3AEncryptionSSEC failing in parallel s3guard runs

HADOOP-13345 ITestS3GuardConcurrentOps failing with -Ddynamodblocal -Ds3guard

HADOOP-13345 add a predicate/option to probe an S3A FS for being consistent

HADOOP-13345 S3Guard: optimize create codepath

HADOOP-13345 LocalDynamoDB missing from latest AWS SDK releases

HADOOP-13345 S3Guard: issues running parallel tests w/ S3N

HADOOP-13345 ITestS3GuardListConsistency is too slow

HADOOP-13345 InconsistentAmazonS3Client adds extra paths to listStatus() after delete.

HADOOP-13345 simplify mkdirs() after S3Guard delete tracking change

HADOOP-13345 ProvidedFileStatusIterator#next() may throw IndexOutOfBoundsException

HADOOP-13345 Findbugs warning in LocalMetadataStore

HADOOP-13345 ITestS3GuardConcurrentOps requires explicit DynamoDB table name to be configured

HADOOP-13345 s3guard w/ failure injection: listStatus fails after renaming file into directory

HADOOP-13345 DirListingMetadata precondition failure messages to include path at fault

HADOOP-13345 Ensure deleted parent directory tombstones are overwritten when implicitly recreated

HADOOP-13345 make InconsistentAmazonS3Client usable in downstream tests

HADOOP-13345 S3Guard: Improve FNFE message when opening a stream

HADOOP-13345 create() does not notify metadataStore of parent directories or ensure they're not existing files

HADOOP-13345 Play nice with ITestS3AEncryptionSSEC

HADOOP-13345 ITestS3GuardConcurrentOps.testConcurrentTableCreations fails without table name configured

HADOOP-13345 s3guard will set file length to -1 on a putObjectDirect(stream, -1) call

HADOOP-13345 S3Guard: S3GuardTool to support provisioning existing metadata store

HADOOP-13345 ITestS3GuardListConsistency failure w/ Local, authoritative metadata store

HADOOP-13345 TestDynamoDBMetadataStore is broken unless we can fail faster without a table version

HADOOP-13345 S3Guard: DynamoDBMetadata::prune() should self interrupt correctly

HADOOP-13345 S3Guard: S3AFileSystem::listFiles() to employ MetadataStore

HADOOP-13345 TestS3GuardTool hangs/fails when offline: it's an IT test

HADOOP-13345 S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store

HADOOP-13345 S3Guard: ITestS3AConcurrentOps is not cleaning up test data

HADOOP-13345 S3Guard: DynamoDBMetadataStore::move() should populate ancestor directories of destination paths

HADOOP-13345 Add s3guardtool dump command

HADOOP-13345 DynamoDB client should waitForActive on existing tables

HADOOP-13345 Add validation of DynamoDB region

HADOOP-13345 S3Guard: import does not import empty directory

HADOOP-13345 S3GuardTool tests should not run if S3Guard is not set up

HADOOP-13345 Add S3Guard.dirListingUnion in S3AFileSystem#listFiles, listLocatedStatus

HADOOP-13345 Ensure GenericOptionParser is used for S3Guard CLI

HADOOP-13345 s3guard: CLI diff non-empty after import on new table

HADOOP-13345 Simplify DynamoDBClientFactory for creating Amazon DynamoDB clients

HADOOP-13345 ITestS3ACredentialsInURL sometimes fails

HADOOP-13345 s3guard tool tests aren't isolated; can't run in parallel

HADOOP-13345 In S3AFileSystem, make getAmazonClient() package private; export getBucketLocation()

HADOOP-13345 ITestS3GuardListConsistency fails intermittently

HADOOP-13345 s3guard: regression in dirListingUnion

HADOOP-13345 Rethink S3GuardTool options

HADOOP-13345 Allow users to specify region for DynamoDB table instead of endpoint

HADOOP-13345 Drop unnecessary type assertion and cast

HADOOP-13345 Fix breaking link in s3guard.md

HADOOP-13345 S3Guard: link docs from index, fix typos

HADOOP-13345 Metastore destruction test creates table without version marker

HADOOP-13345 CLI command to prune old metadata

HADOOP-13345 S3Guard: intermittent duplicate item keys failure

HADOOP-13345 Implicitly creating DynamoDB table ignores endpoint config

HADOOP-13345 S3Guard: DynamoDBMetadataStore logs nonsense region

HADOOP-13345 Optimize dirListingUnion

HADOOP-13345 S3Guard: fix multi-bucket integration tests

HADOOP-13345 Handled dynamo exceptions in translateException

HADOOP-13345 cli to list info about a bucket (S3guard or not)

HADOOP-13345 Merge initial S3guard release into trunk

HADOOP-13345 s3guard cli: make tests easier to run and address failure

HADOOP-13345 s3guard: add a version marker to every table

HADOOP-13345 S3Guard CLI: Add documentation

HADOOP-13345 Add ability to start DDB local server in every test

HADOOP-13345 Initialize DynamoDBMetadataStore without associated S3AFileSystem

HADOOP-13345 S3guard: replace dynamo.describe() call in init with more efficient query

HADOOP-13345 Mock bucket locations in MockS3ClientFactory

HADOOP-13345 S3Guard: DynamoDBMetadataStore#move() could be throwing exception due to BatchWriteItem limits

HADOOP-13345 S3AGuard: Use BatchWriteItem in DynamoDBMetadataStore#put()

HADOOP-13345 S3Guard: S3AFileSystem::listLocatedStatus() to employ MetadataStore

HADOOP-13345 S3Guard: NPE when table is already populated in dynamodb and user specifies "fs.s3a.s3guard.ddb.table.create=false"

HADOOP-13345 s3guard: improve S3AFileStatus#isEmptyDirectory handling

HADOOP-13345 S3Guard: Existing tables may not be initialized correctly in DynamoDBMetadataStore

HADOOP-13345 DynamoDBMetadataStore to handle DDB throttling failures through retry policy

HADOOP-13345 tune dynamodb client & tests

HADOOP-13345 dynamodb dependency -> compile

HADOOP-13345 s3guard: ITestS3AFileOperationCost.testFakeDirectoryDeletion failure

HADOOP-13345 S3Guard: fix TestDynamoDBMetadataStore when fs.s3a.s3guard.ddb.table is set

HADOOP-13345 S3Guard: better support for multi-bucket access

HADOOP-13345 s3guard to log choice of metadata store at debug

HADOOP-13345 s3guard: add inconsistency injection, integration tests

HADOOP-13345 S3Guard: add delete tracking

HADOOP-13345 Change PathMetadata to hold S3AFileStatus instead of FileStatus.

HADOOP-13345 S3Guard: S3AFileSystem Integration with MetadataStore

HADOOP-13345 S3Guard: Provide command line tools to manipulate metadata store.

HADOOP-13345 S3Guard: implement move() for LocalMetadataStore, add unit tests

HADOOP-13345 S3Guard: Allow execution of all S3A integration tests with S3Guard enabled.

HADOOP-13345 S3Guard: create basic contract tests for MetadataStore implementations

HADOOP-13345 S3Guard: Write end user docs, change table autocreate default.

HADOOP-13345 S3Guard: Instrument new functionality with Hadoop metrics.

HADOOP-13345 S3Guard: Implement access policy for intra-client consistency with in-memory metadata store.

HADOOP-13345 S3Guard: Implement access policy using metadata store as source of truth.

HADOOP-13345 S3Guard: Implement access policy providing strong consistency with S3 as source of truth.

HADOOP-13345 S3Guard: Implement DynamoDBMetadataStore.

HADOOP-13345 S3Guard: Define MetadataStore interface.

HADOOP-13345 Support running isolated unit tests separate from AWS integration tests.