[HADOOP-15619] Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0
• Component/s: fs/s3
• Target Version/s: 3.3.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 19/Jul/18 21:26
• Updated: 31/Mar/20 16:42
• Resolved: 31/Mar/20 16:42

Description

Features for S3Guard for Hadoop 3.3. Goal: take the experimental tag off

Issue Links

• depends upon: HADOOP-13980 S3Guard CLI: Add fsck check and fix commands
• is related to: HADOOP-16646 Backport S3A enhancements and fixes from trunk to branch-3.2
• relates to: HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
• supercedes: HADOOP-15226 Über-JIRA: S3Guard Phase III: Hadoop 3.2 features
• links to: GitHub Pull Request #1698

Sub-Tasks

• add bulk delete call to metastore API & DDB impl: Resolved
• s3guard DynamoDB resource not found: tables not ACTIVE state after initial connection: Resolved
• S3Guard to support creating on-demand DDB tables: Resolved
• s3guard set-capacity to not retry on an access denied exception: Resolved
• S3Guard: Provide custom FileSystem Statistics.: Resolved
• s3guard set-capacity command to fail fast if bucket is unguarded: Resolved
• S3Guard, MetadataStore to support atomic create(path, overwrite=false): Resolved
• S3AFileSystem.innerRename() to size metadatastore lists better: Resolved
• S3Guard: remove "experimental" from documentation: Resolved
• S3Guard: DynamoDB can go out of sync with S3AFileSystem.delete(): Resolved
• S3Guard store becomes inconsistent after partial failure of rename: Resolved
• S3Guard: make short-circuit getFileStatus() configurable: Resolved
• hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard: Resolved
• S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing: Resolved
• s3guard bucket-info will create s3guard table if FS is set to do this automatically: Resolved
• ITestS3GuardTool failures in testDestroyNoBucket(): Resolved
• Move S3Guard LocalMetadataStore constants to org.apache.hadoop.fs.s3a.Constants: Resolved
• S3Guard: document how to deal with non-S3Guard processes writing data to S3Guarded buckets: Resolved
• ClassCastException: S3AFileSystem cannot be cast to MockS3AFileSystem when fs caching is disabled: Resolved
• LocalMetadataStore put() does not retain isDeleted in parent listing: Resolved
• ITestS3AEmptyDirectory#testDirectoryBecomesEmpty fails when running with dynamo: Resolved
• ITestS3GuardListConsistency#testConsistentListAfterDelete fails when running with dynamo: Resolved
• NPE in DynamoDBMetadataStore.lambda$listChildren for root + auth S3Guard: Resolved
• Intermittent failures of some S3A tests with S3Guard in parallel test runs: Resolved
• DynamoDB table Update can fail S3A FS init: Resolved
• s3guard init and destroy command will create/destroy tables if ddb.table & region are set: Resolved
• S3Guard testConcurrentTableCreations to set r & w capacity == 0: Resolved
• s3guard bucket-info command to not print a stack trace on bucket-not-found: Resolved
• ITestDynamoDBMetadataStore can leak (large) DDB tables in test failures/timeout: Resolved
• ITestDynamoDBMetadataStore should check if test ddb table set properly before initializing the test: Resolved
• Should be able to set empty directory flag to TRUE in DynamoDBMetadataStore#innerGet when using authoritative directory listings: Resolved
• S3Guard: Better support for out-of-band operations: Resolved
• IndexOutOfBoundsException in ITestS3GuardToolLocal: Resolved
• S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite: Resolved
• S3Guard to support on-demand DDB tables: Resolved
• S3Guard: Handle OOB deletions and creation of a file which has a tombstone marker: Resolved
• S3Guard: NPE in DynamoDBMetadataStore.lambda$listChildren: Resolved
• ITestS3GuardToolDynamoDB test failures: Resolved
• S3Guard: fail write that doesn't update metadata store: Resolved
• Add a way to declare a toplevel path in a non-auth FS as auth: Resolved
• Use configurable dynamo table name prefix in S3Guard tests: Resolved
• transient failure of ITestS3GuardToolDynamoDB.testDestroyUnknownTable: Resolved
• S3Guard: Implement time-based (TTL) expiry for entries (and tombstones): Resolved
• S3Guard: Retry failed read with backoff in Authoritative mode when file can be opened: Resolved
• S3Guard: Remove LocalMetadataStore: Resolved
• "Wrong Billing mode" test failure in ITestS3GuardToolDynamoDB: Resolved
• DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry: Resolved
• S3Guard DDB store prune() doesn't translate AWS exceptions to IOEs: Resolved
• S3Guard table destroy to map IllegalArgumentExceptions to IOEs: Resolved
• S3A list operation doesn't pick up etags from results: Resolved
• ITestS3AMetadataPersistenceException failure: Resolved
• S3A tombstones can confuse empty directory status: Resolved
• Pass ITtlTimeProvider instance in initialize method in MetadataStore interface: Resolved
• S3A: Avoid inconsistencies between DDB and S3: Resolved
• S3Guard Diff tool to list+ compare the etag and version fields: Resolved
• Hadoop S3Guard Prune command to support a -tombstone option.: Resolved
• ITestDynamoDBMetadataStore.testProvisionTable times out intermittently: Resolved
• mark DynamoDBMetadataStore.deleteTrackingValueMap as final: Resolved
• Transient failure of ITestS3ARemoteFileChanged: Resolved
• ITestS3GuardOutOfBandOperations.deleteAfterTombstoneExpiryOobCreate failure: Resolved
• DynamoDBMetaStore deleteSubtree to delete leaf nodes first: Resolved
• S3AFilesystem.delete to incrementally update s3guard with deletions: Resolved
• S3Guard: Filter expired entries and tombstones when listing with MetadataStore#listChildren: Resolved
• S3Guard in auth mode doesn't raise AccessDeniedException on read of 0-byte file: Resolved
• findbugs warning on LocalMetadataStore.ttlTimeProvider sync: Resolved
• S3Guard ProgressiveRenameTracker to mark dest dir as authoritative on success: Resolved
• ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region: Resolved
• S3A to warn or fail if S3Guard is disabled: Resolved
• S3Guard operations log has tombstone/PUT swapped: Resolved
• Avoid/handle cached 404s during S3A file creation: Resolved
• s3guard auth path checks only check against unqualified source path: Resolved
• Race condition in DDB table init and waiting threads: Resolved
• s3guard prune command doesn't get AWS auth chain from FS: Resolved
• Region must be provided when requesting session credentials or SdkClientException will be thrown: Resolved
• ITestS3GuardDDBRootOperations. test_100_FilesystemPrune failure: Resolved
• ITestDynamoDBMetadataStoreScale fails when throttled.: Resolved
• s3guard bucket info to list a bit more about authoritative paths: Resolved
• audit/tune s3a authoritative flag in s3guard DDB Table: Resolved
• Prune -tombstones to remove children entries: Resolved