[HADOOP-13980] S3Guard CLI: Add fsck check and fix commands

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-beta1
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Aaron Fabbri
• Created: 12/Jan/17 03:03
• Updated: 17/Apr/20 07:46

Description

As discussed in HADOOP-13650, we want to add an S3Guard CLI command which compares S3 with MetadataStore, and returns a failure status if any invariants are violated.

Issue Links

• is depended upon by: HADOOP-14936 S3Guard: remove "experimental" from documentation
• is depended upon by: HADOOP-15619 Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features
• is related to: HADOOP-13936 S3Guard: DynamoDB can go out of sync with S3AFileSystem.delete()
• links to: GitHub Pull Request #1060
• links to: WIP docs

Sub-Tasks

• S3Guard fsck: Export MetadataStore and S3 bucket hierarchy: Open
• S3Guard fsck: Implement fixing mechanism for inconsistencies between S3 and MetadataStore: Open
• S3Guard fsck: Check metadata consistency from metadatastore to S3 (log): Open
• Add fsck to S3A tests where additional diagnosis is needed: Open
• S3Guard fsck: Detect if a directory if authoritative and highlight errors if detected in it: Open
• S3Guarld fsck: Add docs to the first iteration (S3->ddbMS, -verify): Open
• S3Guard fsck: Use enum instead of int for severity in violations: Open