[HADOOP-15038] Abstract MetadataStore in S3Guard into a common module.

• Type: New Feature
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.0, 2.9.2, 3.0.3, 3.1.2
• Component/s: fs
• Assignee: wujinhu
• Reporter: Genmao Yu
• Created: 14/Nov/17 07:20
• Updated: 12/Mar/19 01:43

Description

Open this JIRA to discuss if we should move MetadataStore in S3Guard into a common module.

Based on this work, other filesystem or object store can implement their own metastore for optimization (known issues like consistency problem and metadata operation performance). [e-mail] and other guys have done many base and great works in S3Guard. It is very helpful to start work. I did some perf test in HADOOP-14098, and started related work for Aliyun OSS. Indeed there are still works to do for S3Guard, like metadata cache inconsistent with S3 and so on. It also will be a problem for other object store. However, we can do these works in parallel.

[e-mail] fabbri drankye Any suggestion is appreciated.

Issue Links

• is depended upon by: HADOOP-12876 [Azure Data Lake] Support for process level FileStatus cache to optimize GetFileStatus frequent operations
• is related to: HADOOP-14098 AliyunOSS: improve the performance of object metadata operation