[HDDS-5672] Support protocol aware buckets within a single OM

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: OM, Ozone Client, Ozone Filesystem, S3
• Assignee: Aryan Gupta
• Reporter: Rakesh Radhakrishnan
• Created: 26/Aug/21 05:20
• Updated: 29/Apr/22 05:59

Description

Presently key path normalization and prefix(HDDS-2939) feature configurations ozone.om.enable.filesystem.paths and ozone.om.metadata.layout are at OM cluster-level. Using these configurations, OM cluster can be configured and started to satisfy only a specific use case, either ‘Pure S3' or ‘FileSystem’ semantics.

This is an umbrella jira talking about moving the above two cluster-level configurations to a bucket level configuration so that single OM will efficiently hold multiple buckets with different modes at at time. A user who wants to satisfy ‘Pure S3' use case, he can define the bucket in that mode and other user who wants to satisfy ‘FileSystem’ use case, he can define the bucket in that mode.

Please see/create sub-tasks for the overall changes for realising this efforts. Thanks!

Issue Links

• is a child of: HDDS-5076 Follow-on work for FileSystemOptimizations(FSO)
• is related to: HDDS-6040 [FSO] Support upgrade for the bucket layout
• relates to: HDDS-4163 Enable filesystem semantics on bucket during creation of bucket

Sub-Tasks

• [FSO] Support bucket layout in OM: Resolved
• [FSO] Handle OMClientRequest based on the bucket layout: Resolved
• Use bucket layout specific DB table in OmMetadataManager:getOpenKeyTable(): Resolved
• [FSO] Define default bucket layout: Resolved
• Use bucket layout specific DB table in OmMetadataManager.getKeyTable() - part1: Resolved
• Follow up task to use bucket layout specific DB table in OmMetadataManager.getKeyTable() - part2: Resolved
• Make sure buckets created from "ofs" be FILE_SYSTEM_OPTIMIZED layout: Resolved
• Make FSO and OBS bucket layouts independent of normalization config flag: Resolved
• Change option name to bucketlayout instead of type: Resolved
• Remove OMKeyRequest#getBucketLayout overridden method in subclasses: Resolved
• Inaccurate bucket info returned from bucket list command in Shell: Resolved
• Remove ozone.om.metadata.layout config: Resolved
• Refactor usage of bucket type: Resolved
• Remove isBucketFSOptimized: Resolved
• Prefix Parser tool should only work for FSO buckets: Resolved
• [FSO] Allow existing/old buckets with any layout during OM startup: Resolved
• Buckets created via link command do not mirror layout of source bucket: Resolved
• [FSO] Refactor isFileSystemOptimized usage in OzoneManagerUtils: Resolved
• Handles bucket layout validation logic in ofs/o3fs client: Resolved
• [FSO] Revisit robot test cases related to prefix/simple metadata layout: Resolved
• [FSO] Remove ozone.om.metadata.layout config in OM: Resolved
• Move BucketManagerImpl#resolveLinkBucketLayout into Utility Class: Resolved
• Remove OMClientResponse#getBucketLayout override in subclasses - part1: Resolved
• Remove OMClientResponse#getBucketLayout override in subclasses - part2: Resolved
• [FSO] Support backward compatible upgrade to a version with key prefix management: Resolved
• Add bucket layout logging to Audit Logs: Resolved
• Unnecessary duplicate smoketest after defaulting to FSO: Resolved
• Ozone sh key list on OBJECT_STORE bucket: Resolved
• Fix number of keys displayed in Recon Overview: Resolved
• Add metrics - number of FSO bucket creates: Resolved
• Add metrics - number of FSO bucket deletes: Resolved
• [Docs] Update Ozone Prefix based FSO documentation.: Resolved
• [FSO] Disable recursive access check flag for directories with no children.: Resolved
• OmMetadataManagerImpl#isBucketEmpty does not work on FSO buckets: Resolved
• Cleanup OMRequest class constructors: Resolved
• Ozone doesn’t reclaim space after deletion of volume via ofs: Resolved
• Refine OMClientRequest Classes to handle bucket layout better.: Resolved
• OzoneManagerUtils#getBucketLayout should not suppress bucket key lookup errors.: Resolved
• DirectoryDeletionService should use getFileTable to handle bucket delete: Resolved
• Modify default bucket layout to LEGACY: Resolved
• Provide OM#getSupportedBucketLayout RPC call for BucketLayout upgrade finalization: Resolved
• Ozone 1.1.0 client listKeys on FSO bucket returns empty list: Resolved
• [FSO] Handle rename and delete operation from an old client to a FSO bucket type: Resolved
• KeyDeletingService failing with IllegalStateExcception: Resolved
• Remove LEGACY from allowed bucket layouts in Java API.: Resolved
• [FSO] Use OM version in client to determine default bucket layout in a finalized OM.: Resolved
• Bucket Layout type as DEFAULT as an option: Resolved
• Bucket create message with layout type: Resolved
• Refactor OM server bucket layout configuration usage: Resolved
• Add LEGACY to AllowedBucketLayouts in CreateBucketHandler: Resolved
• [FSO] KeyRenameRequestWithFSO fails with "org.apache.hadoop.ozone.om.exceptions.OMException: Unable to get file status": Resolved
• [FSO] Sets OM default to LEGACY for an older client's bucket create request: Resolved
• Do not persist LEGACY bucket layout to RocksDB: Resolved
• OBJECT_STORE isn't flat namespaced: Resolved
• OM Requests that depend on bucket layout should validate the layout of their bucket in validateAndUpdateCache: Resolved
• Setting Bucket Property can corrupt bucket layout: Resolved
• OM crashed with OzoneManagerDoubleBuffer: Resolved
• ListKeys : SubPaths with a mixture of files and dirs breaks sorted order: Resolved
• TestTask: Verify read and write keys to an old bucket after upgrade OM: Resolved
• Add a documentation jira for Bucket type feature: Resolved
• Update Recon for files created with FSO Buckets: Resolved
• OM keyTable and fileTable use different key formats.: Patch Available
• [FSO] Deprecate ozone.om.enable.filesystem.paths config key: Open
• [FSO] remove all the dependencies on OzoneManager#getEnableFileSystemPaths.: Patch Available
• Create OBS buckets by default from S3 API: Resolved
• Make default bucket layout FSO in a backwards compatible way: Resolved