[HDDS-2939] Ozone FS namespace

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Ozone Manager
• Labels: Triaged
• Assignee: Rakesh Radhakrishnan
• Reporter: Supratim Deka
• Created: 27/Jan/20 16:27
• Updated: 19/Aug/22 02:45
• Resolved: 09/Jun/21 03:02

Description

Create the structures and metadata layout required to support efficient FS namespace operations in Ozone - operations involving folders/directories required to support the Hadoop compatible Filesystem interface.

The details are described in the attached document. The work is divided up into sub-tasks as per the task list in the document.

Issue Links

• blocks: HDDS-3545 MR Jobhistory cannot work well with o3fs hadoop compatible file system
• is duplicated by: HDDS-1221 Introduce fine grained lock in Ozone Manager for key operations
• is duplicated by: HDDS-24 Ozone: Rename directory in ozonefs should be atomic
• is related to: HDDS-4222 [OzoneFS optimization] Provide a mechanism for efficient path lookup
• relates to: HDDS-3402 Use proper acls for sub directories created during CreateDirectory operation
• relates to: HDDS-5076 Follow-on work for FileSystemOptimizations(FSO)
• relates to: HDDS-4593 Refine IAccessAuthorizer interface to do recursive ACL check on a path
• links to: GitHub Pull Request #1668
• links to: GitHub Pull Request #2150

Sub-Tasks

• mkdir : create key table entries for intermediate directories in the path: Resolved
• file create : create key table entries for intermediate directories in the path: Resolved
• create file : remove key table iterator: Resolved
• OM File create request does not check for existing directory with the same name: Resolved
• mkdir : store directory entries in a separate table: Resolved
• Incorporate HDDS-4308 quota changes to HDDS-2939 branch: Resolved
• CreateFile : store parent dir entries into DirTable and file entry into separate FileTable: Resolved
• ListFileStatus - do lookup in directory and file tables: Resolved
• Rename : make rename an atomic ops by updating key path entry in dir/file table: Resolved
• Delete : make delete an atomic operation: Resolved
• Fix compilation errors : merge HDDS-4308 and HDDS-4473 changes into the branch: Resolved
• Directory table, fileTable and openFile Table is missing from the OM DB Definition: Resolved
• AllocateBlock : lookup and update open file table for the given path: Resolved
• Putkey : create key table entries for intermediate directories in the key path: Resolved
• LookupKey: do lookup in dir and file tables: Resolved
• Fix TestOzoneFileSystemV1 and TestObjectStoreV1 cases: Resolved
• RenameKey : add unit test to verify bucket#renameKey: Resolved
• Feature Config : Make proper enableFSPaths and OFS optimized flag combinations: Resolved
• [FSO]S3MultiPart: Implement InitiateMultiPartUpload: Resolved
• [FSO]Fix findbugs issues after HDDS-2195: Resolved
• [FSO]S3MultiPart: Implement create and commit upload part file: Resolved
• Make trash work with FS Optimised Buckets.: Resolved
• [FSO]S3Multipart: Implement UploadCompleteRequest: Resolved
• [FSO]OzoneContract unit test case fixes: Resolved
• [FSO]ListKeys: do lookup in dir and file tables: Resolved
• [FSO]RenameAndDelete : make ofs#rename and ofs#delete an atomic operation: Resolved
• [FSO]S3Multipart: Implement UploadAbortRequest: Resolved
• [FSO]S3Multipart: Implement OzoneBucket#listParts: Resolved
• [FSO]S3Multipart: Implement OzoneBucket#listParts: Resolved
• [FSO] Missed to cleanup new FileTables in OMRequests: Resolved
• [FSO]Add a tool to parse entries in the prefix format: Resolved
• [FSO]Implement ACL requests for new layout: Resolved
• [FSO] Provide list subpaths function to perform recursive ACL check during delete and rename op: Resolved
• [FSO]Delete : Implement async cleanup of garbage and orphan sub-dirs/files: Resolved
• [FSO] Add robot tests for new Filesystem layout: Resolved
• [FSO]Authorizer: OM can do recursive ACL check for subpaths: Resolved
• [FSO]OMConfig: Modify 'ozone.om.layout.version' config: Resolved
• [FSO] Addendum patch to fix compilation error: Resolved
• [FSO] Add documentation: Resolved
• Object ID generator for directory and file: Resolved
• Clean up unused code after HDDS-2940 and HDDS-2942: Resolved
• [FSO]OzoneNativeAuthorizer : Recursive ACL check for subpaths: Resolved
• ListFileStatus : list all the descendants of a given dir recursively: Resolved
• Fix logging in OMFileRequest and OzoneManager: Resolved
• Assign Object ID from OMKeyInfo in PrefixManagerImpl.addAcl and setAcl: Resolved
• [FSO] Improve KeyDeletingService to cleanup FSO files: Resolved
• [FSO] Rename om metadata layout configuration value LEGACY: Resolved
• [FSO] Update status and links in the design doc section of the Ozone docs: Resolved
• [FSO] Avoid using V1 postfixes for prefix related classes: Resolved
• [FSO] Reducing time of ozonefs acceptance testmatrix: Resolved
• [FSO] Cleanup integration tests and reduce the execution time: Resolved
• [FSO] Fail OM startup when turn on prefix layout with old buckets: Resolved
• [FSO] S3MultiPart: Use existing ozone key format for MPU Info in DB: Resolved
• [FSO]S3Multipart: Run S3 acceptance test with prefix layout: Resolved
• [FSO] Fix checkstyle warnings after branch rebase: Resolved
• [FSO] Fix minor checkstyle warnings: Resolved
• [FSO] Add Volume and Bucket ID to the key path for FSO Objects: Resolved
• Update PrefixFSO.md after HDDS-6768: Open
• Add VolumeID and BucketID to OmDirectoryInfo: Open