Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-2939

Ozone FS namespace

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.2.0
    • Ozone Manager

    Description

      Create the structures and metadata layout required to support efficient FS namespace operations in Ozone - operations involving folders/directories required to support the Hadoop compatible Filesystem interface.

      The details are described in the attached document. The work is divided up into sub-tasks as per the task list in the document.

      Attachments

        1. Performance Comparison Between Master and HDDS-2939 branch-Report-001.pdf
          408 kB
          Rakesh Radhakrishnan
        2. OzoneFS Optimizations_DesignOverview_ HDDS-2939.pdf
          93 kB
          Rakesh Radhakrishnan
        3. Ozone FS Namespace Proposal v1.0.docx
          22 kB
          Supratim Deka
        4. OzoneFS - Brainstorming Notes.xlsx
          1.43 MB
          Rakesh Radhakrishnan

        Issue Links

          1.
          mkdir : create key table entries for intermediate directories in the path Sub-task Resolved Supratim Deka

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          2.
          file create : create key table entries for intermediate directories in the path Sub-task Resolved Supratim Deka

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          3.
          create file : remove key table iterator Sub-task Resolved Supratim Deka  
          4.
          OM File create request does not check for existing directory with the same name Sub-task Resolved Rakesh Radhakrishnan

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          5.
          mkdir : store directory entries in a separate table Sub-task Resolved Rakesh Radhakrishnan  
          6.
          Incorporate HDDS-4308 quota changes to HDDS-2939 branch Sub-task Resolved Rakesh Radhakrishnan  
          7.
          CreateFile : store parent dir entries into DirTable and file entry into separate FileTable Sub-task Resolved Rakesh Radhakrishnan  
          8.
          ListFileStatus - do lookup in directory and file tables Sub-task Resolved Rakesh Radhakrishnan  
          9.
          Rename : make rename an atomic ops by updating key path entry in dir/file table Sub-task Resolved Rakesh Radhakrishnan  
          10.
          Delete : make delete an atomic operation Sub-task Resolved Rakesh Radhakrishnan  
          11.
          Fix compilation errors : merge HDDS-4308 and HDDS-4473 changes into the branch Sub-task Resolved Rakesh Radhakrishnan  
          12.
          Directory table, fileTable and openFile Table is missing from the OM DB Definition Sub-task Resolved Mukul Kumar Singh  
          13.
          AllocateBlock : lookup and update open file table for the given path Sub-task Resolved Rakesh Radhakrishnan  
          14.
          Putkey : create key table entries for intermediate directories in the key path Sub-task Resolved Rakesh Radhakrishnan  
          15.
          LookupKey: do lookup in dir and file tables Sub-task Resolved Rakesh Radhakrishnan  
          16.
          Fix TestOzoneFileSystemV1 and TestObjectStoreV1 cases Sub-task Resolved Rakesh Radhakrishnan  
          17.
          RenameKey : add unit test to verify bucket#renameKey Sub-task Resolved Rakesh Radhakrishnan  
          18.
          Feature Config : Make proper enableFSPaths and OFS optimized flag combinations Sub-task Resolved Rakesh Radhakrishnan  
          19.
          [FSO]S3MultiPart: Implement InitiateMultiPartUpload Sub-task Resolved Rakesh Radhakrishnan  
          20.
          [FSO]Fix findbugs issues after HDDS-2195 Sub-task Resolved Rakesh Radhakrishnan  
          21.
          [FSO]S3MultiPart: Implement create and commit upload part file Sub-task Resolved Rakesh Radhakrishnan  
          22.
          Make trash work with FS Optimised Buckets. Sub-task Resolved Sadanand Shenoy  
          23.
          [FSO]S3Multipart: Implement UploadCompleteRequest Sub-task Resolved Rakesh Radhakrishnan  
          24.
          [FSO]OzoneContract unit test case fixes Sub-task Resolved Sadanand Shenoy  
          25.
          [FSO]ListKeys: do lookup in dir and file tables Sub-task Resolved Rakesh Radhakrishnan  
          26.
          [FSO]RenameAndDelete : make ofs#rename and ofs#delete an atomic operation Sub-task Resolved Sadanand Shenoy  
          27.
          [FSO]S3Multipart: Implement UploadAbortRequest Sub-task Resolved Rakesh Radhakrishnan  
          28.
          [FSO]S3Multipart: Implement OzoneBucket#listParts Sub-task Resolved Rakesh Radhakrishnan  
          29.
          [FSO]S3Multipart: Implement OzoneBucket#listParts Sub-task Resolved Rakesh Radhakrishnan  
          30.
          [FSO] Missed to cleanup new FileTables in OMRequests Sub-task Resolved Rakesh Radhakrishnan  
          31.
          [FSO]Add a tool to parse entries in the prefix format Sub-task Resolved Mukul Kumar Singh  
          32.
          [FSO]Implement ACL requests for new layout Sub-task Resolved Sadanand Shenoy  
          33.
          [FSO] Provide list subpaths function to perform recursive ACL check during delete and rename op Sub-task Resolved Rakesh Radhakrishnan  
          34.
          [FSO]Delete : Implement async cleanup of garbage and orphan sub-dirs/files Sub-task Resolved Rakesh Radhakrishnan  
          35.
          [FSO] Add robot tests for new Filesystem layout Sub-task Resolved Sadanand Shenoy  
          36.
          [FSO]Authorizer: OM can do recursive ACL check for subpaths Sub-task Resolved Sadanand Shenoy  
          37.
          [FSO]OMConfig: Modify 'ozone.om.layout.version' config Sub-task Resolved Sadanand Shenoy  
          38.
          [FSO] Addendum patch to fix compilation error Sub-task Resolved Rakesh Radhakrishnan  
          39.
          [FSO] Add documentation Sub-task Resolved Rakesh Radhakrishnan  
          40.
          Object ID generator for directory and file Sub-task Resolved Rakesh Radhakrishnan  
          41.
          Clean up unused code after HDDS-2940 and HDDS-2942 Sub-task Resolved Siyao Meng  
          42.
          [FSO]OzoneNativeAuthorizer : Recursive ACL check for subpaths Sub-task Resolved Unassigned  
          43.
          ListFileStatus : list all the descendants of a given dir recursively Sub-task Resolved Rakesh Radhakrishnan  
          44.
          Fix logging in OMFileRequest and OzoneManager Sub-task Resolved Supratim Deka

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 10m
          45.
          Assign Object ID from OMKeyInfo in PrefixManagerImpl.addAcl and setAcl Sub-task Resolved Supratim Deka  
          46.
          [FSO] Improve KeyDeletingService to cleanup FSO files Sub-task Resolved Sadanand Shenoy  
          47.
          [FSO] Rename om metadata layout configuration value LEGACY Sub-task Resolved Rakesh Radhakrishnan  
          48.
          [FSO] Update status and links in the design doc section of the Ozone docs Sub-task Resolved Marton Elek  
          49.
          [FSO] Avoid using V1 postfixes for prefix related classes Sub-task Resolved Rakesh Radhakrishnan  
          50.
          [FSO] Reducing time of ozonefs acceptance testmatrix Sub-task Resolved Marton Elek  
          51.
          [FSO] Cleanup integration tests and reduce the execution time Sub-task Resolved Rakesh Radhakrishnan  
          52.
          [FSO] Fail OM startup when turn on prefix layout with old buckets Sub-task Resolved Rakesh Radhakrishnan  
          53.
          [FSO] S3MultiPart: Use existing ozone key format for MPU Info in DB Sub-task Resolved Rakesh Radhakrishnan  
          54.
          [FSO]S3Multipart: Run S3 acceptance test with prefix layout Sub-task Resolved Rakesh Radhakrishnan  
          55.
          [FSO] Fix checkstyle warnings after branch rebase Sub-task Resolved Rakesh Radhakrishnan  
          56.
          [FSO] Fix minor checkstyle warnings Sub-task Resolved Rakesh Radhakrishnan  
          57.
          [FSO] Add Volume and Bucket ID to the key path for FSO Objects Sub-task Resolved Nandakumar  
          58.
          Update PrefixFSO.md after HDDS-6768 Sub-task Open pratap chandu  
          59.
          Add VolumeID and BucketID to OmDirectoryInfo Sub-task Open Unassigned  

          Activity

            People

              rakeshr Rakesh Radhakrishnan
              sdeka Supratim Deka
              Votes:
              1 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m