Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-5672

Support protocol aware buckets within a single OM

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None

    Description

      Presently key path normalization and prefix(HDDS-2939) feature configurations ozone.om.enable.filesystem.paths and ozone.om.metadata.layout are at OM cluster-level. Using these configurations, OM cluster can be configured and started to satisfy only a specific use case, either ‘Pure S3' or ‘FileSystem’ semantics.

      This is an umbrella jira talking about moving the above two cluster-level configurations to a bucket level configuration so that single OM will efficiently hold multiple buckets with different modes at at time. A user who wants to satisfy ‘Pure S3' use case, he can define the bucket in that mode and other user who wants to satisfy ‘FileSystem’ use case, he can define the bucket in that mode.

      Please see/create sub-tasks for the overall changes for realising this efforts. Thanks!

      Attachments

        1. Ozone Bucket Types_23_June_2021.pdf
          472 kB
          Rakesh Radhakrishnan

        Issue Links

          1.
          [FSO] Support bucket layout in OM Sub-task Resolved Aryan Gupta
          2.
          [FSO] Handle OMClientRequest based on the bucket layout Sub-task Resolved Aryan Gupta
          3.
          Use bucket layout specific DB table in OmMetadataManager:getOpenKeyTable() Sub-task Resolved Aryan Gupta
          4.
          [FSO] Define default bucket layout Sub-task Resolved Jyotinder Singh
          5.
          Use bucket layout specific DB table in OmMetadataManager.getKeyTable() - part1 Sub-task Resolved Aryan Gupta
          6.
          Follow up task to use bucket layout specific DB table in OmMetadataManager.getKeyTable() - part2 Sub-task Resolved Aryan Gupta
          7.
          Make sure buckets created from "ofs" be FILE_SYSTEM_OPTIMIZED layout Sub-task Resolved Jyotinder Singh
          8.
          Make FSO and OBS bucket layouts independent of normalization config flag Sub-task Resolved Aryan Gupta
          9.
          Change option name to bucketlayout instead of type Sub-task Resolved Rakesh Radhakrishnan
          10.
          Remove OMKeyRequest#getBucketLayout overridden method in subclasses Sub-task Resolved Rakesh Radhakrishnan
          11.
          Inaccurate bucket info returned from bucket list command in Shell Sub-task Resolved Jyotinder Singh
          12.
          Remove ozone.om.metadata.layout config Sub-task Resolved Mukul Kumar Singh
          13.
          Refactor usage of bucket type Sub-task Resolved Mukul Kumar Singh
          14.
          Remove isBucketFSOptimized Sub-task Resolved Mukul Kumar Singh
          15.
          Prefix Parser tool should only work for FSO buckets Sub-task Resolved Mukul Kumar Singh
          16.
          [FSO] Allow existing/old buckets with any layout during OM startup Sub-task Resolved zhengchenyu
          17.
          Buckets created via link command do not mirror layout of source bucket Sub-task Resolved Jyotinder Singh
          18.
          [FSO] Refactor isFileSystemOptimized usage in OzoneManagerUtils Sub-task Resolved Rakesh Radhakrishnan
          19.
          Handles bucket layout validation logic in ofs/o3fs client Sub-task Resolved Jyotinder Singh
          20.
          [FSO] Revisit robot test cases related to prefix/simple metadata layout Sub-task Resolved Jyotinder Singh
          21.
          [FSO] Remove ozone.om.metadata.layout config in OM Sub-task Resolved Jyotinder Singh
          22.
          Move BucketManagerImpl#resolveLinkBucketLayout into Utility Class Sub-task Resolved Jyotinder Singh
          23.
          Remove OMClientResponse#getBucketLayout override in subclasses - part1 Sub-task Resolved Aryan Gupta
          24.
          Remove OMClientResponse#getBucketLayout override in subclasses - part2 Sub-task Resolved Jyotinder Singh
          25.
          [FSO] Support backward compatible upgrade to a version with key prefix management Sub-task Resolved Ethan Rose
          26.
          Add bucket layout logging to Audit Logs Sub-task Resolved Jyotinder Singh
          27.
          Unnecessary duplicate smoketest after defaulting to FSO Sub-task Resolved Attila Doroszlai
          28.
          Ozone sh key list on OBJECT_STORE bucket Sub-task Resolved Jyotinder Singh
          29.
          Fix number of keys displayed in Recon Overview Sub-task Resolved Jyotinder Singh
          30.
          Add metrics - number of FSO bucket creates Sub-task Resolved Tanvi Penumudy
          31.
          Add metrics - number of FSO bucket deletes Sub-task Resolved Tanvi Penumudy
          32.
          [Docs] Update Ozone Prefix based FSO documentation. Sub-task Resolved Jyotinder Singh
          33.
          [FSO] Disable recursive access check flag for directories with no children. Sub-task Resolved Jyotinder Singh
          34.
          OmMetadataManagerImpl#isBucketEmpty does not work on FSO buckets Sub-task Resolved Jyotinder Singh
          35.
          Cleanup OMRequest class constructors Sub-task Resolved Jyotinder Singh
          36.
          Ozone doesn’t reclaim space after deletion of volume via ofs Sub-task Resolved Sadanand Shenoy
          37.
          Refine OMClientRequest Classes to handle bucket layout better. Sub-task Resolved Jyotinder Singh
          38.
          OzoneManagerUtils#getBucketLayout should not suppress bucket key lookup errors. Sub-task Resolved Jyotinder Singh
          39.
          DirectoryDeletionService should use getFileTable to handle bucket delete Sub-task Resolved Rakesh Radhakrishnan
          40.
          Modify default bucket layout to LEGACY Sub-task Resolved Jyotinder Singh
          41.
          Provide OM#getSupportedBucketLayout RPC call for BucketLayout upgrade finalization Sub-task Resolved Jyotinder Singh
          42.
          Ozone 1.1.0 client listKeys on FSO bucket returns empty list Sub-task Resolved Jyotinder Singh
          43.
          [FSO] Handle rename and delete operation from an old client to a FSO bucket type Sub-task Resolved Rakesh Radhakrishnan
          44.
          KeyDeletingService failing with IllegalStateExcception Sub-task Resolved Jyotinder Singh
          45.
          Remove LEGACY from allowed bucket layouts in Java API. Sub-task Resolved Jyotinder Singh
          46.
          [FSO] Use OM version in client to determine default bucket layout in a finalized OM. Sub-task Resolved Unassigned
          47.
          Bucket Layout type as DEFAULT as an option Sub-task Resolved Tanvi Penumudy
          48.
          Bucket create message with layout type Sub-task Resolved Tanvi Penumudy
          49.
          Refactor OM server bucket layout configuration usage Sub-task Resolved Tanvi Penumudy
          50.
          Add LEGACY to AllowedBucketLayouts in CreateBucketHandler Sub-task Resolved Tanvi Penumudy
          51.
          [FSO] KeyRenameRequestWithFSO fails with "org.apache.hadoop.ozone.om.exceptions.OMException: Unable to get file status" Sub-task Resolved Rakesh Radhakrishnan
          52.
          [FSO] Sets OM default to LEGACY for an older client's bucket create request Sub-task Resolved Rakesh Radhakrishnan
          53.
          Do not persist LEGACY bucket layout to RocksDB Sub-task Resolved Jyotinder Singh
          54.
          OBJECT_STORE isn't flat namespaced Sub-task Resolved Rakesh Radhakrishnan
          55.
          OM Requests that depend on bucket layout should validate the layout of their bucket in validateAndUpdateCache Sub-task Resolved Jyotinder Singh
          56.
          Setting Bucket Property can corrupt bucket layout Sub-task Resolved George Jahad
          57.
          OM crashed with OzoneManagerDoubleBuffer Sub-task Resolved Rakesh Radhakrishnan
          58.
          ListKeys : SubPaths with a mixture of files and dirs breaks sorted order Sub-task Resolved Rakesh Radhakrishnan
          59.
          TestTask: Verify read and write keys to an old bucket after upgrade OM Sub-task Resolved Aryan Gupta
          60.
          Add a documentation jira for Bucket type feature Sub-task Resolved Rakesh Radhakrishnan
          61.
          Update Recon for files created with FSO Buckets Sub-task Open Jyotinder Singh
          62.
          OM keyTable and fileTable use different key formats. Sub-task Patch Available Jyotinder Singh
          63.
          [FSO] Deprecate ozone.om.enable.filesystem.paths config key Sub-task Open Jyotinder Singh
          64.
          [FSO] remove all the dependencies on OzoneManager#getEnableFileSystemPaths. Sub-task Patch Available Jyotinder Singh
          65.
          Create OBS buckets by default from S3 API Sub-task Open Ethan Rose
          66.
          Make default bucket layout FSO in a backwards compatible way Sub-task Open Ethan Rose

          Activity

            People

              aryangupta1998 Aryan Gupta
              rakeshr Rakesh Radhakrishnan
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: