XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      HDFS Scalability-v2.pdf describes areas where HDFS does well and its scaling challenges and how to address those challenges. Scaling HDFS requires scaling the namespace layer and also the block layer. This jira provides a new block layer,  Hadoop Distributed Storage Layer (HDSL), that scales the block layer by grouping blocks into containers thereby reducing the block-to-location map and also reducing the number of block reports and their processing

      A scalable namespace can be put on top this scalable block layer:

      • HDFS-10419 describes how the existing NN can be modified to use the new block layer.
      • HDFS-13074  also provides, as an interim step; a scalable flat Key-Value namespace  on top of the new block layer; while it does not provide the HDFS API, it does support the Hadoop FS APIs (Hadoop FileSystem, FileContext).

       

      Old Description

      This jira proposes to add object store capabilities into HDFS.
      As part of the federation work (HDFS-1052) we separated block storage as a generic storage layer. Using the Block Pool abstraction, new kinds of namespaces can be built on top of the storage layer i.e. datanodes.
      In this jira I will explore building an object store using the datanode storage, but independent of namespace metadata.

      I will soon update with a detailed design document.

        Attachments

        1. Ozonedesignupdate.pdf
          970 kB
          Anu Engineer
        2. Ozone-architecture-v1.pdf
          1.60 MB
          Jitendra Nath Pandey
        3. ozone_user_v0.pdf
          4.48 MB
          Anu Engineer
        4. MeetingMinutes.pdf
          94 kB
          Anu Engineer
        5. HDFS Scalability-v2.pdf
          443 kB
          Sanjay Radia
        6. HDFS Scalability and Ozone.pdf
          328 kB
          Sanjay Radia
        7. HDFS-7240.007.patch
          6.15 MB
          Lokesh Jain
        8. HDFS-7240.006.patch
          6.13 MB
          Mukul Kumar Singh
        9. HDFS-7240.005.patch
          6.03 MB
          Mukul Kumar Singh
        10. HDFS-7240.004.patch
          5.96 MB
          Mukul Kumar Singh
        11. HDFS-7240.003.patch
          5.96 MB
          Mukul Kumar Singh
        12. HDFS-7240.003.patch
          5.96 MB
          Mukul Kumar Singh
        13. HDFS-7240.002.patch
          5.96 MB
          Mukul Kumar Singh
        14. HDFS-7240.001.patch
          5.83 MB
          Mukul Kumar Singh
        15. HadoopStorageLayerSecurity.pdf
          445 kB
          Anu Engineer

          Issue Links

          1.
          Ozone: XceiverClientManager should cache objects based on pipeline name Sub-task Patch Available Mukul Kumar Singh
          2.
          Ozone: Improve SQLCLI performance Sub-task Patch Available Yuanbo Liu
          3.
          Ozone: Container : Add key versioning support-2 Sub-task Patch Available Chen Liang
          4.
          Ozone: Non-admin user is unable to run InfoVolume to the volume owned by itself Sub-task In Progress Lokesh Jain
          5.
          Ozone: KSM: Allocate key should honour volume quota if quota is set on the volume Sub-task In Progress Lokesh Jain
          6.
          Ozone : debug cli: add support to load user-provided SQL query Sub-task In Progress Chen Liang
          7.
          Ozone: Container : Add key versioning support-3 Sub-task In Progress Chen Liang
          8.
          Ozone : add read/write random access to Chunks of a key Sub-task In Progress Chen Liang
          9.
          Ozone: Container : Add key versioning support-4 Sub-task In Progress Chen Liang
          10.
          Ozone: Parallelize ChunkOutputSream Writes to container Sub-task Patch Available Shashikant Banerjee
          11.
          Ozone: SCM: avoid synchronously loading all the keys from containers upon SCM datanode start Sub-task Open Xiaoyu Yao
          12.
          Ozone: Ratis: Moving Ratis pipeline creation to client Sub-task Open Unassigned
          13.
          Ozone: write deleted block to RAFT log for consensus on datanodes Sub-task Open Unassigned
          14.
          Ozone: Documentation: Add Ozone-defaults documentation Sub-task Open Ajay Kumar
          15.
          Ozone: Purge metadata of deleted blocks after max retry times Sub-task Open Yuanbo Liu
          16.
          Ozone: SCM CLI: Implement get container metrics command Sub-task Open Yuanbo Liu
          17.
          Ozone: fix and reenable TestKeysRatis#testPutAndGetKeyWithDnRestart Sub-task Open Mukul Kumar Singh
          18.
          Ozone: Support range in getKey operation Sub-task Open Mukul Kumar Singh
          19.
          Ozone:SCM: explore if we need 3 maps for tracking the state of nodes Sub-task Open Unassigned
          20.
          Ozone: add TestDistributedOzoneVolumesRatis, TestOzoneRestWithMiniClusterRatis and TestOzoneWebAccessRatis Sub-task Open Tsz-wo Sze
          21.
          Ozone: SCM: clean up containers that timeout during creation Sub-task Open Xiaoyu Yao
          22.
          Ozone: In Ratis, leader should validate ContainerCommandRequestProto before propagating it to followers Sub-task Open Tsz-wo Sze
          23.
          Ozone: Ozone shell: the root is assumed to hdfs Sub-task Open Shashikant Banerjee
          24.
          Ozone: change TestRatisManager to check cluster with data Sub-task Open Tsz-wo Sze
          25.
          Ozone : Optimize putKey operation to be async and consensus Sub-task Open Weiwei Yang
          26.
          Ozone: provide a way to validate ContainerCommandRequestProto Sub-task Open Anu Engineer
          27.
          ChunkManager functions do not use the argument keyName Sub-task Open Chen Liang
          28.
          Ozone: Document ozone metadata directory structure Sub-task Open Xiaoyu Yao
          29.
          OZone: SCM CLI: Implement get container command Sub-task Open Chen Liang
          30.
          Ozone: Compact DB should be called on Open Containers. Sub-task Open Weiwei Yang
          31.
          Ozone: Support SCM multiple instance for HA Sub-task Open Anu Engineer
          32.
          Ozone: KSM : Support for simulated file system operations Sub-task Open Shashikant Banerjee
          33.
          Ozone: Handle potential inconsistent states while listing keys Sub-task Open Weiwei Yang
          34.
          Ozone: Corona: Support for online mode Sub-task Open Nanda kumar
          35.
          Ozone: Add client Block cache for SCM Sub-task Open Mukul Kumar Singh
          36.
          Ozone: Put key operation concurrent executes failed on Windows Sub-task Open Unassigned
          37.
          Ozone: SCM: Lease support for pipeline creation Sub-task Open Unassigned
          38.
          Ozone: OzoneFileSystem: Add StorageStatistics to OzoneFileSystem Sub-task Open Mukul Kumar Singh
          39.
          Ozone: SCM: Make container report processing async Sub-task Open Unassigned
          40.
          Ozone: Service Discovery: Limit the number of datanodes (REST servers) returned by KeySpaceManager#getServiceList call Sub-task Open Unassigned
          41.
          Ozone: Add support for Larger key sizes in Corona Sub-task Open Mukul Kumar Singh
          42.
          Ozone: OzoneFileSystem: Implement seek functionality for rest client Sub-task Open Lokesh Jain
          43.
          Ozone: OzoneClient: Add configuration parameter to choose RestServerSelector implementation which will be used by RestClient Sub-task Open Unassigned
          44.
          Ozone: JMX name of RocksDbMetastore should be sanitized Sub-task Patch Available Marton Elek
          45.
          Ozone: Handle datanode failures in Storage Container Manager Sub-task Patch Available Shashikant Banerjee
          46.
          Ozone: Format open containers on datanode restart Sub-task Patch Available Lokesh Jain
          47.
          Ozone: DeleteContainer call in SCM does not remove the container info from the container inmemory maps. Sub-task Open Shashikant Banerjee
          48.
          Ozone: Introduce a new SCM Exception which will be thrown when mandatory property is missing Sub-task Patch Available LiXin Ge
          49.
          Ozone: Store only datanode UUID in datanode.id file Sub-task Open Unassigned
          50.
          Ozone: Remove commands from command queue when the datanode is declared dead Sub-task Open Shashikant Banerjee
          51.
          Ozone: Handle Datanode Registration failure Sub-task Patch Available Hanisha Koneru
          52.
          Ozone: TestBlockDeletingService test-case is failing Sub-task Patch Available LiXin Ge
          53.
          Ozone: Fix precommit hook for Ozone/Hdds on trunk Sub-task Open Marton Elek

            Activity

              People

              • Assignee:
                jnp Jitendra Nath Pandey
                Reporter:
                jnp Jitendra Nath Pandey
              • Votes:
                21 Vote for this issue
                Watchers:
                237 Start watching this issue

                Dates

                • Created:
                  Updated: