Details

    • Type: New Feature
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      HDFS Scalability-v2.pdf describes areas where HDFS does well and its scaling challenges and how to address those challenges. Scaling HDFS requires scaling the namespace layer and also the block layer. This jira provides a new block layer,  Hadoop Distributed Storage Layer (HDSL), that scales the block layer by grouping blocks into containers thereby reducing the block-to-location map and also reducing the number of block reports and their processing

      A scalable namespace can be put on top this scalable block layer:

      • HDFS-10419 describes how the existing NN can be modified to use the new block layer.
      • HDFS-13074  also provides, as an interim step; a scalable flat Key-Value namespace  on top of the new block layer; while it does not provide the HDFS API, it does support the Hadoop FS APIs (Hadoop FileSystem, FileContext).

       

      Old Description

      This jira proposes to add object store capabilities into HDFS.
      As part of the federation work (HDFS-1052) we separated block storage as a generic storage layer. Using the Block Pool abstraction, new kinds of namespaces can be built on top of the storage layer i.e. datanodes.
      In this jira I will explore building an object store using the datanode storage, but independent of namespace metadata.

      I will soon update with a detailed design document.

        Attachments

        1. HDFS-7240.007.patch
          6.15 MB
          Lokesh Jain
        2. HDFS Scalability-v2.pdf
          443 kB
          Sanjay Radia
        3. HadoopStorageLayerSecurity.pdf
          445 kB
          Anu Engineer
        4. MeetingMinutes.pdf
          94 kB
          Anu Engineer
        5. HDFS-7240.006.patch
          6.13 MB
          Mukul Kumar Singh
        6. HDFS-7240.005.patch
          6.03 MB
          Mukul Kumar Singh
        7. HDFS Scalability and Ozone.pdf
          328 kB
          Sanjay Radia
        8. HDFS-7240.004.patch
          5.96 MB
          Mukul Kumar Singh
        9. HDFS-7240.003.patch
          5.96 MB
          Mukul Kumar Singh
        10. HDFS-7240.003.patch
          5.96 MB
          Mukul Kumar Singh
        11. HDFS-7240.002.patch
          5.96 MB
          Mukul Kumar Singh
        12. HDFS-7240.001.patch
          5.83 MB
          Mukul Kumar Singh
        13. Ozonedesignupdate.pdf
          970 kB
          Anu Engineer
        14. ozone_user_v0.pdf
          4.48 MB
          Anu Engineer
        15. Ozone-architecture-v1.pdf
          1.60 MB
          Jitendra Nath Pandey

          Issue Links

          1.
          Ozone: XceiverClientManager should cache objects based on pipeline name Sub-task Patch Available Mukul Kumar Singh
          2.
          Ozone: Improve SQLCLI performance Sub-task Patch Available Yuanbo Liu
          3.
          Ozone: Container : Add key versioning support-2 Sub-task Patch Available Chen Liang
          4.
          Ozone: Non-admin user is unable to run InfoVolume to the volume owned by itself Sub-task In Progress Lokesh Jain
          5.
          Ozone: KSM: Allocate key should honour volume quota if quota is set on the volume Sub-task In Progress Lokesh Jain
          6.
          Ozone : debug cli: add support to load user-provided SQL query Sub-task In Progress Chen Liang
          7.
          Ozone: Container : Add key versioning support-3 Sub-task In Progress Chen Liang
          8.
          Ozone : add read/write random access to Chunks of a key Sub-task In Progress Chen Liang
          9.
          Ozone: Container : Add key versioning support-4 Sub-task In Progress Chen Liang
          10.
          Ozone: Parallelize ChunkOutputSream Writes to container Sub-task Patch Available Shashikant Banerjee
          11.
          Make ContainerStateMachine#applyTransaction async Sub-task Patch Available Lokesh Jain
          12.
          Ozone: SCM: avoid synchronously loading all the keys from containers upon SCM datanode start Sub-task Open Xiaoyu Yao
          13.
          Ozone: Ratis: Moving Ratis pipeline creation to client Sub-task Open Unassigned
          14.
          Ozone: write deleted block to RAFT log for consensus on datanodes Sub-task Open Unassigned
          15.
          Ozone: Documentation: Add Ozone-defaults documentation Sub-task Open Ajay Kumar
          16.
          Ozone: Purge metadata of deleted blocks after max retry times Sub-task Open Yuanbo Liu
          17.
          Ozone: SCM CLI: Implement get container metrics command Sub-task Open Yuanbo Liu
          18.
          Ozone: fix and reenable TestKeysRatis#testPutAndGetKeyWithDnRestart Sub-task Open Mukul Kumar Singh
          19.
          Ozone: Support range in getKey operation Sub-task Open Mukul Kumar Singh
          20.
          Ozone:SCM: explore if we need 3 maps for tracking the state of nodes Sub-task Open Unassigned
          21.
          Ozone: add TestDistributedOzoneVolumesRatis, TestOzoneRestWithMiniClusterRatis and TestOzoneWebAccessRatis Sub-task Open Tsz Wo Nicholas Sze
          22.
          Ozone: SCM: clean up containers that timeout during creation Sub-task Open Xiaoyu Yao
          23.
          Ozone: In Ratis, leader should validate ContainerCommandRequestProto before propagating it to followers Sub-task Open Tsz Wo Nicholas Sze
          24.
          Ozone: Ozone shell: the root is assumed to hdfs Sub-task Open Shashikant Banerjee
          25.
          Ozone: change TestRatisManager to check cluster with data Sub-task Open Tsz Wo Nicholas Sze
          26.
          Ozone : Optimize putKey operation to be async and consensus Sub-task Open Weiwei Yang
          27.
          Ozone: provide a way to validate ContainerCommandRequestProto Sub-task Open Anu Engineer
          28.
          ChunkManager functions do not use the argument keyName Sub-task Open Chen Liang
          29.
          Ozone: Document ozone metadata directory structure Sub-task Open Xiaoyu Yao
          30.
          OZone: SCM CLI: Implement get container command Sub-task Open Chen Liang
          31.
          Ozone: Compact DB should be called on Open Containers. Sub-task Open Weiwei Yang
          32.
          Ozone: Support SCM multiple instance for HA Sub-task Open Anu Engineer
          33.
          Ozone: KSM : Support for simulated file system operations Sub-task Open Shashikant Banerjee
          34.
          Ozone: Handle potential inconsistent states while listing keys Sub-task Open Weiwei Yang
          35.
          Ozone: Corona: Support for online mode Sub-task Open Nanda kumar
          36.
          Ozone: Add client Block cache for SCM Sub-task Open Mukul Kumar Singh
          37.
          Ozone: Put key operation concurrent executes failed on Windows Sub-task Open Unassigned
          38.
          Ozone: SCM: Lease support for pipeline creation Sub-task Open Unassigned
          39.
          Ozone: OzoneFileSystem: Add StorageStatistics to OzoneFileSystem Sub-task Open Mukul Kumar Singh
          40.
          Ozone: SCM: Make container report processing async Sub-task Open Unassigned
          41.
          Ozone: Service Discovery: Limit the number of datanodes (REST servers) returned by KeySpaceManager#getServiceList call Sub-task Open Unassigned
          42.
          Ozone: Add support for Larger key sizes in Corona Sub-task Open Mukul Kumar Singh
          43.
          Ozone: OzoneFileSystem: Implement seek functionality for rest client Sub-task Open Lokesh Jain
          44.
          Ozone: OzoneClient: Add configuration parameter to choose RestServerSelector implementation which will be used by RestClient Sub-task Open Unassigned
          45.
          Ozone: JMX name of RocksDbMetastore should be sanitized Sub-task Patch Available Elek, Marton
          46.
          Ozone: Handle datanode failures in Storage Container Manager Sub-task Patch Available Shashikant Banerjee
          47.
          Ozone: Format open containers on datanode restart Sub-task Patch Available Lokesh Jain
          48.
          Ozone: DeleteContainer call in SCM does not remove the container info from the container inmemory maps. Sub-task Open Shashikant Banerjee
          49.
          Ozone: Introduce a new SCM Exception which will be thrown when mandatory property is missing Sub-task Patch Available LiXin Ge
          50.
          Ozone: Store only datanode UUID in datanode.id file Sub-task Open Unassigned
          51.
          Ozone: Remove commands from command queue when the datanode is declared dead Sub-task Open Shashikant Banerjee
          52.
          Ozone: Handle Datanode Registration failure Sub-task Patch Available Hanisha Koneru
          53.
          Ozone: TestBlockDeletingService test-case is failing Sub-task Patch Available LiXin Ge
          54.
          Httpserver start failure should be non fatal for KSM and SCM startup Sub-task Patch Available Ajay Kumar
          55.
          Ozone: Fix precommit hook for Ozone/Hdds on trunk Sub-task Open Elek, Marton

            Activity

              People

              • Assignee:
                jnp Jitendra Nath Pandey
                Reporter:
                jnp Jitendra Nath Pandey
              • Votes:
                18 Vote for this issue
                Watchers:
                225 Start watching this issue

                Dates

                • Created:
                  Updated: