Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-48

ContainerIO - Storage Management

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.2.1
    • None
    • None
    • HDDS Acadia

    Description

      We propose refactoring the HDDS DataNode IO path to enforce clean separation between the Container management and the Storage layers. All components requiring access to HDDS containers on a Datanode should do so via this Storage layer.

      The proposed Storage layer would be responsible for end-to-end disk and volume management. This involves running disk checks and detecting disk failures, distributing data across disks as per the configured policy, collecting performance statistics. 

      Attached Design Doc gives an overview of the proposed class diagram.

      Attachments

        1. ContainerIO-StorageManagement-DesignDoc.pdf
          306 kB
          Hanisha Koneru
        2. HDDS DataNode Disk Layout.pdf
          201 kB
          Bharat Viswanadham
        3. HDDS-48.00.patch
          420 kB
          Bharat Viswanadham
        4. HDDS-48.01.patch
          644 kB
          Bharat Viswanadham
        5. HDDS-48.03.patch
          642 kB
          Arpit Agarwal
        1.
        		 Remove .meta file during creation of container Sub-task Resolved Bharat Viswanadham
        2.
        		 Send ContainerType to Datanode during container creation Sub-task Resolved Bharat Viswanadham
        3.
        		 Merge ContainerData and ContainerStatus classes Sub-task Resolved Bharat Viswanadham
        4.
        		 Create ContainerData, Container classes Sub-task Resolved Bharat Viswanadham
        5.
        		 Use DBType during parsing datanode .container files Sub-task Resolved Bharat Viswanadham
        6.
        		 Create Version File in Datanode Sub-task Resolved Bharat Viswanadham
        7.
        		 Implement VolumeSet to manage disk volumes Sub-task Resolved Hanisha Koneru
        8.
        		 ContainerSet class to manage ContainerMap Sub-task Resolved Bharat Viswanadham
        9.
        		 Change format of .container files to Yaml Sub-task Resolved Bharat Viswanadham
        10.
        		 Rename dbPath,containerFilePath Sub-task Resolved Bharat Viswanadham
        11.
        		 Add DU usage to VolumeInfo Sub-task Resolved Hanisha Koneru
        12.
        		 Implement KeyValueContainer and adopt new disk layout for the containers Sub-task Resolved Bharat Viswanadham
        13.
        		 Implement HDDSVolume to manage volume state Sub-task Resolved Hanisha Koneru
        14.
        		 Refactor KeyManager, ChunkManager Sub-task Resolved Bharat Viswanadham
        15.
        		 Refactor Dispatcher and implement Handler for new ContainerIO design Sub-task Resolved Hanisha Koneru
        16.
        		 Integrate Volumeset, ContainerSet and HddsDispatcher Sub-task Resolved Bharat Viswanadham
        17.
        		 Add Volume IO Stats Sub-task Resolved Bharat Viswanadham
        18.
        		 Add keyCount and container maximum size to ContainerData Sub-task Resolved Bharat Viswanadham
        19.
        		 Add metrics to HddsDispatcher Sub-task Resolved Bharat Viswanadham
        20.
        		 Remove singleton for Handler Sub-task Resolved Bharat Viswanadham
        21.
        		 CleanUp Reimplemented classes Sub-task Resolved Hanisha Koneru
        22.
        		 Modify Integration tests for new ContainerIO classes Sub-task Resolved Bharat Viswanadham
        23.
        		 Add a create container Lock Sub-task Resolved Bharat Viswanadham
        24.
        		 Add updateDeleteTransactionId Sub-task Resolved Bharat Viswanadham
        25.
        		 Handle Container Already Exists exception on client side Sub-task Resolved Unassigned
        26.
        		 Single lock to synchronize KeyValueContainer#update Sub-task Resolved Hanisha Koneru

        Activity

          People

            hanishakoneru Hanisha Koneru
            hanishakoneru Hanisha Koneru
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: