Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2832 Enable support for heterogeneous storages in HDFS - DN as a collection of storages
  3. HDFS-4988

Datanode must support all the volumes as individual storages

    XMLWordPrintableJSON

Details

    Description

      Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires:

      1. A unique storage ID for each storage
        • This needs to be generated during formatting
      2. There should be an option to allow existing disks to be reported as single storage unit for backward compatibility.
      3. A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units.
      4. Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)
      5. Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis.

      My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss.

      Attachments

        1. HDFS-4988.09.patch
          53 kB
          Arpit Agarwal
        2. HDFS-4988.08.patch
          54 kB
          Arpit Agarwal
        3. HDFS-4988.07.patch
          56 kB
          Arpit Agarwal
        4. HDFS-4988.06.patch
          61 kB
          Arpit Agarwal
        5. HDFS-4988.05.patch
          101 kB
          Arpit Agarwal
        6. HDFS-4988.02.patch
          147 kB
          Arpit Agarwal
        7. HDFS-4988.01.patch
          141 kB
          Arpit Agarwal

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue HDFS-4985 Add storage type to the protocol and expose it in block report and block locations

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDFS-5000 DataNode configuration should allow specifying storage type

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDFS-5232 Protocol changes to transmit StorageUuid

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDFS-5233 Use Datanode UUID to identify Datanodes

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-5153 Datanode should send block reports for each storage in a separate message

          • Major - Major loss of function.
          • Closed
          is duplicated by

          Sub-task - The sub-task of the issue HDFS-5155 Deprecate API of getStorageID() and setStorageID() in DatanodeID as multiple storages

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HDFS-5264 Consider renaming StorageID

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDFS-5377 Heartbeats from Datandode should include one storage report per storage directory

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDFS-5390 Send one incremental block report per storage directory

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              arp Arpit Agarwal
              sureshms Suresh Srinivas
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: