Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1362

Provide volume management functionality for DataNode

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 2.7.0
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      Hide
      Based on the reconfiguration framework provided by HADOOP-7001, enable reconfigure the dfs.datanode.data.dir and add new volumes into service.
      Show
      Based on the reconfiguration framework provided by HADOOP-7001 , enable reconfigure the dfs.datanode.data.dir and add new volumes into service.
    • Tags:
      datanode, volume, reconfigure

      Description

      The current management unit in Hadoop is a node, i.e. if a node failed, it will be kicked out and all the data on the node will be replicated.

      As almost all SATA controller support hotplug, we add a new command line interface to datanode, thus it can list, add or remove a volume online, which means we can change a disk without node decommission. Moreover, if the failed disk still readable and the node has enouth space, it can migrate data on the disks to other disks in the same node.

      A more detailed design document will be attached.

      The original version in our lab is implemented against 0.20 datanode directly, and is it better to implemented it in contrib? Or any other suggestion?

        Attachments

        1. DataNode Volume Refreshment in HDFS-1362.pdf
          33 kB
          Wang Xu
        2. HDFS-1362.4_w7001.txt
          13 kB
          Wang Xu
        3. HDFS-1362.5.patch
          18 kB
          Wang Xu
        4. HDFS-1362.6.patch
          19 kB
          Wang Xu
        5. HDFS-1362.7.patch
          18 kB
          Wang Xu
        6. HDFS-1362.8.patch
          20 kB
          Wang Xu
        7. HDFS-1362.txt
          46 kB
          Wang Xu
        8. Provide_volume_management_for_DN_v1.pdf
          45 kB
          Wang Xu

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                gnawux Wang Xu
                Reporter:
                gnawux Wang Xu
              • Votes:
                0 Vote for this issue
                Watchers:
                44 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: