Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1362

Provide volume management functionality for DataNode

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.23.0
    • Fix Version/s: None
    • Component/s: datanode
    • Labels:
      None
    • Release Note:
      Hide
      Based on the reconfiguration framework provided by HADOOP-7001, enable reconfigure the dfs.datanode.data.dir and add new volumes into service.
      Show
      Based on the reconfiguration framework provided by HADOOP-7001 , enable reconfigure the dfs.datanode.data.dir and add new volumes into service.
    • Tags:
      datanode, volume, reconfigure
    • Target Version/s:

      Description

      The current management unit in Hadoop is a node, i.e. if a node failed, it will be kicked out and all the data on the node will be replicated.

      As almost all SATA controller support hotplug, we add a new command line interface to datanode, thus it can list, add or remove a volume online, which means we can change a disk without node decommission. Moreover, if the failed disk still readable and the node has enouth space, it can migrate data on the disks to other disks in the same node.

      A more detailed design document will be attached.

      The original version in our lab is implemented against 0.20 datanode directly, and is it better to implemented it in contrib? Or any other suggestion?

      1. HDFS-1362.8.patch
        20 kB
        Wang Xu
      2. HDFS-1362.7.patch
        18 kB
        Wang Xu
      3. HDFS-1362.6.patch
        19 kB
        Wang Xu
      4. HDFS-1362.5.patch
        18 kB
        Wang Xu
      5. DataNode Volume Refreshment in HDFS-1362.pdf
        33 kB
        Wang Xu
      6. HDFS-1362.4_w7001.txt
        13 kB
        Wang Xu
      7. HDFS-1362.txt
        46 kB
        Wang Xu
      8. Provide_volume_management_for_DN_v1.pdf
        45 kB
        Wang Xu

        Issue Links

          Activity

            People

            • Assignee:
              Wang Xu
              Reporter:
              Wang Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              34 Start watching this issue

              Dates

              • Created:
                Updated:

                Development