Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1362

Provide volume management functionality for DataNode

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.23.0
    • 2.7.0
    • datanode
    • None
    • Hide
      Based on the reconfiguration framework provided by HADOOP-7001, enable reconfigure the dfs.datanode.data.dir and add new volumes into service.
      Show
      Based on the reconfiguration framework provided by HADOOP-7001 , enable reconfigure the dfs.datanode.data.dir and add new volumes into service.
    • datanode, volume, reconfigure

    Description

      The current management unit in Hadoop is a node, i.e. if a node failed, it will be kicked out and all the data on the node will be replicated.

      As almost all SATA controller support hotplug, we add a new command line interface to datanode, thus it can list, add or remove a volume online, which means we can change a disk without node decommission. Moreover, if the failed disk still readable and the node has enouth space, it can migrate data on the disks to other disks in the same node.

      A more detailed design document will be attached.

      The original version in our lab is implemented against 0.20 datanode directly, and is it better to implemented it in contrib? Or any other suggestion?

      Attachments

        1. Provide_volume_management_for_DN_v1.pdf
          45 kB
          Wang Xu
        2. HDFS-1362.txt
          46 kB
          Wang Xu
        3. HDFS-1362.4_w7001.txt
          13 kB
          Wang Xu
        4. DataNode Volume Refreshment in HDFS-1362.pdf
          33 kB
          Wang Xu
        5. HDFS-1362.5.patch
          18 kB
          Wang Xu
        6. HDFS-1362.6.patch
          19 kB
          Wang Xu
        7. HDFS-1362.7.patch
          18 kB
          Wang Xu
        8. HDFS-1362.8.patch
          20 kB
          Wang Xu

        Issue Links

          Activity

            People

              gnawux Wang Xu
              gnawux Wang Xu
              Votes:
              0 Vote for this issue
              Watchers:
              43 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: