Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11257

Evacuate DN when the remaining is negative

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: datanode
    • Labels:
      None

      Description

      Datanodes have a maximum amount of disk they can use. This is set using dfs.datanode.du.reserved. For example, if we have a 1TB disk and we set the reserved to 100GB, the DN can only use ~900GB. However, if we fill the DN and later other processes (e.g., logs or co-located services) start to use the disk space, the remaining space will go to a negative and the used storage >100%.

      The Rebalancer or decommissioning would cover this situation. However, both approaches require administrator intervention while this is a situation that violates the settings. Note that decommisioning, would be too extreme as it would evacuate all the data.

        Attachments

        1. HDFS-11257.000.patch
          29 kB
          Anbang Hu
        2. HDFS-11257.001.patch
          29 kB
          Anbang Hu

          Issue Links

            Activity

              People

              • Assignee:
                huanbang1993 Anbang Hu
                Reporter:
                elgoiri Íñigo Goiri
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: