Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13283

Percentage based Reserved Space Calculation for DataNode

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Currently, the only way to configure reserved disk space for non-HDFS data on a DataNode is a constant value via dfs.datanode.du.reserved. This can be an issue in non-heterogeneous clusters where size of DNs can differ. The proposed solution is to allow percentage based configuration (and their combination):

      1. ABSOLUTE
        • based on absolute number of reserved space
      2. PERCENTAGE
        • based on percentage of total capacity in the storage
      3. CONSERVATIVE
        • calculates both of the above and takes the one that will yield more reserved space
      4. AGGRESSIVE
        • calculates 1. 2. and takes the one that will yield less reserved space

       

        Attachments

        1. HDFS-13283.007.patch
          32 kB
          Lukas Majercak
        2. HDFS-13283.006.patch
          32 kB
          Lukas Majercak
        3. HDFS-13283.005.patch
          31 kB
          Chris Douglas
        4. HDFS-13283.004.patch
          31 kB
          Chris Douglas
        5. HDFS-13283.003.patch
          31 kB
          Lukas Majercak
        6. HDFS-13283.002.patch
          31 kB
          Lukas Majercak
        7. HDFS-13283.001.patch
          30 kB
          Lukas Majercak
        8. HDFS-13283.000.patch
          28 kB
          Lukas Majercak
        9. HDFS-13283_branch-3.0.000.patch
          30 kB
          Lukas Majercak
        10. HDFS-13283_branch-2.000.patch
          27 kB
          Lukas Majercak

          Issue Links

            Activity

              People

              • Assignee:
                lukmajercak Lukas Majercak
                Reporter:
                lukmajercak Lukas Majercak
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: