Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13283

Percentage based Reserved Space Calculation for DataNode

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      Currently, the only way to configure reserved disk space for non-HDFS data on a DataNode is a constant value via dfs.datanode.du.reserved. This can be an issue in non-heterogeneous clusters where size of DNs can differ. The proposed solution is to allow percentage based configuration (and their combination):

      1. ABSOLUTE
        • based on absolute number of reserved space
      2. PERCENTAGE
        • based on percentage of total capacity in the storage
      3. CONSERVATIVE
        • calculates both of the above and takes the one that will yield more reserved space
      4. AGGRESSIVE
        • calculates 1. 2. and takes the one that will yield less reserved space

       

      Attachments

        1. HDFS-13283.007.patch
          32 kB
          Lukas Majercak
        2. HDFS-13283.006.patch
          32 kB
          Lukas Majercak
        3. HDFS-13283.005.patch
          31 kB
          Christopher Douglas
        4. HDFS-13283.004.patch
          31 kB
          Christopher Douglas
        5. HDFS-13283.003.patch
          31 kB
          Lukas Majercak
        6. HDFS-13283.002.patch
          31 kB
          Lukas Majercak
        7. HDFS-13283.001.patch
          30 kB
          Lukas Majercak
        8. HDFS-13283.000.patch
          28 kB
          Lukas Majercak
        9. HDFS-13283_branch-3.0.000.patch
          30 kB
          Lukas Majercak
        10. HDFS-13283_branch-2.000.patch
          27 kB
          Lukas Majercak

        Issue Links

          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-13378 Add the option to set the du reserve by percent on datanode

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-14010 Pass correct DF usage to ReservedSpaceCalculator builder

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              lukmajercak Lukas Majercak
              lukmajercak Lukas Majercak
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: