Details

    • Type: New Feature New Feature
    • Status: Reopened
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: datanode
    • Labels:
      None

      Description

      Filing this issue in response to ``full disk woes`` on hdfs-user.

      Datanodes fill their storage directories unevenly, leading to situations where certain disks are full while others are significantly less used. Users at many different sites have experienced this issue, and HDFS administrators are taking steps like:

      • Manually rebalancing blocks in storage directories
      • Decomissioning nodes & later readding them

      There's a tradeoff between making use of all available spindles, and filling disks at the sameish rate. Possible solutions include:

      • Weighting less-used disks heavier when placing new blocks on the datanode. In write-heavy environments this will still make use of all spindles, equalizing disk use over time.
      • Rebalancing blocks locally. This would help equalize disk use as disks are added/replaced in older cluster nodes.

      Datanodes should actively manage their local disk so operator intervention is not needed.

        Issue Links

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Travis Crawford
            • Votes:
              22 Vote for this issue
              Watchers:
              67 Start watching this issue

              Dates

              • Created:
                Updated:

                Development