Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1312

Re-balance disks within a Datanode

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      The Disk Balancer lets administrators rebalance data across multiple disks of a DataNode. It is useful to correct skewed data distribution often seen after adding or replacing disks. Disk Balancer can be enabled by setting dfs.disk.balancer.enabled to true in hdfs-site.xml. It can be invoked by running "hdfs diskbalancer". See the "HDFS Diskbalancer" section in the HDFS Commands guide for detailed usage.
      Show
      The Disk Balancer lets administrators rebalance data across multiple disks of a DataNode. It is useful to correct skewed data distribution often seen after adding or replacing disks. Disk Balancer can be enabled by setting dfs.disk.balancer.enabled to true in hdfs-site.xml. It can be invoked by running "hdfs diskbalancer". See the "HDFS Diskbalancer" section in the HDFS Commands guide for detailed usage.

      Description

      Filing this issue in response to ``full disk woes`` on hdfs-user.

      Datanodes fill their storage directories unevenly, leading to situations where certain disks are full while others are significantly less used. Users at many different sites have experienced this issue, and HDFS administrators are taking steps like:

      • Manually rebalancing blocks in storage directories
      • Decomissioning nodes & later readding them

      There's a tradeoff between making use of all available spindles, and filling disks at the sameish rate. Possible solutions include:

      • Weighting less-used disks heavier when placing new blocks on the datanode. In write-heavy environments this will still make use of all spindles, equalizing disk use over time.
      • Rebalancing blocks locally. This would help equalize disk use as disks are added/replaced in older cluster nodes.

      Datanodes should actively manage their local disk so operator intervention is not needed.

        Attachments

        1. Architecture_and_test_update.pdf
          273 kB
          Anu Engineer
        2. Architecture_and_testplan.pdf
          218 kB
          Anu Engineer
        3. disk-balancer-proposal.pdf
          328 kB
          Anu Engineer
        4. HDFS-1312.001.patch
          739 kB
          Anu Engineer
        5. HDFS-1312.002.patch
          739 kB
          Anu Engineer
        6. HDFS-1312.003.patch
          738 kB
          Arpit Agarwal
        7. HDFS-1312.004.patch
          743 kB
          Anu Engineer
        8. HDFS-1312.005.patch
          743 kB
          Arpit Agarwal
        9. HDFS-1312.006.patch
          743 kB
          Anu Engineer
        10. HDFS-1312.007.patch
          744 kB
          Arpit Agarwal

        Issue Links

        1.
        DiskBalancer : Add DataModels Sub-task Resolved Anu Engineer Actions
        2.
        DiskBalancer : Add connectors Sub-task Resolved Anu Engineer Actions
        3.
        DiskBalancer : Add Planner Sub-task Resolved Anu Engineer Actions
        4.
        DiskBalancer: change htrace...JsonIgnore to codehaus...JsonIgnore Sub-task Resolved Xiaobing Zhou Actions
        5.
        DiskBalancer : Replace htrace json imports with jackson Sub-task Resolved Anu Engineer Actions
        6.
        DiskBalancer : Add submitDiskbalancer RPC Sub-task Resolved Anu Engineer Actions
        7.
        DiskBalancer : Add cancelPlan RPC Sub-task Resolved Anu Engineer Actions
        8.
        DiskBalancer : Add Query RPC Sub-task Resolved Anu Engineer Actions
        9.
        DiskBalancer : Add getRuntimeSettings Sub-task Resolved Anu Engineer Actions
        10.
        DiskBalancer : SubmitPlan implementation Sub-task Resolved Anu Engineer Actions
        11.
        DiskBalancer : Add QueryPlan implementation Sub-task Resolved Anu Engineer Actions
        12.
        DiskBalancer : Add cancelPlan implementation Sub-task Resolved Anu Engineer Actions
        13.
        Suppress Jenkins warning for sample JSON file Sub-task Resolved Xiaobing Zhou Actions
        14.
        Use SLF4J in new classes Sub-task Resolved Anu Engineer Actions
        15.
        Stop tracking CHANGES.txt in the HDFS-1312 feature branch. Sub-task Resolved Xiaobing Zhou Actions
        16.
        DiskBalancer : getVolumeMap implementation Sub-task Resolved Anu Engineer Actions
        17.
        DiskBalancer : getBandwidth implementation Sub-task Resolved Anu Engineer Actions
        18.
        DiskBalancer: Force rebase diskbalancer branch Sub-task Resolved Anu Engineer Actions
        19.
        DiskBalancer : Add tests for disk balancer using a Mock Mover class. Sub-task Resolved Anu Engineer Actions
        20.
        DiskBalancer : Add configuration parameters Sub-task Resolved Anu Engineer Actions
        21.
        DiskBalancer : Command utitlities Sub-task Resolved Anu Engineer Actions
        22.
        DiskBalancer : Refactor moveBlockAcrossStorage to be used by disk balancer Sub-task Resolved Anu Engineer Actions
        23.
        DiskBalancer : Add Data mover Sub-task Resolved Anu Engineer Actions
        24.
        DiskBalancer : Add Plan Command Sub-task Resolved Anu Engineer Actions
        25.
        DiskBalancer : Add Execute command Sub-task Resolved Anu Engineer Actions
        26.
        DiskBalancer: Add JMX for DiskBalancer Sub-task Resolved Anu Engineer Actions
        27.
        DiskBalancer: Add QueryStatus command Sub-task Resolved Anu Engineer Actions
        28.
        DiskBalancer: Add cancel command Sub-task Resolved Anu Engineer Actions
        29.
        DiskBalancer : Add user documentation Sub-task Resolved Anu Engineer Actions
        30.
        DiskBalancer: Plan command output directory should be a sub-directory Sub-task Resolved Anu Engineer Actions
        31.
        DiskBalancer: resolve volume path names Sub-task Resolved Anu Engineer Actions
        32.
        DiskBalancer: ExecuteCommand checks planFile in a wrong way Sub-task Resolved Lei (Eddy) Xu Actions
        33.
        Diskbalancer: Print out information when a plan is not generated. Sub-task Resolved Anu Engineer Actions
        34.
        DiskBalancer: Use the default datanode port if port is not provided. Sub-task Resolved Anu Engineer Actions
        35.
        DiskBalancer: Add Report Command Sub-task Resolved Xiaobing Zhou Actions
        36.
        DiskBalancer: Refactor Plan Command Sub-task Resolved Xiaobing Zhou Actions
        37.
        DiskBalancer: Refactor Execute Command Sub-task Resolved Xiaobing Zhou Actions
        38.
        DiskBalancer: Support help command Sub-task Resolved Anu Engineer Actions
        39.
        DiskBalancer: Pretty-print json in Query command Sub-task Resolved Anu Engineer Actions
        40.
        DiskBalancer: Fix Checkstyle issues in test code Sub-task Resolved Anu Engineer Actions
        41.
        Diskbalancer: The CLI error message for disk balancer is not enabled is not clear. Sub-task Resolved Anu Engineer Actions
        42.
        Diskbalancer: When no actions in plan, error message says "Plan was generated more than 24 hours ago" Sub-task Resolved Anu Engineer Actions
        43.
        DiskBalancer: PlanCommand should use -fs instead of -uri to be consistent with other hdfs commands Sub-task Resolved Anu Engineer Actions
        44.
        DiskBalancer: fix whitespace issue in doc files Sub-task Resolved Anu Engineer Actions
        45.
        DiskBalancer: fix issue of order dependency in iteration in ReportCommand test Sub-task Resolved Xiaobing Zhou Actions
        46.
        Fix handling of the -fs Generic option Sub-task Resolved Arpit Agarwal Actions
        47.
        o.a.h.h.s.diskbalancer.command.Command does not actually verify options as expected. Sub-task Resolved Anu Engineer Actions
        48.
        DiskBalancer "-query" results in NPE if no plan for the node Sub-task Resolved Anu Engineer Actions
        49.
        TestDiskBalancerCommand#testPlanNode failed with IllegalArgumentException Sub-task Resolved Xiaobing Zhou Actions
        50.
        DiskBalancer: Add missing package-info Sub-task Resolved Anu Engineer Actions

          Activity

            People

            • Assignee:
              aengineer Anu Engineer
              Reporter:
              traviscrawford Travis Crawford

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment