Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1312

Re-balance disks within a Datanode

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0-alpha1
    • datanode
    • None
    • Reviewed
    • Hide
      The Disk Balancer lets administrators rebalance data across multiple disks of a DataNode. It is useful to correct skewed data distribution often seen after adding or replacing disks. Disk Balancer can be enabled by setting dfs.disk.balancer.enabled to true in hdfs-site.xml. It can be invoked by running "hdfs diskbalancer". See the "HDFS Diskbalancer" section in the HDFS Commands guide for detailed usage.
      Show
      The Disk Balancer lets administrators rebalance data across multiple disks of a DataNode. It is useful to correct skewed data distribution often seen after adding or replacing disks. Disk Balancer can be enabled by setting dfs.disk.balancer.enabled to true in hdfs-site.xml. It can be invoked by running "hdfs diskbalancer". See the "HDFS Diskbalancer" section in the HDFS Commands guide for detailed usage.

    Description

      Filing this issue in response to ``full disk woes`` on hdfs-user.

      Datanodes fill their storage directories unevenly, leading to situations where certain disks are full while others are significantly less used. Users at many different sites have experienced this issue, and HDFS administrators are taking steps like:

      • Manually rebalancing blocks in storage directories
      • Decomissioning nodes & later readding them

      There's a tradeoff between making use of all available spindles, and filling disks at the sameish rate. Possible solutions include:

      • Weighting less-used disks heavier when placing new blocks on the datanode. In write-heavy environments this will still make use of all spindles, equalizing disk use over time.
      • Rebalancing blocks locally. This would help equalize disk use as disks are added/replaced in older cluster nodes.

      Datanodes should actively manage their local disk so operator intervention is not needed.

      Attachments

        1. Architecture_and_test_update.pdf
          273 kB
          Anu Engineer
        2. Architecture_and_testplan.pdf
          218 kB
          Anu Engineer
        3. disk-balancer-proposal.pdf
          328 kB
          Anu Engineer
        4. HDFS-1312.001.patch
          739 kB
          Anu Engineer
        5. HDFS-1312.002.patch
          739 kB
          Anu Engineer
        6. HDFS-1312.003.patch
          738 kB
          Arpit Agarwal
        7. HDFS-1312.004.patch
          743 kB
          Anu Engineer
        8. HDFS-1312.005.patch
          743 kB
          Arpit Agarwal
        9. HDFS-1312.006.patch
          743 kB
          Anu Engineer
        10. HDFS-1312.007.patch
          744 kB
          Arpit Agarwal

        Issue Links

          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-4408 Balancer for DataNode's Volumes

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-10576 DiskBalancer followup work items

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HDFS-1804 Add a new block-volume device choosing policy that looks at free space

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1121 Allow HDFS client to measure distribution of blocks across devices for a specific DataNode

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1120 Make DataNode's block-to-device placement policy pluggable

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-8538 Change the default volume choosing policy to AvailableSpaceVolumeChoosingPolicy

          • Major - Major loss of function.
          • Patch Available
          1.
          		 DiskBalancer : Add DataModels Sub-task Resolved Anu Engineer
          2.
          		 DiskBalancer : Add connectors Sub-task Resolved Anu Engineer
          3.
          		 DiskBalancer : Add Planner Sub-task Resolved Anu Engineer
          4.
          		 DiskBalancer: change htrace...JsonIgnore to codehaus...JsonIgnore Sub-task Resolved Xiaobing Zhou
          5.
          		 DiskBalancer : Replace htrace json imports with jackson Sub-task Resolved Anu Engineer
          6.
          		 DiskBalancer : Add submitDiskbalancer RPC Sub-task Resolved Anu Engineer
          7.
          		 DiskBalancer : Add cancelPlan RPC Sub-task Resolved Anu Engineer
          8.
          		 DiskBalancer : Add Query RPC Sub-task Resolved Anu Engineer
          9.
          		 DiskBalancer : Add getRuntimeSettings Sub-task Resolved Anu Engineer
          10.
          		 DiskBalancer : SubmitPlan implementation Sub-task Resolved Anu Engineer
          11.
          		 DiskBalancer : Add QueryPlan implementation Sub-task Resolved Anu Engineer
          12.
          		 DiskBalancer : Add cancelPlan implementation Sub-task Resolved Anu Engineer
          13.
          		 Suppress Jenkins warning for sample JSON file Sub-task Resolved Xiaobing Zhou
          14.
          		 Use SLF4J in new classes Sub-task Resolved Anu Engineer
          15.
          		 Stop tracking CHANGES.txt in the HDFS-1312 feature branch. Sub-task Resolved Xiaobing Zhou
          16.
          		 DiskBalancer : getVolumeMap implementation Sub-task Resolved Anu Engineer
          17.
          		 DiskBalancer : getBandwidth implementation Sub-task Resolved Anu Engineer
          18.
          		 DiskBalancer: Force rebase diskbalancer branch Sub-task Resolved Anu Engineer
          19.
          		 DiskBalancer : Add tests for disk balancer using a Mock Mover class. Sub-task Resolved Anu Engineer
          20.
          		 DiskBalancer : Add configuration parameters Sub-task Resolved Anu Engineer
          21.
          		 DiskBalancer : Command utitlities Sub-task Resolved Anu Engineer
          22.
          		 DiskBalancer : Refactor moveBlockAcrossStorage to be used by disk balancer Sub-task Resolved Anu Engineer
          23.
          		 DiskBalancer : Add Data mover Sub-task Resolved Anu Engineer
          24.
          		 DiskBalancer : Add Plan Command Sub-task Resolved Anu Engineer
          25.
          		 DiskBalancer : Add Execute command Sub-task Resolved Anu Engineer
          26.
          		 DiskBalancer: Add JMX for DiskBalancer Sub-task Resolved Anu Engineer
          27.
          		 DiskBalancer: Add QueryStatus command Sub-task Resolved Anu Engineer
          28.
          		 DiskBalancer: Add cancel command Sub-task Resolved Anu Engineer
          29.
          		 DiskBalancer : Add user documentation Sub-task Resolved Anu Engineer
          30.
          		 DiskBalancer: Plan command output directory should be a sub-directory Sub-task Resolved Anu Engineer
          31.
          		 DiskBalancer: resolve volume path names Sub-task Resolved Anu Engineer
          32.
          		 DiskBalancer: ExecuteCommand checks planFile in a wrong way Sub-task Resolved Lei (Eddy) Xu
          33.
          		 Diskbalancer: Print out information when a plan is not generated. Sub-task Resolved Anu Engineer
          34.
          		 DiskBalancer: Use the default datanode port if port is not provided. Sub-task Resolved Anu Engineer
          35.
          		 DiskBalancer: Add Report Command Sub-task Resolved Xiaobing Zhou
          36.
          		 DiskBalancer: Refactor Plan Command Sub-task Resolved Xiaobing Zhou
          37.
          		 DiskBalancer: Refactor Execute Command Sub-task Resolved Xiaobing Zhou
          38.
          		 DiskBalancer: Support help command Sub-task Resolved Anu Engineer
          39.
          		 DiskBalancer: Pretty-print json in Query command Sub-task Resolved Anu Engineer
          40.
          		 DiskBalancer: Fix Checkstyle issues in test code Sub-task Resolved Anu Engineer
          41.
          		 Diskbalancer: The CLI error message for disk balancer is not enabled is not clear. Sub-task Resolved Anu Engineer
          42.
          		 Diskbalancer: When no actions in plan, error message says "Plan was generated more than 24 hours ago" Sub-task Resolved Anu Engineer
          43.
          		 DiskBalancer: PlanCommand should use -fs instead of -uri to be consistent with other hdfs commands Sub-task Resolved Anu Engineer
          44.
          		 DiskBalancer: fix whitespace issue in doc files Sub-task Resolved Anu Engineer
          45.
          		 DiskBalancer: fix issue of order dependency in iteration in ReportCommand test Sub-task Resolved Xiaobing Zhou
          46.
          		 Fix handling of the -fs Generic option Sub-task Resolved Arpit Agarwal
          47.
          		 o.a.h.h.s.diskbalancer.command.Command does not actually verify options as expected. Sub-task Resolved Anu Engineer
          48.
          		 DiskBalancer "-query" results in NPE if no plan for the node Sub-task Resolved Anu Engineer
          49.
          		 TestDiskBalancerCommand#testPlanNode failed with IllegalArgumentException Sub-task Resolved Xiaobing Zhou
          50.
          		 DiskBalancer: Add missing package-info Sub-task Resolved Anu Engineer

          Activity

            People

              aengineer Anu Engineer
              traviscrawford Travis Crawford
              Votes:
              27 Vote for this issue
              Watchers:
              107 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: