Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4430

Namenode Web UI capacity report is inconsistent with Balancer

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Changed reporting in the NameNode Web UI to more closely reflect the behavior of the re-balancer. Removed no longer used config parameter dfs.datanode.du.pct from hadoop-default.xml.
      Show
      Changed reporting in the NameNode Web UI to more closely reflect the behavior of the re-balancer. Removed no longer used config parameter dfs.datanode.du.pct from hadoop-default.xml.

      Description

      Solution to 2816 changed

      • Total Capacity definition from (the disk space of all data directories) to (the disk space of all the data directories - the reserved space)
      • We added a new element Present Capacity to the report. It is set to (Used Capacity + Remaining Capacity)
      • We changed the Used Percentage reported from (Used Capacity)/(Total Capacity) to (Used Capacity)/(Present Capacity)
      • All these changes are displayed on Namenode Web UI.

      Balancer functionality
      Balancer script is started with a threshold parameter. It tries to move the blocks from the nodes that have Used % that is more than (Cluster average + threshold) to the nodes that have less than (Cluster average - threshold). Essentially balancer gets all the datanodes used % to with in (the Cluster average +/- threshold).

      Inconsistencies due to the change in 2816
      When MapReduce jobs are run, temporary files are generated. This eats away a lot of space from Present Capacity. The difference between the Total Capacity and the Present Capacity can be huge. Currently balancer computes Used Percentage based (Used Capacity)/(Total Capacity). The Used % the balancer uses could be significantly different from Used % displayed on the Namenode Web UI. When balancer is done balancing, the Namenode Used % might still appear unbalanced.

        Attachments

        1. HADOOP-4430.patch
          22 kB
          Suresh Srinivas
        2. HADOOP-4430.patch
          22 kB
          Suresh Srinivas
        3. HADOOP-4430.patch
          22 kB
          Suresh Srinivas

          Issue Links

            Activity

              People

              • Assignee:
                sureshms Suresh Srinivas
                Reporter:
                sureshms Suresh Srinivas
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: