Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1748

Balancer utilization classification is incomplete

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Federation Branch, 0.23.0
    • Component/s: balancer
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      //Balancer.java
        /* Return true if the given datanode is overUtilized */
        private boolean isOverUtilized(BalancerDatanode datanode) {
          return datanode.utilization > (avgUtilization+threshold);
        }
        
        /* Return true if the given datanode is above average utilized
         * but not overUtilized */
        private boolean isAboveAvgUtilized(BalancerDatanode datanode) {
          return (datanode.utilization <= (avgUtilization+threshold))
              && (datanode.utilization > avgUtilization);
        }
        
        /* Return true if the given datanode is underUtilized */
        private boolean isUnderUtilized(BalancerDatanode datanode) {
          return datanode.utilization < (avgUtilization-threshold);
        }
      
        /* Return true if the given datanode is below average utilized 
         * but not underUtilized */
        private boolean isBelowAvgUtilized(BalancerDatanode datanode) {
              return (datanode.utilization >= (avgUtilization-threshold))
                       && (datanode.utilization < avgUtilization);
        }
      

      Where is datanode.utilization == avgUtilization?

      1. h1748_20110310.patch
        2 kB
        Tsz Wo Nicholas Sze

        Activity

        Hide
        Tsz Wo Nicholas Sze added a comment -

        h1748_20110310.patch: change isBelowAvgUtilized to isBelowOrEqualAvgUtilized

        Show
        Tsz Wo Nicholas Sze added a comment - h1748_20110310.patch: change isBelowAvgUtilized to isBelowOrEqualAvgUtilized
        Hide
        Tsz Wo Nicholas Sze added a comment -

        The recent failures of TestBalancer seems related to this.

        Show
        Tsz Wo Nicholas Sze added a comment - The recent failures of TestBalancer seems related to this.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12473349/h1748_20110310.patch
        against trunk revision 1080380.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.hdfs.TestFileConcurrentReader

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/248//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/248//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/248//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12473349/h1748_20110310.patch against trunk revision 1080380. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/248//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/248//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/248//console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        +1 for the patch. Patch looks good.

        Minor comment: should belowAvgUtilizedDatanode be renamed belowOrEqualAvgUtilizedDatanode for consistency? However I am ok if you do not want to make this change.

        Show
        Suresh Srinivas added a comment - +1 for the patch. Patch looks good. Minor comment: should belowAvgUtilizedDatanode be renamed belowOrEqualAvgUtilizedDatanode for consistency? However I am ok if you do not want to make this change.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks Suresh for the review.

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks Suresh for the review.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #558 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/558/)
        HDFS-1748. Balancer utilization classification is incomplete.

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #558 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/558/ ) HDFS-1748 . Balancer utilization classification is incomplete.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Merged this (r1080836) from trunk to federation.

        Show
        Tsz Wo Nicholas Sze added a comment - Merged this (r1080836) from trunk to federation.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/ )

          People

          • Assignee:
            Tsz Wo Nicholas Sze
            Reporter:
            Tsz Wo Nicholas Sze
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development