Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4635

Move BlockManager#computeCapacity to LightWeightGSet

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0, 2.1.0-beta
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The computeCapacity in BlockManager that calculates the LightWeightGSet capacity as the percentage of total JVM memory should be moved to LightWeightGSet. This helps in other maps that are based on the GSet to make use of the same functionality.

      1. HDFS-4635.b1.patch
        7 kB
        Suresh Srinivas
      2. HDFS-4635.b1.patch
        7 kB
        Suresh Srinivas
      3. HDFS-4635.patch
        9 kB
        Suresh Srinivas
      4. HDFS-4635.patch
        9 kB
        Suresh Srinivas
      5. HDFS-4635.patch
        9 kB
        Suresh Srinivas

        Issue Links

          Activity

          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.
          Hide
          Suresh Srinivas added a comment -

          I committed the branch-1 to branch-1 and branch-1.2.

          Show
          Suresh Srinivas added a comment - I committed the branch-1 to branch-1 and branch-1.2.
          Hide
          Suresh Srinivas added a comment -

          Updated patch that fixes indentation issue. I am going to commit this soon.

          Show
          Suresh Srinivas added a comment - Updated patch that fixes indentation issue. I am going to commit this soon.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The indentation below is off. Otherwise, patch looks good. +1

          + // Use 2% of total memory to size the GSet capacity
          +    this.capacity = LightWeightGSet.computeCapacity(2.0, "BlocksMap");
          
          Show
          Tsz Wo Nicholas Sze added a comment - The indentation below is off. Otherwise, patch looks good. +1 + // Use 2% of total memory to size the GSet capacity + this .capacity = LightWeightGSet.computeCapacity(2.0, "BlocksMap" );
          Hide
          Suresh Srinivas added a comment -

          branch-1 patch.

          Show
          Suresh Srinivas added a comment - branch-1 patch.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1385 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1385/)
          HDFS-4635. Move BlockManager#computeCapacity to LightWeightGSet. Contributed by Suresh Srinivas. (Revision 1461364)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461364
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightGSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestGSet.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1385 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1385/ ) HDFS-4635 . Move BlockManager#computeCapacity to LightWeightGSet. Contributed by Suresh Srinivas. (Revision 1461364) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461364 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightGSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestGSet.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1357 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1357/)
          HDFS-4635. Move BlockManager#computeCapacity to LightWeightGSet. Contributed by Suresh Srinivas. (Revision 1461364)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461364
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightGSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestGSet.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1357 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1357/ ) HDFS-4635 . Move BlockManager#computeCapacity to LightWeightGSet. Contributed by Suresh Srinivas. (Revision 1461364) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461364 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightGSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestGSet.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #168 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/168/)
          HDFS-4635. Move BlockManager#computeCapacity to LightWeightGSet. Contributed by Suresh Srinivas. (Revision 1461364)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461364
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightGSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestGSet.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #168 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/168/ ) HDFS-4635 . Move BlockManager#computeCapacity to LightWeightGSet. Contributed by Suresh Srinivas. (Revision 1461364) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461364 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightGSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestGSet.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3531 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3531/)
          HDFS-4635. Move BlockManager#computeCapacity to LightWeightGSet. Contributed by Suresh Srinivas. (Revision 1461364)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461364
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightGSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestGSet.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3531 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3531/ ) HDFS-4635 . Move BlockManager#computeCapacity to LightWeightGSet. Contributed by Suresh Srinivas. (Revision 1461364) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461364 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightGSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestGSet.java
          Hide
          Suresh Srinivas added a comment -

          I committed the patch to trunk and branch-2. Thank you Nicholas for the reviews.

          Show
          Suresh Srinivas added a comment - I committed the patch to trunk and branch-2. Thank you Nicholas for the reviews.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12575534/HDFS-4635.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4150//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4150//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575534/HDFS-4635.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4150//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4150//console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          That is a better of power of two implementation. Changed the patch to incorporate the comments.

          Show
          Suresh Srinivas added a comment - That is a better of power of two implementation. Changed the patch to incorporate the comments.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          TestGSet failed by the round off error in TestGSet.isPowerOfTwo(..): 2^29 = 536870912 but isPowerOfTwo(536870912) = false. We may fix it as below.

            private static boolean isPowerOfTwo(int num) {
              if (num == 0) {
                return true;
              }
              return num > 0 && Integer.bitCount(num) == 1;
            }
          
          Show
          Tsz Wo Nicholas Sze added a comment - TestGSet failed by the round off error in TestGSet.isPowerOfTwo(..): 2^29 = 536870912 but isPowerOfTwo(536870912) = false. We may fix it as below. private static boolean isPowerOfTwo( int num) { if (num == 0) { return true ; } return num > 0 && Integer .bitCount(num) == 1; }
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12575477/HDFS-4635.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.util.TestGSet
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4148//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4148//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575477/HDFS-4635.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.util.TestGSet org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4148//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4148//console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          For percentage parameter, I think it is better to use a double with range [0.0, 1.0] than an int. We may call it "fraction" as well.

          I feel percentage is better than fraction and easy to understand. I changed percentage to double.

          All other comments addressed.

          Show
          Suresh Srinivas added a comment - For percentage parameter, I think it is better to use a double with range [0.0, 1.0] than an int. We may call it "fraction" as well. I feel percentage is better than fraction and easy to understand. I changed percentage to double. All other comments addressed.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Some comments for LightWeightGSet.computeCapacity(..):

          • For percentage parameter, I think it is better to use a double with range [0.0, 1.0] than an int. We may call it "fraction" as well.
          • The debug message "2%" below should be updated. I suggest using StringUtils.TraditionalBinaryPrefix.long2String(..) since it is more general.
            +      LightWeightGSet.LOG.debug("2% max memory = " + pcMem/(1 << 20) + " MB")
            
          • LightWeightGSet.LOG can be replaced with LOG.
          Show
          Tsz Wo Nicholas Sze added a comment - Some comments for LightWeightGSet.computeCapacity(..): For percentage parameter, I think it is better to use a double with range [0.0, 1.0] than an int. We may call it "fraction" as well. The debug message "2%" below should be updated. I suggest using StringUtils.TraditionalBinaryPrefix.long2String(..) since it is more general. + LightWeightGSet.LOG.debug( "2% max memory = " + pcMem/(1 << 20) + " MB" ) LightWeightGSet.LOG can be replaced with LOG.
          Hide
          Suresh Srinivas added a comment -

          Attached patch has the following changes:

          1. Moves computeCapacity to the LightWeightGSet
          2. Makes that method generic to computer any given percentage instead of 2%
          3. Added many unit tests for invalid values and various set of valid values.
          Show
          Suresh Srinivas added a comment - Attached patch has the following changes: Moves computeCapacity to the LightWeightGSet Makes that method generic to computer any given percentage instead of 2% Added many unit tests for invalid values and various set of valid values.

            People

            • Assignee:
              Suresh Srinivas
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development