Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4635

Move BlockManager#computeCapacity to LightWeightGSet

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0, 2.1.0-beta
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The computeCapacity in BlockManager that calculates the LightWeightGSet capacity as the percentage of total JVM memory should be moved to LightWeightGSet. This helps in other maps that are based on the GSet to make use of the same functionality.

      1. HDFS-4635.b1.patch
        7 kB
        Suresh Srinivas
      2. HDFS-4635.b1.patch
        7 kB
        Suresh Srinivas
      3. HDFS-4635.patch
        9 kB
        Suresh Srinivas
      4. HDFS-4635.patch
        9 kB
        Suresh Srinivas
      5. HDFS-4635.patch
        9 kB
        Suresh Srinivas

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          2m 58s 1 Suresh Srinivas 25/Mar/13 22:27
          Patch Available Patch Available Resolved Resolved
          1d 38m 1 Suresh Srinivas 26/Mar/13 23:05
          Resolved Resolved Closed Closed
          49d 6h 10m 1 Matt Foley 15/May/13 05:15
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.
          Gavin made changes -
          Link This issue is depended upon by HDFS-4434 [ HDFS-4434 ]
          Gavin made changes -
          Link This issue blocks HDFS-4434 [ HDFS-4434 ]
          Suresh Srinivas made changes -
          Fix Version/s 1.2.0 [ 12321657 ]
          Hide
          Suresh Srinivas added a comment -

          I committed the branch-1 to branch-1 and branch-1.2.

          Show
          Suresh Srinivas added a comment - I committed the branch-1 to branch-1 and branch-1.2.
          Suresh Srinivas made changes -
          Attachment HDFS-4635.b1.patch [ 12579065 ]
          Hide
          Suresh Srinivas added a comment -

          Updated patch that fixes indentation issue. I am going to commit this soon.

          Show
          Suresh Srinivas added a comment - Updated patch that fixes indentation issue. I am going to commit this soon.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The indentation below is off. Otherwise, patch looks good. +1

          + // Use 2% of total memory to size the GSet capacity
          +    this.capacity = LightWeightGSet.computeCapacity(2.0, "BlocksMap");
          
          Show
          Tsz Wo Nicholas Sze added a comment - The indentation below is off. Otherwise, patch looks good. +1 + // Use 2% of total memory to size the GSet capacity + this .capacity = LightWeightGSet.computeCapacity(2.0, "BlocksMap" );
          Suresh Srinivas made changes -
          Attachment HDFS-4635.b1.patch [ 12579030 ]
          Hide
          Suresh Srinivas added a comment -

          branch-1 patch.

          Show
          Suresh Srinivas added a comment - branch-1 patch.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1385 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1385/)
          HDFS-4635. Move BlockManager#computeCapacity to LightWeightGSet. Contributed by Suresh Srinivas. (Revision 1461364)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461364
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightGSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestGSet.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1385 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1385/ ) HDFS-4635 . Move BlockManager#computeCapacity to LightWeightGSet. Contributed by Suresh Srinivas. (Revision 1461364) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461364 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightGSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestGSet.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1357 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1357/)
          HDFS-4635. Move BlockManager#computeCapacity to LightWeightGSet. Contributed by Suresh Srinivas. (Revision 1461364)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461364
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightGSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestGSet.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1357 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1357/ ) HDFS-4635 . Move BlockManager#computeCapacity to LightWeightGSet. Contributed by Suresh Srinivas. (Revision 1461364) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461364 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightGSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestGSet.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #168 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/168/)
          HDFS-4635. Move BlockManager#computeCapacity to LightWeightGSet. Contributed by Suresh Srinivas. (Revision 1461364)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461364
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightGSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestGSet.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #168 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/168/ ) HDFS-4635 . Move BlockManager#computeCapacity to LightWeightGSet. Contributed by Suresh Srinivas. (Revision 1461364) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461364 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightGSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestGSet.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3531 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3531/)
          HDFS-4635. Move BlockManager#computeCapacity to LightWeightGSet. Contributed by Suresh Srinivas. (Revision 1461364)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461364
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightGSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestGSet.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3531 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3531/ ) HDFS-4635 . Move BlockManager#computeCapacity to LightWeightGSet. Contributed by Suresh Srinivas. (Revision 1461364) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461364 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightGSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestGSet.java
          Suresh Srinivas made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.0.5-beta [ 12324031 ]
          Resolution Fixed [ 1 ]
          Hide
          Suresh Srinivas added a comment -

          I committed the patch to trunk and branch-2. Thank you Nicholas for the reviews.

          Show
          Suresh Srinivas added a comment - I committed the patch to trunk and branch-2. Thank you Nicholas for the reviews.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12575534/HDFS-4635.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4150//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4150//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575534/HDFS-4635.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4150//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4150//console This message is automatically generated.
          Suresh Srinivas made changes -
          Attachment HDFS-4635.patch [ 12575534 ]
          Hide
          Suresh Srinivas added a comment -

          That is a better of power of two implementation. Changed the patch to incorporate the comments.

          Show
          Suresh Srinivas added a comment - That is a better of power of two implementation. Changed the patch to incorporate the comments.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          TestGSet failed by the round off error in TestGSet.isPowerOfTwo(..): 2^29 = 536870912 but isPowerOfTwo(536870912) = false. We may fix it as below.

            private static boolean isPowerOfTwo(int num) {
              if (num == 0) {
                return true;
              }
              return num > 0 && Integer.bitCount(num) == 1;
            }
          
          Show
          Tsz Wo Nicholas Sze added a comment - TestGSet failed by the round off error in TestGSet.isPowerOfTwo(..): 2^29 = 536870912 but isPowerOfTwo(536870912) = false. We may fix it as below. private static boolean isPowerOfTwo( int num) { if (num == 0) { return true ; } return num > 0 && Integer .bitCount(num) == 1; }
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12575477/HDFS-4635.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.util.TestGSet
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4148//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4148//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575477/HDFS-4635.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.util.TestGSet org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4148//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4148//console This message is automatically generated.
          Suresh Srinivas made changes -
          Attachment HDFS-4635.patch [ 12575477 ]
          Hide
          Suresh Srinivas added a comment -

          For percentage parameter, I think it is better to use a double with range [0.0, 1.0] than an int. We may call it "fraction" as well.

          I feel percentage is better than fraction and easy to understand. I changed percentage to double.

          All other comments addressed.

          Show
          Suresh Srinivas added a comment - For percentage parameter, I think it is better to use a double with range [0.0, 1.0] than an int. We may call it "fraction" as well. I feel percentage is better than fraction and easy to understand. I changed percentage to double. All other comments addressed.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Some comments for LightWeightGSet.computeCapacity(..):

          • For percentage parameter, I think it is better to use a double with range [0.0, 1.0] than an int. We may call it "fraction" as well.
          • The debug message "2%" below should be updated. I suggest using StringUtils.TraditionalBinaryPrefix.long2String(..) since it is more general.
            +      LightWeightGSet.LOG.debug("2% max memory = " + pcMem/(1 << 20) + " MB")
            
          • LightWeightGSet.LOG can be replaced with LOG.
          Show
          Tsz Wo Nicholas Sze added a comment - Some comments for LightWeightGSet.computeCapacity(..): For percentage parameter, I think it is better to use a double with range [0.0, 1.0] than an int. We may call it "fraction" as well. The debug message "2%" below should be updated. I suggest using StringUtils.TraditionalBinaryPrefix.long2String(..) since it is more general. + LightWeightGSet.LOG.debug( "2% max memory = " + pcMem/(1 << 20) + " MB" ) LightWeightGSet.LOG can be replaced with LOG.
          Suresh Srinivas made changes -
          Link This issue blocks HDFS-4434 [ HDFS-4434 ]
          Suresh Srinivas made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Suresh Srinivas made changes -
          Field Original Value New Value
          Attachment HDFS-4635.patch [ 12575412 ]
          Hide
          Suresh Srinivas added a comment -

          Attached patch has the following changes:

          1. Moves computeCapacity to the LightWeightGSet
          2. Makes that method generic to computer any given percentage instead of 2%
          3. Added many unit tests for invalid values and various set of valid values.
          Show
          Suresh Srinivas added a comment - Attached patch has the following changes: Moves computeCapacity to the LightWeightGSet Makes that method generic to computer any given percentage instead of 2% Added many unit tests for invalid values and various set of valid values.
          Suresh Srinivas created issue -

            People

            • Assignee:
              Suresh Srinivas
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development