Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3816

Invalidate work percentage default value should be 0.32f instead of 32

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha, 3.0.0
    • Fix Version/s: 2.0.2-alpha
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HDFS-3475 used blocksInvalidateWorkPct/DFS_NAMENODE_INVALIDATE_WORK_PCT_PER_ITERATION_DEFAULT as a percentage float number. Thus DFS_NAMENODE_INVALIDATE_WORK_PCT_PER_ITERATION_DEFAULT should be defined as a float number with value 0.32f.

      1. HDFS-3816.patch
        9 kB
        Jing Zhao
      2. HDFS-3816.patch
        1 kB
        Jing Zhao

        Issue Links

          Activity

          Jing Zhao created issue -
          Jing Zhao made changes -
          Field Original Value New Value
          Link This issue is related to HDFS-3475 [ HDFS-3475 ]
          Jing Zhao made changes -
          Link This issue is related to HDFS-3814 [ HDFS-3814 ]
          Jing Zhao made changes -
          Attachment HDFS-3816.patch [ 12541451 ]
          Jing Zhao made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541451/HDFS-3816.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3039//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3039//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541451/HDFS-3816.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3039//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3039//console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          Jing can you please add the tests you added in HDFS-3814 - see comment

          Show
          Suresh Srinivas added a comment - Jing can you please add the tests you added in HDFS-3814 - see comment
          Hide
          Jing Zhao added a comment -

          Added testcases for trunk.

          Show
          Jing Zhao added a comment - Added testcases for trunk.
          Jing Zhao made changes -
          Attachment HDFS-3816.patch [ 12541493 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541493/HDFS-3816.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3040//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3040//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541493/HDFS-3816.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3040//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3040//console This message is automatically generated.
          Suresh Srinivas made changes -
          Summary DFSConfigKeys.DFS_NAMENODE_INVALIDATE_WORK_PCT_PER_ITERATION_DEFAULT should be 0.32f, instead of 32 .Invalidate work percentage default value should be 0.32f, instead of 32
          Hide
          Suresh Srinivas added a comment -

          +1 for the patch.

          Show
          Suresh Srinivas added a comment - +1 for the patch.
          Suresh Srinivas made changes -
          Summary .Invalidate work percentage default value should be 0.32f, instead of 32 Invalidate work percentage default value should be 0.32f instead of 32
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2666 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2666/)
          HDFS-3816. Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2666 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2666/ ) HDFS-3816 . Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2602 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2602/)
          HDFS-3816. Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2602 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2602/ ) HDFS-3816 . Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Suresh Srinivas made changes -
          Fix Version/s 2.1.0-alpha [ 12321440 ]
          Affects Version/s 2.1.0-alpha [ 12321440 ]
          Affects Version/s 3.0.0 [ 12320356 ]
          Hide
          Suresh Srinivas added a comment -

          I committed the patch to trunk and branch-2. Thank you Jing.

          Show
          Suresh Srinivas added a comment - I committed the patch to trunk and branch-2. Thank you Jing.
          Suresh Srinivas made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Suresh Srinivas made changes -
          Component/s name-node [ 12312926 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2631 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2631/)
          HDFS-3816. Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2631 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2631/ ) HDFS-3816 . Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1139 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1139/)
          HDFS-3816. Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1139 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1139/ ) HDFS-3816 . Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1171 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1171/)
          HDFS-3816. Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1171 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1171/ ) HDFS-3816 . Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Hide
          Aaron T. Myers added a comment -

          Good find, Jing.

          One suggestion: perhaps in DFSUtil#getInvalidateWorkPctPerIteration we should also check that the configured value of this config is not greater than 1.0f? If I'm reading the code correctly, it doesn't make any sense to set this value to anything outside of the range 0.0f-1.0f. Jing, if you agree with this, mind filing another JIRA? I'd be happy to review it if you post a patch. If not, I'll take care of it.

          Also, Suresh, did you commit this to branch-2.1.0-alpha as well as branch-2? If not, I believe the fix version should be set to 2.2.0-alpha. Do you agree?

          Show
          Aaron T. Myers added a comment - Good find, Jing. One suggestion: perhaps in DFSUtil#getInvalidateWorkPctPerIteration we should also check that the configured value of this config is not greater than 1.0f? If I'm reading the code correctly, it doesn't make any sense to set this value to anything outside of the range 0.0f-1.0f. Jing, if you agree with this, mind filing another JIRA? I'd be happy to review it if you post a patch. If not, I'll take care of it. Also, Suresh, did you commit this to branch-2.1.0-alpha as well as branch-2? If not, I believe the fix version should be set to 2.2.0-alpha. Do you agree?
          Hide
          Jing Zhao added a comment -

          Thanks very much for the suggestion, Aaron. I will file another JIRA for it.

          Show
          Jing Zhao added a comment - Thanks very much for the suggestion, Aaron. I will file another JIRA for it.
          Hide
          Suresh Srinivas added a comment -

          Aaron, you are right. Update the Fix Versions field.

          Show
          Suresh Srinivas added a comment - Aaron, you are right. Update the Fix Versions field.
          Suresh Srinivas made changes -
          Fix Version/s 2.2.0-alpha [ 12322472 ]
          Fix Version/s 2.1.0-alpha [ 12321440 ]
          Hide
          Aaron T. Myers added a comment -

          Cool. Thanks guys.

          Show
          Aaron T. Myers added a comment - Cool. Thanks guys.
          Jing Zhao made changes -
          Link This issue relates to HDFS-3819 [ HDFS-3819 ]
          Arun C Murthy made changes -
          Affects Version/s 2.0.0-alpha [ 12320353 ]
          Affects Version/s 2.1.0-alpha [ 12321440 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Allen Wittenauer made changes -
          Fix Version/s 3.0.0 [ 12320356 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          2m 26s 1 Jing Zhao 18/Aug/12 00:53
          Patch Available Patch Available Resolved Resolved
          19h 6m 1 Suresh Srinivas 18/Aug/12 19:59
          Resolved Resolved Closed Closed
          53d 21h 46m 1 Arun C Murthy 11/Oct/12 17:46

            People

            • Assignee:
              Jing Zhao
              Reporter:
              Jing Zhao
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development