Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3816

Invalidate work percentage default value should be 0.32f instead of 32

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha, 3.0.0
    • Fix Version/s: 2.0.2-alpha
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HDFS-3475 used blocksInvalidateWorkPct/DFS_NAMENODE_INVALIDATE_WORK_PCT_PER_ITERATION_DEFAULT as a percentage float number. Thus DFS_NAMENODE_INVALIDATE_WORK_PCT_PER_ITERATION_DEFAULT should be defined as a float number with value 0.32f.

      1. HDFS-3816.patch
        1 kB
        Jing Zhao
      2. HDFS-3816.patch
        9 kB
        Jing Zhao

        Issue Links

          Activity

          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541451/HDFS-3816.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3039//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3039//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541451/HDFS-3816.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3039//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3039//console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          Jing can you please add the tests you added in HDFS-3814 - see comment

          Show
          Suresh Srinivas added a comment - Jing can you please add the tests you added in HDFS-3814 - see comment
          Hide
          Jing Zhao added a comment -

          Added testcases for trunk.

          Show
          Jing Zhao added a comment - Added testcases for trunk.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541493/HDFS-3816.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3040//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3040//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541493/HDFS-3816.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3040//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3040//console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          +1 for the patch.

          Show
          Suresh Srinivas added a comment - +1 for the patch.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2666 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2666/)
          HDFS-3816. Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2666 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2666/ ) HDFS-3816 . Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2602 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2602/)
          HDFS-3816. Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2602 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2602/ ) HDFS-3816 . Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Hide
          Suresh Srinivas added a comment -

          I committed the patch to trunk and branch-2. Thank you Jing.

          Show
          Suresh Srinivas added a comment - I committed the patch to trunk and branch-2. Thank you Jing.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2631 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2631/)
          HDFS-3816. Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2631 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2631/ ) HDFS-3816 . Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1139 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1139/)
          HDFS-3816. Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1139 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1139/ ) HDFS-3816 . Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1171 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1171/)
          HDFS-3816. Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1171 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1171/ ) HDFS-3816 . Invalidate work percentage default value should be 0.32f. Contributed by Jing Zhao (Revision 1374645) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374645 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
          Hide
          Aaron T. Myers added a comment -

          Good find, Jing.

          One suggestion: perhaps in DFSUtil#getInvalidateWorkPctPerIteration we should also check that the configured value of this config is not greater than 1.0f? If I'm reading the code correctly, it doesn't make any sense to set this value to anything outside of the range 0.0f-1.0f. Jing, if you agree with this, mind filing another JIRA? I'd be happy to review it if you post a patch. If not, I'll take care of it.

          Also, Suresh, did you commit this to branch-2.1.0-alpha as well as branch-2? If not, I believe the fix version should be set to 2.2.0-alpha. Do you agree?

          Show
          Aaron T. Myers added a comment - Good find, Jing. One suggestion: perhaps in DFSUtil#getInvalidateWorkPctPerIteration we should also check that the configured value of this config is not greater than 1.0f? If I'm reading the code correctly, it doesn't make any sense to set this value to anything outside of the range 0.0f-1.0f. Jing, if you agree with this, mind filing another JIRA? I'd be happy to review it if you post a patch. If not, I'll take care of it. Also, Suresh, did you commit this to branch-2.1.0-alpha as well as branch-2? If not, I believe the fix version should be set to 2.2.0-alpha. Do you agree?
          Hide
          Jing Zhao added a comment -

          Thanks very much for the suggestion, Aaron. I will file another JIRA for it.

          Show
          Jing Zhao added a comment - Thanks very much for the suggestion, Aaron. I will file another JIRA for it.
          Hide
          Suresh Srinivas added a comment -

          Aaron, you are right. Update the Fix Versions field.

          Show
          Suresh Srinivas added a comment - Aaron, you are right. Update the Fix Versions field.
          Hide
          Aaron T. Myers added a comment -

          Cool. Thanks guys.

          Show
          Aaron T. Myers added a comment - Cool. Thanks guys.

            People

            • Assignee:
              Jing Zhao
              Reporter:
              Jing Zhao
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development