Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4862

SafeModeInfo.isManual() returns true when resources are low even if it wasn't entered into manually

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 0.23.7, 2.0.4-alpha
    • Fix Version/s: 2.1.0-beta, 0.23.9
    • Component/s: None
    • Labels:
      None

      Description

      HDFS-1594 changed isManual to this

      private boolean isManual() {
        return extension == Integer.MAX_VALUE && !resourcesLow;
      }
      

      One immediate impact of this is that when resources are low, the NN will throw away all block reports from DNs. This is undesirable.

      1. HDFS-4862.patch
        6 kB
        Ravi Prakash
      2. HDFS-4862.patch
        6 kB
        Ravi Prakash
      3. HDFS-4862.branch-0.23.patch
        6 kB
        Ravi Prakash
      4. HDFS-4862.patch
        5 kB
        Ravi Prakash
      5. HDFS-4862.patch
        6 kB
        Ravi Prakash
      6. HDFS-4862.patch
        5 kB
        Ravi Prakash
      7. HDFS-4862.branch-0.23.patch
        6 kB
        Ravi Prakash
      8. HDFS-4862.branch-0.23.patch
        6 kB
        Ravi Prakash
      9. HDFS-4862.patch
        6 kB
        Ravi Prakash

        Issue Links

          Activity

          Hide
          Ravi Prakash added a comment -

          Linking. If you folks had a different vision, could you please clarify here?

          Show
          Ravi Prakash added a comment - Linking. If you folks had a different vision, could you please clarify here?
          Hide
          Kihwal Lee added a comment -

          isManual() itself seems okay. Resource monitor will automatically enter NN to safe mode on resource shortage. The problem is isInStartupSafeMode() relying on it.

          Following methods in BlockManager call isInStartupSafeMode():
          processReport(), addStoredBlockImmediate(), countLiveNodes()
          They all assume isInStartupSafeMode() to do what its name implies.

          We can perhaps add a variable to keep track and check it && safeMode.isOn() from isInStartupSafeMode().

          Show
          Kihwal Lee added a comment - isManual() itself seems okay. Resource monitor will automatically enter NN to safe mode on resource shortage. The problem is isInStartupSafeMode() relying on it. Following methods in BlockManager call isInStartupSafeMode() : processReport(), addStoredBlockImmediate(), countLiveNodes() They all assume isInStartupSafeMode() to do what its name implies. We can perhaps add a variable to keep track and check it && safeMode.isOn() from isInStartupSafeMode() .
          Hide
          Ravi Prakash added a comment -

          Hi Kihwal! There already exists a method to check for low resources (areResourcesLow()). So I don't understand why we need to club that in isManual. To me isManual clearly means that the safemode was entered into manually. Moreover I could also argue that the NN should be taken out of low-resource safemode automatically when ResourceMonitor detects adequate resources, so it may not necessarily be a manual step.
          This is a patch which IMHO corrects these behaviors. Could you please review it?

          Show
          Ravi Prakash added a comment - Hi Kihwal! There already exists a method to check for low resources (areResourcesLow()). So I don't understand why we need to club that in isManual. To me isManual clearly means that the safemode was entered into manually. Moreover I could also argue that the NN should be taken out of low-resource safemode automatically when ResourceMonitor detects adequate resources, so it may not necessarily be a manual step. This is a patch which IMHO corrects these behaviors. Could you please review it?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12585996/HDFS-4862.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestSafeMode

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4472//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4472//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12585996/HDFS-4862.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestSafeMode +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4472//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4472//console This message is automatically generated.
          Hide
          Ravi Prakash added a comment -

          Reverted changes in getTurnOffTip(). That madness needs to be fixed in another JIRA.

          Show
          Ravi Prakash added a comment - Reverted changes in getTurnOffTip(). That madness needs to be fixed in another JIRA.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12586114/HDFS-4862.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4473//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4473//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12586114/HDFS-4862.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4473//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4473//console This message is automatically generated.
          Hide
          Kihwal Lee added a comment -

          Other than the following unnecessary import, the patch looks good.

          import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_SAFEMODE_EXTENSION_KEY;

          Show
          Kihwal Lee added a comment - Other than the following unnecessary import, the patch looks good. import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_SAFEMODE_EXTENSION_KEY;
          Hide
          Ravi Prakash added a comment -

          Thanks Kihwal! This is the patch ported to 0.23

          Show
          Ravi Prakash added a comment - Thanks Kihwal! This is the patch ported to 0.23
          Hide
          Ravi Prakash added a comment -

          Thanks Kihwal! Removed the unused import from the trunk patch.

          Show
          Ravi Prakash added a comment - Thanks Kihwal! Removed the unused import from the trunk patch.
          Hide
          Ravi Prakash added a comment -

          I'll file a separate JIRA to fix getTurnOffToolTip(). This patch is for trunk

          Show
          Ravi Prakash added a comment - I'll file a separate JIRA to fix getTurnOffToolTip(). This patch is for trunk
          Hide
          Ravi Prakash added a comment -

          Sorry! Had forgot to remove the unused import

          Show
          Ravi Prakash added a comment - Sorry! Had forgot to remove the unused import
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12586521/HDFS-4862.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.qjournal.client.TestQuorumJournalManager

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4487//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4487//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12586521/HDFS-4862.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.qjournal.client.TestQuorumJournalManager +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4487//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4487//console This message is automatically generated.
          Hide
          Ravi Prakash added a comment -

          The patch for branch-0.23

          Show
          Ravi Prakash added a comment - The patch for branch-0.23
          Hide
          Ravi Prakash added a comment -

          The patch for trunk applies to branch-2 too. I've filed HDFS-4888 to fix up the getTurnOffTip() madness.

          Show
          Ravi Prakash added a comment - The patch for trunk applies to branch-2 too. I've filed HDFS-4888 to fix up the getTurnOffTip() madness.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12586547/HDFS-4862.branch-0.23.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4490//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12586547/HDFS-4862.branch-0.23.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4490//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12586543/HDFS-4862.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4489//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4489//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12586543/HDFS-4862.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4489//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4489//console This message is automatically generated.
          Hide
          Kihwal Lee added a comment -

          Please use junit assert methods in your test.

          Show
          Kihwal Lee added a comment - Please use junit assert methods in your test.
          Hide
          Ravi Prakash added a comment -

          Using junit asserts

          Show
          Ravi Prakash added a comment - Using junit asserts
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12586604/HDFS-4862.branch-0.23.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4491//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12586604/HDFS-4862.branch-0.23.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4491//console This message is automatically generated.
          Hide
          Ravi Prakash added a comment -

          The patch for trunk

          Show
          Ravi Prakash added a comment - The patch for trunk
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12586608/HDFS-4862.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4492//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4492//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12586608/HDFS-4862.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4492//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4492//console This message is automatically generated.
          Hide
          Kihwal Lee added a comment -

          +1 the new patch looks good.

          Show
          Kihwal Lee added a comment - +1 the new patch looks good.
          Hide
          Kihwal Lee added a comment -

          I've committed this to trunk, branch-2.1.0, branch-2 and branch-0.23. Thanks fot the patch, Ravi.

          Show
          Kihwal Lee added a comment - I've committed this to trunk, branch-2.1.0, branch-2 and branch-0.23. Thanks fot the patch, Ravi.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3877 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3877/)
          HDFS-4862. SafeModeInfo.isManual() returns true when resources are low even if it wasn't entered into manually. Contributed by Ravi Prakash. (Revision 1490486)

          Result = SUCCESS
          kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1490486
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3877 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3877/ ) HDFS-4862 . SafeModeInfo.isManual() returns true when resources are low even if it wasn't entered into manually. Contributed by Ravi Prakash. (Revision 1490486) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1490486 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java

            People

            • Assignee:
              Ravi Prakash
              Reporter:
              Ravi Prakash
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development