Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1594

When the disk becomes full Namenode is getting shutdown and not able to recover

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.21.1, 0.22.0
    • Fix Version/s: 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Environment:

      Linux linux124 2.6.27.19-5-default #1 SMP 2009-02-28 04:40:21 +0100 x86_64 x86_64 x86_64 GNU/Linux

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Implemented a daemon thread to monitor the disk usage for periodically and if the disk usage reaches the threshold value, put the name node into Safe mode so that no modification to file system will occur. Once the disk usage reaches below the threshold, name node will be put out of the safe mode. Here threshold value and interval to check the disk usage are configurable.
      Show
      Implemented a daemon thread to monitor the disk usage for periodically and if the disk usage reaches the threshold value, put the name node into Safe mode so that no modification to file system will occur. Once the disk usage reaches below the threshold, name node will be put out of the safe mode. Here threshold value and interval to check the disk usage are configurable.

      Description

      When the disk becomes full name node is shutting down and if we try to start after making the space available It is not starting and throwing the below exception.

       
      
      2011-01-24 23:23:33,727 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
      java.io.EOFException
      	at java.io.DataInputStream.readFully(DataInputStream.java:180)
      	at org.apache.hadoop.io.UTF8.readFields(UTF8.java:117)
      	at org.apache.hadoop.hdfs.server.namenode.FSImageSerialization.readString(FSImageSerialization.java:201)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:185)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:93)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:60)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1089)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:1041)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:487)
      	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:149)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:306)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:284)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:328)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:356)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:577)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:570)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1529)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1538)
      2011-01-24 23:23:33,729 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.EOFException
      	at java.io.DataInputStream.readFully(DataInputStream.java:180)
      	at org.apache.hadoop.io.UTF8.readFields(UTF8.java:117)
      	at org.apache.hadoop.hdfs.server.namenode.FSImageSerialization.readString(FSImageSerialization.java:201)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:185)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:93)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:60)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1089)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:1041)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:487)
      	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:149)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:306)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:284)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:328)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:356)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:577)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:570)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1529)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1538)
      
      2011-01-24 23:23:33,730 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: 
      /************************************************************
      SHUTDOWN_MSG: Shutting down NameNode at linux124/10.18.52.124
      ************************************************************/
      
      
      
      1. hadoop-root-namenode-linux124.log
        36 kB
        Devaraj K
      2. hdfs-1594.0.patch
        0.8 kB
        Aaron T. Myers
      3. hdfs-1594.1.patch
        19 kB
        Aaron T. Myers
      4. hdfs-1594.2.patch
        19 kB
        Aaron T. Myers
      5. hdfs-1594.3.patch
        23 kB
        Aaron T. Myers
      6. hdfs-1594.4.patch
        23 kB
        Aaron T. Myers
      7. hdfs-1594.5.patch
        24 kB
        Aaron T. Myers
      8. hdfs-1594.6.patch
        24 kB
        Aaron T. Myers
      9. HDFS-1594.patch
        18 kB
        Konstantin Boudnik
      10. HDFS-1594.patch
        17 kB
        Konstantin Boudnik
      11. HDFS-1594.patch
        103 kB
        Devaraj K

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #650 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/650/)
          HDFS-1862. Improve test reliability of HDFS-1594. Contributed by Aaron T. Myers

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #650 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/650/ ) HDFS-1862 . Improve test reliability of HDFS-1594 . Contributed by Aaron T. Myers
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #610 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/610/)
          HDFS-1862. Improve test reliability of HDFS-1594. Contributed by Aaron T. Myers

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #610 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/610/ ) HDFS-1862 . Improve test reliability of HDFS-1594 . Contributed by Aaron T. Myers
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #603 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/603/)
          HDFS-1594. When the disk becomes full Namenode is getting shutdown and not able to recover. Contributed by Aaron T. Myers

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #603 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/603/ ) HDFS-1594 . When the disk becomes full Namenode is getting shutdown and not able to recover. Contributed by Aaron T. Myers
          Hide
          Eli Collins added a comment -

          I've committed this. Thanks atm!

          Show
          Eli Collins added a comment - I've committed this. Thanks atm!
          Hide
          Eli Collins added a comment -

          +1 latest patch lgtm

          Show
          Eli Collins added a comment - +1 latest patch lgtm
          Hide
          Aaron T. Myers added a comment -

          I'm pretty sure those test failures are unrelated to this patch. The javadoc warning appears to have been transient:

          [javadoc] javadoc: warning - Error fetching URL: http://java.sun.com/javase/6/docs/api/package-list
          
          Show
          Aaron T. Myers added a comment - I'm pretty sure those test failures are unrelated to this patch. The javadoc warning appears to have been transient: [javadoc] javadoc: warning - Error fetching URL: http: //java.sun.com/javase/6/docs/api/ package -list
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12477093/hdfs-1594.6.patch
          against trunk revision 1095830.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.TestFileAppend4
          org.apache.hadoop.hdfs.TestLargeBlock
          org.apache.hadoop.hdfs.TestWriteConfigurationToDFS

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/409//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/409//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/409//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12477093/hdfs-1594.6.patch against trunk revision 1095830. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestFileAppend4 org.apache.hadoop.hdfs.TestLargeBlock org.apache.hadoop.hdfs.TestWriteConfigurationToDFS +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/409//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/409//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/409//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Updated patch which makes the test more robust by:

          1. Making the threshold/test buffer larger.
          2. Making sure the test tmp file is uniquely named.
          3. Deleting the test tmp file on test completion.
          4. Waiting up to a minute (with a check every second) to give the NNRM thread time to run.

          Hopefully this will get it to pass on Hudson. If not, I'll probably figure out a way to mock some of this out to be more reliable.

          Show
          Aaron T. Myers added a comment - Updated patch which makes the test more robust by: Making the threshold/test buffer larger. Making sure the test tmp file is uniquely named. Deleting the test tmp file on test completion. Waiting up to a minute (with a check every second) to give the NNRM thread time to run. Hopefully this will get it to pass on Hudson. If not, I'll probably figure out a way to mock some of this out to be more reliable.
          Hide
          Eli Collins added a comment -

          The TestNameNodeResourceChecker check passes for me locally but looks like it failed on Hudson:

          junit.framework.AssertionFailedError: NN should be in safe mode after resources crossed threshold
          	at org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker.testCheckThatNameNodeResourceMonitorIsRunning(TestNameNodeResourceChecker.java:132)
          
          Show
          Eli Collins added a comment - The TestNameNodeResourceChecker check passes for me locally but looks like it failed on Hudson: junit.framework.AssertionFailedError: NN should be in safe mode after resources crossed threshold at org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker.testCheckThatNameNodeResourceMonitorIsRunning(TestNameNodeResourceChecker.java:132)
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12477054/hdfs-1594.5.patch
          against trunk revision 1095830.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/403//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/403//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/403//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12477054/hdfs-1594.5.patch against trunk revision 1095830. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/403//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/403//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/403//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Thanks a lot for pointer to HDFS-1566, Cos. Like that JIRA, I propose we punt on adding that test for later.

          Show
          Aaron T. Myers added a comment - Thanks a lot for pointer to HDFS-1566 , Cos. Like that JIRA, I propose we punt on adding that test for later.
          Hide
          Eli Collins added a comment -

          +1

          Latest patch looks great.

          Show
          Eli Collins added a comment - +1 Latest patch looks great.
          Hide
          Aaron T. Myers added a comment -

          In the mean time note that the TestStartup and TestEditLog failures are due to the patch (the NN is going into SM, looks like the threshold is getting crossed).

          The tests were failing because of a bug, but not quite that one. The previous implementation waited for the NN resource monitoring thread to run before checking for available resources. In this patch, I've changed things around slightly so that an initial resource check is done at FSNamesystem initialization time.

          Could you fold the call to safeMode.setResourcesLow() on line 2891 into enterSafeMode? Ie I think enterSafeMode(true) should always result in a call to setResourcesLow.

          Good point. Done.

          Perhaps include "resouce" in the "dfs.nn.du.reserved" and "dfs.nn.checked.volumes" key names so it's clear there's a relationship between them and "dfs.nn.resource.check.interval".

          Changed to "dfs.namenode.resource.du.reserved" and "dfs.namenode.resource.checked.volumes".

          In the NameNodeResourceChecker class header comment what do you mean by "heap space available on all volumes"?

          Good catch. This was a relic from the original version of the patch posted by Devaraj. I removed this functionality, but missed the comment. Fixed.

          Would it be hard to write a test that crosses the threshold, eg set the limit based on current available space minus say 500KB then create a large file and assert the NN went into SM?

          Nope. Done.

          Nits:

          FSNameSystem line 570: missing space after "if", and extra space after "interrupt". line 4058 has extra parens.
          isResourcesLow -> areResourcesLow or just resourcesLow
          NameNodeResourceChecker line 118, <= should technically be <, or the message should say something like the available space has reached the reserved amount.
          In the test extra newline on line 68 ("in case of errors")

          Fixed.

          Show
          Aaron T. Myers added a comment - In the mean time note that the TestStartup and TestEditLog failures are due to the patch (the NN is going into SM, looks like the threshold is getting crossed). The tests were failing because of a bug, but not quite that one. The previous implementation waited for the NN resource monitoring thread to run before checking for available resources. In this patch, I've changed things around slightly so that an initial resource check is done at FSNamesystem initialization time. Could you fold the call to safeMode.setResourcesLow() on line 2891 into enterSafeMode? Ie I think enterSafeMode(true) should always result in a call to setResourcesLow. Good point. Done. Perhaps include "resouce" in the "dfs.nn.du.reserved" and "dfs.nn.checked.volumes" key names so it's clear there's a relationship between them and "dfs.nn.resource.check.interval". Changed to " dfs.namenode.resource.du.reserved " and " dfs.namenode.resource.checked.volumes ". In the NameNodeResourceChecker class header comment what do you mean by "heap space available on all volumes"? Good catch. This was a relic from the original version of the patch posted by Devaraj. I removed this functionality, but missed the comment. Fixed. Would it be hard to write a test that crosses the threshold, eg set the limit based on current available space minus say 500KB then create a large file and assert the NN went into SM? Nope. Done. Nits: FSNameSystem line 570: missing space after "if", and extra space after "interrupt". line 4058 has extra parens. isResourcesLow -> areResourcesLow or just resourcesLow NameNodeResourceChecker line 118, <= should technically be <, or the message should say something like the available space has reached the reserved amount. In the test extra newline on line 68 ("in case of errors") Fixed.
          Hide
          Eli Collins added a comment -

          Looks great. Minor comments follow.

          Could you fold the call to safeMode.setResourcesLow() on line 2891 into enterSafeMode? Ie I think enterSafeMode(true) should always result in a call to setResourcesLow.

          Perhaps include "resouce" in the "dfs.nn.du.reserved" and "dfs.nn.checked.volumes" key names so it's clear there's a relationship between them and "dfs.nn.resource.check.interval".

          In the NameNodeResourceChecker class header comment what do you mean by "heap space available on all volumes"?

          Would it be hard to write a test that crosses the threshold, eg set the limit based on current available space minus say 500KB then create a large file and assert the NN went into SM?

          Nits:

          • FSNameSystem line 570: missing space after "if", and extra space after "interrupt". line 4058 has extra parens.
          • isResourcesLow -> areResourcesLow or just resourcesLow
          • NameNodeResourceChecker line 118, <= should technically be <, or the message should say something like the available space has reached the reserved amount.
          • In the test extra newline on line 68 ("in case of errors")
          Show
          Eli Collins added a comment - Looks great. Minor comments follow. Could you fold the call to safeMode.setResourcesLow() on line 2891 into enterSafeMode? Ie I think enterSafeMode(true) should always result in a call to setResourcesLow. Perhaps include "resouce" in the "dfs.nn.du.reserved" and "dfs.nn.checked.volumes" key names so it's clear there's a relationship between them and "dfs.nn.resource.check.interval". In the NameNodeResourceChecker class header comment what do you mean by "heap space available on all volumes"? Would it be hard to write a test that crosses the threshold, eg set the limit based on current available space minus say 500KB then create a large file and assert the NN went into SM? Nits: FSNameSystem line 570: missing space after "if", and extra space after "interrupt". line 4058 has extra parens. isResourcesLow -> areResourcesLow or just resourcesLow NameNodeResourceChecker line 118, <= should technically be <, or the message should say something like the available space has reached the reserved amount. In the test extra newline on line 68 ("in case of errors")
          Hide
          Konstantin Boudnik added a comment -

          Aaron, for the test case you can check HDFS-1566, perhaps

          Show
          Konstantin Boudnik added a comment - Aaron, for the test case you can check HDFS-1566 , perhaps
          Hide
          Eli Collins added a comment -

          I disagree. I think losing one of the (supposedly redundant) volumes is sufficient cause for alarm as to warrant the whole thing being put into SM.

          The NN can tolerate the failure of an edit log volume and continue operation, why is running out of disk space warrant a differetn outcome? You could make the case that running out of disk space is a correlated failure (is likely to affect the other disks equally if they have similar capacities), but it may not be (eg the network mount may run out early or have more free space) in which case we'd be less available than we otherwise could me. However I think what you have make sense for now. Afterall an admin can always prevent the NN from entering SM by monitoring and ensuring all volumes have sufficient free space.

          I'm in favor of the current implementation - a single configurable threshold, which applies to each volume separately.

          That sounds good to me. Since the threshold is a factor of how much slack the NN needs (ie is independent of the volume size) a single value enforeced across all volumes makes sense.

          I'll check out your latest patch. In the mean time note that the TestStartup and TestEditLog failures are due to the patch (the NN is going into SM, looks like the threshold is getting crossed).

          Show
          Eli Collins added a comment - I disagree. I think losing one of the (supposedly redundant) volumes is sufficient cause for alarm as to warrant the whole thing being put into SM. The NN can tolerate the failure of an edit log volume and continue operation, why is running out of disk space warrant a differetn outcome? You could make the case that running out of disk space is a correlated failure (is likely to affect the other disks equally if they have similar capacities), but it may not be (eg the network mount may run out early or have more free space) in which case we'd be less available than we otherwise could me. However I think what you have make sense for now. Afterall an admin can always prevent the NN from entering SM by monitoring and ensuring all volumes have sufficient free space. I'm in favor of the current implementation - a single configurable threshold, which applies to each volume separately. That sounds good to me. Since the threshold is a factor of how much slack the NN needs (ie is independent of the volume size) a single value enforeced across all volumes makes sense. I'll check out your latest patch. In the mean time note that the TestStartup and TestEditLog failures are due to the patch (the NN is going into SM, looks like the threshold is getting crossed).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12476965/hdfs-1594.4.patch
          against trunk revision 1095789.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.server.namenode.TestEditLog
          org.apache.hadoop.hdfs.server.namenode.TestStartup
          org.apache.hadoop.hdfs.TestAbandonBlock

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/398//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/398//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/398//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12476965/hdfs-1594.4.patch against trunk revision 1095789. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.namenode.TestEditLog org.apache.hadoop.hdfs.server.namenode.TestStartup org.apache.hadoop.hdfs.TestAbandonBlock +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/398//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/398//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/398//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Updated patch to fix up the tests a little bit.

          Eli, I couldn't figure out a good way to write a test to test the interaction when creating a checkpoint itself would cause the NN to go above threshold on the namedir volume. Everything I came up with would almost certainly have resulted in a flaky test, one way or another. Unless some one can suggest a good strategy for doing it, I propose we break that out into a separate JIRA.

          Show
          Aaron T. Myers added a comment - Updated patch to fix up the tests a little bit. Eli, I couldn't figure out a good way to write a test to test the interaction when creating a checkpoint itself would cause the NN to go above threshold on the namedir volume. Everything I came up with would almost certainly have resulted in a flaky test, one way or another. Unless some one can suggest a good strategy for doing it, I propose we break that out into a separate JIRA.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12476669/hdfs-1594.3.patch
          against trunk revision 1094748.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.server.namenode.TestEditLog
          org.apache.hadoop.hdfs.server.namenode.TestStartup

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/383//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/383//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/383//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12476669/hdfs-1594.3.patch against trunk revision 1094748. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.namenode.TestEditLog org.apache.hadoop.hdfs.server.namenode.TestStartup -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/383//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/383//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/383//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          I agree, should be pulled out to a separate jira.

          Removed.

          Good idea, there's nothing edits specific here. Would need to add a test that if the admin does pass in the volume that hosts the edits log it doesn't conflict with the default behavior (eg double monitoring).

          Done. I used a HashMap indexed by volume, and added tests to make sure we only check a single volume at most once.

          What's the intended behavior if there are n disks and one fills up before the others? Seems like this volume should be taken off-line and the NN does not enter SM.

          I disagree. I think losing one of the (supposedly redundant) volumes is sufficient cause for alarm as to warrant the whole thing being put into SM.

          If there's just a global threshold would this cause the overall threshold to drop (because the removed volume's free space not longer counts towards the total), causing a cascade where the other volumes go off-line? This would suggest a threshold per volume. Though if we can make a single, simple threshold work that seems better from a usability perspective.

          I should have been more clear. The current implementation is indeed that there is a threshold per volume, it's just the same for all volumes. I was not trying to distinguish between a single total threshold vs. per-volume thresholds. Rather, the question I was trying to ask is "should the user be able to specify distinct thresholds per volume? e.g. 100MB on /disk/1 and 1GB on /disk/2".

          I'm in favor of the current implementation - a single configurable threshold, which applies to each volume separately.

          In both cases I think the admin would want to have to manually tell the NN to leave SM while they are working (eg w/o them explicitly telling it to do so). If they want automatic behavior they can continuosly monitor/roll on these volumes so they don't get into this scenario, and they don't want the monitoring/rolling to race with the free space detection (eg they'd want to have to take action if this process ever crosses the threshold they set). Ie seems like once you've gone into SM due to lack of free space you should stay there until the admin has had a chance to rectify.

          Agreed. Done.

          Another test to add: the interaction between this detection and the CN check-pointing.

          The current patch does not contain such a test. I'm thinking about the best way to implement this.

          Show
          Aaron T. Myers added a comment - I agree, should be pulled out to a separate jira. Removed. Good idea, there's nothing edits specific here. Would need to add a test that if the admin does pass in the volume that hosts the edits log it doesn't conflict with the default behavior (eg double monitoring). Done. I used a HashMap indexed by volume, and added tests to make sure we only check a single volume at most once. What's the intended behavior if there are n disks and one fills up before the others? Seems like this volume should be taken off-line and the NN does not enter SM. I disagree. I think losing one of the (supposedly redundant) volumes is sufficient cause for alarm as to warrant the whole thing being put into SM. If there's just a global threshold would this cause the overall threshold to drop (because the removed volume's free space not longer counts towards the total), causing a cascade where the other volumes go off-line? This would suggest a threshold per volume. Though if we can make a single, simple threshold work that seems better from a usability perspective. I should have been more clear. The current implementation is indeed that there is a threshold per volume, it's just the same for all volumes. I was not trying to distinguish between a single total threshold vs. per-volume thresholds. Rather, the question I was trying to ask is "should the user be able to specify distinct thresholds per volume? e.g. 100MB on /disk/1 and 1GB on /disk/2". I'm in favor of the current implementation - a single configurable threshold, which applies to each volume separately. In both cases I think the admin would want to have to manually tell the NN to leave SM while they are working (eg w/o them explicitly telling it to do so). If they want automatic behavior they can continuosly monitor/roll on these volumes so they don't get into this scenario, and they don't want the monitoring/rolling to race with the free space detection (eg they'd want to have to take action if this process ever crosses the threshold they set). Ie seems like once you've gone into SM due to lack of free space you should stay there until the admin has had a chance to rectify. Agreed. Done. Another test to add: the interaction between this detection and the CN check-pointing. The current patch does not contain such a test. I'm thinking about the best way to implement this.
          Hide
          Aaron T. Myers added a comment -

          Updated patch to address most of Eli's comments. Detailed responses forthcoming.

          Show
          Aaron T. Myers added a comment - Updated patch to address most of Eli's comments. Detailed responses forthcoming.
          Hide
          Eli Collins added a comment -

          Another test to add: the interaction between this detection and the CN check-pointing.

          Show
          Eli Collins added a comment - Another test to add: the interaction between this detection and the CN check-pointing.
          Hide
          Eli Collins added a comment -

          1. The original JIRA description indicated that this problem was caused only by the disk filling up, yet the original patches also monitor for a near-full JVM heap. I left the memory checking in this reworked patch, but I think this feature should probably be removed. Unclear how this would interact with Java GC, and unclear if entering safemode would actually help the situation.

          I agree, should be pulled out to a separate jira.

          3. Todd mentioned that he's seen edit log corruptions from the log volume filling up. Perhaps we should add an additional configuration option to let the user specify arbitrary volumes to check, besides just the volumes containing the edits/name dirs?

          Good idea, there's nothing edits specific here. Would need to add a test that if the admin does pass in the volume that hosts the edits log it doesn't conflict with the default behavior (eg double monitoring).

          3. I switched the configuration of disk space amount from a percentage to a number of bytes remaining, since volume sizes may differ, and thus a fixed amount of space reserved seems more appropriate. Perhaps there should be a way to specify the threshold per-volume?

          What's the intended behavior if there are n disks and one fills up before the others? Seems like this volume should be taken off-line and the NN does not enter SM. If there's just a global threshold would this cause the overall threshold to drop (because the removed volume's free space not longer counts towards the total), causing a cascade where the other volumes go off-line? This would suggest a threshold per volume. Though if we can make a single, simple threshold work that seems better from a usability perspective.

          4. I'm a little concerned that we might see a problem where the NN will reach the threshold and then thrash in and out of safemode as it sits on the cusp of the configured free space. Perhaps we should not automatically leave safemode in the event the resources later return to normal? Or make this behavior configurable? It seems to me that an NN volume running out of space should be a cause for concern, so it might be reasonable for an admin to have to manually force the NN out of safe mode.

          Seems there are two scenarios here:
          a. The admin can easily free up some space
          b. The admin can not easily free up space (eg needs to compact the log because the 2NN wasn't running for a while, need to resize the volume, replace the disk etc).

          In both cases I think the admin would want to have to manually tell the NN to leave SM while they are working (eg w/o them explicitly telling it to do so). If they want automatic behavior they can continuosly monitor/roll on these volumes so they don't get into this scenario, and they don't want the monitoring/rolling to race with the free space detection (eg they'd want to have to take action if this process ever crosses the threshold they set). Ie seems like once you've gone into SM due to lack of free space you should stay there until the admin has had a chance to rectify.

          Show
          Eli Collins added a comment - 1. The original JIRA description indicated that this problem was caused only by the disk filling up, yet the original patches also monitor for a near-full JVM heap. I left the memory checking in this reworked patch, but I think this feature should probably be removed. Unclear how this would interact with Java GC, and unclear if entering safemode would actually help the situation. I agree, should be pulled out to a separate jira. 3. Todd mentioned that he's seen edit log corruptions from the log volume filling up. Perhaps we should add an additional configuration option to let the user specify arbitrary volumes to check, besides just the volumes containing the edits/name dirs? Good idea, there's nothing edits specific here. Would need to add a test that if the admin does pass in the volume that hosts the edits log it doesn't conflict with the default behavior (eg double monitoring). 3. I switched the configuration of disk space amount from a percentage to a number of bytes remaining, since volume sizes may differ, and thus a fixed amount of space reserved seems more appropriate. Perhaps there should be a way to specify the threshold per-volume? What's the intended behavior if there are n disks and one fills up before the others? Seems like this volume should be taken off-line and the NN does not enter SM. If there's just a global threshold would this cause the overall threshold to drop (because the removed volume's free space not longer counts towards the total), causing a cascade where the other volumes go off-line? This would suggest a threshold per volume. Though if we can make a single, simple threshold work that seems better from a usability perspective. 4. I'm a little concerned that we might see a problem where the NN will reach the threshold and then thrash in and out of safemode as it sits on the cusp of the configured free space. Perhaps we should not automatically leave safemode in the event the resources later return to normal? Or make this behavior configurable? It seems to me that an NN volume running out of space should be a cause for concern, so it might be reasonable for an admin to have to manually force the NN out of safe mode. Seems there are two scenarios here: a. The admin can easily free up some space b. The admin can not easily free up space (eg needs to compact the log because the 2NN wasn't running for a while, need to resize the volume, replace the disk etc). In both cases I think the admin would want to have to manually tell the NN to leave SM while they are working (eg w/o them explicitly telling it to do so). If they want automatic behavior they can continuosly monitor/roll on these volumes so they don't get into this scenario, and they don't want the monitoring/rolling to race with the free space detection (eg they'd want to have to take action if this process ever crosses the threshold they set). Ie seems like once you've gone into SM due to lack of free space you should stay there until the admin has had a chance to rectify.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12476151/hdfs-1594.2.patch
          against trunk revision 1091515.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.server.datanode.TestBlockReport
          org.apache.hadoop.hdfs.TestFileAppend4
          org.apache.hadoop.hdfs.TestFileConcurrentReader
          org.apache.hadoop.hdfs.TestLargeBlock
          org.apache.hadoop.hdfs.TestWriteConfigurationToDFS

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/351//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/351//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/351//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12476151/hdfs-1594.2.patch against trunk revision 1091515. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.datanode.TestBlockReport org.apache.hadoop.hdfs.TestFileAppend4 org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.hdfs.TestLargeBlock org.apache.hadoop.hdfs.TestWriteConfigurationToDFS -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/351//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/351//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/351//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Thanks for the review, Cos. Not sure how I failed to notice that.

          Updated patch attached.

          Show
          Aaron T. Myers added a comment - Thanks for the review, Cos. Not sure how I failed to notice that. Updated patch attached.
          Hide
          Konstantin Boudnik added a comment -

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          There's also new static analysis warning

          Show
          Konstantin Boudnik added a comment - -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. There's also new static analysis warning
          Hide
          Aaron T. Myers added a comment -

          The failing tests are unrelated to this patch.

          Show
          Aaron T. Myers added a comment - The failing tests are unrelated to this patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12476093/hdfs-1594.1.patch
          against trunk revision 1091131.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.TestFileConcurrentReader

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/349//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/349//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/349//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12476093/hdfs-1594.1.patch against trunk revision 1091131. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/349//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/349//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/349//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          The previous patch was generated in error. Updated patch attached.

          Show
          Aaron T. Myers added a comment - The previous patch was generated in error. Updated patch attached.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12476078/hdfs-1594.0.patch
          against trunk revision 1091131.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/348//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12476078/hdfs-1594.0.patch against trunk revision 1091131. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/348//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Modified the patch posted on 2/16 in the following ways:

          • Changed disk amount configuration from "percentage used" to "bytes remaining".
          • Separated configuration of minimum disk space reserved from configuration of minimum heap remaining.
          • Renamed NameNodeResourceBean to NameNodeResourceChecker.
          • Added some log output.
          • Some style clean up.
          • Added more comments.

          I have a few open questions which I'd love to get some feedback on:

          1. The original JIRA description indicated that this problem was caused only by the disk filling up, yet the original patches also monitor for a near-full JVM heap. I left the memory checking in this reworked patch, but I think this feature should probably be removed. Unclear how this would interact with Java GC, and unclear if entering safemode would actually help the situation.
          2. Todd mentioned that he's seen edit log corruptions from the log volume filling up. Perhaps we should add an additional configuration option to let the user specify arbitrary volumes to check, besides just the volumes containing the edits/name dirs?
          3. I switched the configuration of disk space amount from a percentage to a number of bytes remaining, since volume sizes may differ, and thus a fixed amount of space reserved seems more appropriate. Perhaps there should be a way to specify the threshold per-volume?
          4. I'm a little concerned that we might see a problem where the NN will reach the threshold and then thrash in and out of safemode as it sits on the cusp of the configured free space. Perhaps we should not automatically leave safemode in the event the resources later return to normal? Or make this behavior configurable? It seems to me that an NN volume running out of space should be a cause for concern, so it might be reasonable for an admin to have to manually force the NN out of safe mode.

          I should also mention that I did some manual testing of this patch by setting the reserve amount to a little less than the amount of space free on my hard drive, and then creating a large file from /dev/urandom to observe the NN entering/leaving safemode as the threshold was reached.

          Show
          Aaron T. Myers added a comment - Modified the patch posted on 2/16 in the following ways: Changed disk amount configuration from "percentage used" to "bytes remaining". Separated configuration of minimum disk space reserved from configuration of minimum heap remaining. Renamed NameNodeResourceBean to NameNodeResourceChecker. Added some log output. Some style clean up. Added more comments. I have a few open questions which I'd love to get some feedback on: The original JIRA description indicated that this problem was caused only by the disk filling up, yet the original patches also monitor for a near-full JVM heap. I left the memory checking in this reworked patch, but I think this feature should probably be removed. Unclear how this would interact with Java GC, and unclear if entering safemode would actually help the situation. Todd mentioned that he's seen edit log corruptions from the log volume filling up. Perhaps we should add an additional configuration option to let the user specify arbitrary volumes to check, besides just the volumes containing the edits/name dirs? I switched the configuration of disk space amount from a percentage to a number of bytes remaining, since volume sizes may differ, and thus a fixed amount of space reserved seems more appropriate. Perhaps there should be a way to specify the threshold per-volume? I'm a little concerned that we might see a problem where the NN will reach the threshold and then thrash in and out of safemode as it sits on the cusp of the configured free space. Perhaps we should not automatically leave safemode in the event the resources later return to normal? Or make this behavior configurable? It seems to me that an NN volume running out of space should be a cause for concern, so it might be reasonable for an admin to have to manually force the NN out of safe mode. I should also mention that I did some manual testing of this patch by setting the reserve amount to a little less than the amount of space free on my hard drive, and then creating a large file from /dev/urandom to observe the NN entering/leaving safemode as the threshold was reached.
          Hide
          dhruba borthakur added a comment -

          @Devaraj: ur proposal to make the NN go into safemode if the fsedits partition is almost full sounds good.

          @Todd: thanks for the info. If there is a possibility of fsedit corruption when the disk is full, (irrespective of the patch suggested in this JIRA), then we need to fix it. This patch tries to avoid this situation but is not foolproof. If the NN can (somehow) know that this is the last partial transaction, then is can safely ignore it. maybe, when we pre-allocate the editslog, we should fill it up with a specific pattern so that it is easy to detect if the partial transaction is the last one?

          Show
          dhruba borthakur added a comment - @Devaraj: ur proposal to make the NN go into safemode if the fsedits partition is almost full sounds good. @Todd: thanks for the info. If there is a possibility of fsedit corruption when the disk is full, (irrespective of the patch suggested in this JIRA), then we need to fix it. This patch tries to avoid this situation but is not foolproof. If the NN can (somehow) know that this is the last partial transaction, then is can safely ignore it. maybe, when we pre-allocate the editslog, we should fill it up with a specific pattern so that it is easy to detect if the partial transaction is the last one?
          Hide
          Todd Lipcon added a comment -

          Dhruba: I haven't dug into it yet, but I've also seen this before when the drive hosting the log4j logs fills up. I think there are some cases in which we do a System.exit(1) which can corrupt the logs – the issue being that, even though we fsync after every edit, there's no guarantee that we don't get half of the preceding edit. Hairong's proposal to checksum each edit should help here.

          Show
          Todd Lipcon added a comment - Dhruba: I haven't dug into it yet, but I've also seen this before when the drive hosting the log4j logs fills up. I think there are some cases in which we do a System.exit(1) which can corrupt the logs – the issue being that, even though we fsync after every edit, there's no guarantee that we don't get half of the preceding edit. Hairong's proposal to checksum each edit should help here.
          Hide
          Devaraj K added a comment -

          During write operation, if the name node encounters disk full condition for updating the fsimage it is abruptly writing till disk space available and immediately name node is shutting down.

          When we restart name node, it is trying to read the fsimage for initialization of the name system and it is giving the EOFException as shown in the description because the fsimage is not having the expected data. Please find the exception stack trace in the description for the exact failure point.

          Show
          Devaraj K added a comment - During write operation, if the name node encounters disk full condition for updating the fsimage it is abruptly writing till disk space available and immediately name node is shutting down. When we restart name node, it is trying to read the fsimage for initialization of the name system and it is giving the EOFException as shown in the description because the fsimage is not having the expected data. Please find the exception stack trace in the description for the exact failure point.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12471142/HDFS-1594.patch
          against trunk revision 1080836.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.TestFileConcurrentReader

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/255//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/255//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/255//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12471142/HDFS-1594.patch against trunk revision 1080836. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/255//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/255//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/255//console This message is automatically generated.
          Hide
          dhruba borthakur added a comment -

          another question related to this one. Can we find the root cause why the edits log get corrupted when disk device is full? As far as the code is concerned, the namenode will fail to preallocate the transaction log and should exit (expected behaviour). One reason why it can refuse to restart is because the nn tries to take a new checkpoint at startup time, can you pl say whether this was what happened in your situation?

          new method: numLiveDataNodes()?

          Show
          dhruba borthakur added a comment - another question related to this one. Can we find the root cause why the edits log get corrupted when disk device is full? As far as the code is concerned, the namenode will fail to preallocate the transaction log and should exit (expected behaviour). One reason why it can refuse to restart is because the nn tries to take a new checkpoint at startup time, can you pl say whether this was what happened in your situation? new method: numLiveDataNodes()?
          Hide
          Konstantin Boudnik added a comment -

          If HDFS-1726 gets committed before this one does, this change should also make sure that the safemode status field gets set to SAFEMODE_AUTOMATIC.

          Show
          Konstantin Boudnik added a comment - If HDFS-1726 gets committed before this one does, this change should also make sure that the safemode status field gets set to SAFEMODE_AUTOMATIC.
          Hide
          Aaron T. Myers added a comment -

          If HDFS-1726 gets committed before this one does, this change should also make sure that the safemode status field gets set to SAFEMODE_AUTOMATIC.

          Show
          Aaron T. Myers added a comment - If HDFS-1726 gets committed before this one does, this change should also make sure that the safemode status field gets set to SAFEMODE_AUTOMATIC.
          Hide
          Devaraj K added a comment -

          Thanks Konstantin, changes are good.

          Show
          Devaraj K added a comment - Thanks Konstantin, changes are good.
          Hide
          Konstantin Boudnik added a comment -

          Now with a negative test. Anyone care to review?

          Show
          Konstantin Boudnik added a comment - Now with a negative test. Anyone care to review?
          Hide
          Konstantin Boudnik added a comment -

          I feel like the feature is nice to have and the patch OP's is apparently too hard to follow. So I have cleaned it and changed as per my own comments.

          I will try to have another one later today with a negative test of some kind.

          Show
          Konstantin Boudnik added a comment - I feel like the feature is nice to have and the patch OP's is apparently too hard to follow. So I have cleaned it and changed as per my own comments. I will try to have another one later today with a negative test of some kind.
          Hide
          Devaraj K added a comment -

          The submitted patch was prepared for 0.22.0 branch and some unnecessary spaces have introduced in the patch file which are causing difficulty for review. I will resubmit the patch for trunk and by fixing all the comments given above.

          Show
          Devaraj K added a comment - The submitted patch was prepared for 0.22.0 branch and some unnecessary spaces have introduced in the patch file which are causing difficulty for review. I will resubmit the patch for trunk and by fixing all the comments given above.
          Hide
          Konstantin Boudnik added a comment -

          bq . I am finding it hard to review this diff because it has lots of diffs that are not inherently connected with the attempted fix.
          That was exactly the point of asking for another round-trip

          Show
          Konstantin Boudnik added a comment - bq . I am finding it hard to review this diff because it has lots of diffs that are not inherently connected with the attempted fix. That was exactly the point of asking for another round-trip
          Hide
          dhruba borthakur added a comment -

          I am finding it hard to review this diff because it has lots of diffs that are not inherently connected with the attempted fix.

          Show
          dhruba borthakur added a comment - I am finding it hard to review this diff because it has lots of diffs that are not inherently connected with the attempted fix.
          Hide
          Devaraj K added a comment -

          Thanks Konstantin. I will do all the changes and test with the trunk and I will submit the patch.

          Show
          Devaraj K added a comment - Thanks Konstantin. I will do all the changes and test with the trunk and I will submit the patch.
          Hide
          Konstantin Boudnik added a comment -

          Coupla nits in your patch:

          • + resourceRecheckInterval = conf.getLong(DFSConfigKeys.DFS_NAMENODE_RESOURCE_CHECK_INTERVAL, 5000);
          • pl. introduce a contant for default threshold rather than declaring it explicitly in the code
            add a default value's constant instead of explicit int here
          • TestNameNodeResourceBean is written for JUnit v3 with some JUnit v4 elements (such as @Test). Please make it JUnit v4 ony
          • in the test all setups and testdowns should be done in approapriate @Before and @After methods
          • to compliment testResourceAvailabilityForALowerDiskUsageThanThreshold it'd be nice to have a negative test. Do you think it is possible?

          The patch introduces a number of white-space changes such as
          <noformat>

          • +
            <noformat>
            or
            <noformat>

          • * <ul>
            + * <ul>
            <noformat>
            or
            <noformat>
          • public LeaseManager leaseManager = new LeaseManager(this);
            + public LeaseManager leaseManager = new LeaseManager(this);
            <noformat>
            and so on...
            Please clean them up and resubmit because it is kinda hard to follow right now.
          Show
          Konstantin Boudnik added a comment - Coupla nits in your patch: + resourceRecheckInterval = conf.getLong(DFSConfigKeys.DFS_NAMENODE_RESOURCE_CHECK_INTERVAL, 5000); pl. introduce a contant for default threshold rather than declaring it explicitly in the code add a default value's constant instead of explicit int here TestNameNodeResourceBean is written for JUnit v3 with some JUnit v4 elements (such as @Test). Please make it JUnit v4 ony in the test all setups and testdowns should be done in approapriate @Before and @After methods to compliment testResourceAvailabilityForALowerDiskUsageThanThreshold it'd be nice to have a negative test. Do you think it is possible? The patch introduces a number of white-space changes such as <noformat> + <noformat> or <noformat> * <ul> + * <ul> <noformat> or <noformat> public LeaseManager leaseManager = new LeaseManager(this); + public LeaseManager leaseManager = new LeaseManager(this); <noformat> and so on... Please clean them up and resubmit because it is kinda hard to follow right now.
          Hide
          Konstantin Boudnik added a comment -

          Also, I have posted the system test for exactly this situation so now you can easily verify if your patch works

          Show
          Konstantin Boudnik added a comment - Also, I have posted the system test for exactly this situation so now you can easily verify if your patch works
          Hide
          Konstantin Boudnik added a comment -

          You test has failed the application to the trunk (this is where test-patch works). Considering from the list of affected versions your patch isn't intended for trunk, right?

          Show
          Konstantin Boudnik added a comment - You test has failed the application to the trunk (this is where test-patch works). Considering from the list of affected versions your patch isn't intended for trunk, right?
          Hide
          Devaraj K added a comment -

          It passed in my system and shown +1 but it failed to apply the patch in Hudson.

          Below is the result shown in my system,

           
           [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec] 
               [exec]     +1 system test framework.  The patch passed system test framework compile.
               [exec] 
               [exec] 
               [exec] 
               [exec] 
               [exec] ======================================================================
               [exec] ======================================================================
               [exec]     Finished build.
               [exec] ======================================================================
               [exec] ======================================================================
               [exec] 
               [exec] 
          
          BUILD SUCCESSFUL
          Total time: 22 minutes 46 seconds
          
          Show
          Devaraj K added a comment - It passed in my system and shown +1 but it failed to apply the patch in Hudson. Below is the result shown in my system, [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== [exec] [exec] BUILD SUCCESSFUL Total time: 22 minutes 46 seconds
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12469160/HDFS-1594.patch
          against trunk revision 1062052.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/127//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12469160/HDFS-1594.patch against trunk revision 1062052. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/127//console This message is automatically generated.
          Hide
          Devaraj K added a comment -

          Provided the patch as per above solution.

          Show
          Devaraj K added a comment - Provided the patch as per above solution.
          Hide
          Devaraj K added a comment -

          When the disk becomes full, name node file system (fsimage, edits) is getting corrupted and also name node is getting shutdown. When we try to restart, name node is not starting because the name node file system is corrupted.

          This can be avoided this way,

          We can implement a daemon to monitor the disk usage for periodically and if the disk usage reaches the threshold value, put the name node into Safe mode so that no modification to file system will occur. Once the disk usage reaches below the threshold, name node will be put out of the safe mode.

          Show
          Devaraj K added a comment - When the disk becomes full, name node file system (fsimage, edits) is getting corrupted and also name node is getting shutdown. When we try to restart, name node is not starting because the name node file system is corrupted. This can be avoided this way, We can implement a daemon to monitor the disk usage for periodically and if the disk usage reaches the threshold value, put the name node into Safe mode so that no modification to file system will occur. Once the disk usage reaches below the threshold, name node will be put out of the safe mode.

            People

            • Assignee:
              Aaron T. Myers
              Reporter:
              Devaraj K
            • Votes:
              1 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development