Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1952

FSEditLog.open() appears to succeed even if all EDITS directories fail

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0, 0.23.0
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: None
    • Labels:

      Description

      FSEditLog.open() appears to "succeed" even if all of the individual directories failed to allow creation of an EditLogOutputStream. The problem and solution are essentially similar to that of HDFS-1505.

      1. hdfs-1952-0.22.patch
        4 kB
        Andrew
      2. hdfs-1952.patch
        5 kB
        Andrew
      3. hdfs-1952.patch
        5 kB
        Andrew
      4. hdfs-1952.patch
        5 kB
        Andrew

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-22-branch #70 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/70/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #70 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/70/ )
          Hide
          Matt Foley added a comment -

          Committed to v22. Thanks, Andrew!

          Show
          Matt Foley added a comment - Committed to v22. Thanks, Andrew!
          Hide
          Todd Lipcon added a comment -

          This still needs to be committed to 0.22.

          Show
          Todd Lipcon added a comment - This still needs to be committed to 0.22.
          Hide
          Owen O'Malley added a comment -

          Resolving this, since it was committed to trunk.

          Show
          Owen O'Malley added a comment - Resolving this, since it was committed to trunk.
          Hide
          Matt Foley added a comment -

          +1 on the v22 version.
          Confirmed compiles successfully, and new unit test fails before the FSEditLog change, passes after.

          Show
          Matt Foley added a comment - +1 on the v22 version. Confirmed compiles successfully, and new unit test fails before the FSEditLog change, passes after.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/ )
          Hide
          Matt Foley added a comment -

          Hudson automated QA only handles patches against trunk.
          Turning off "Patch" to stop spurious tests on v22 patch.

          Show
          Matt Foley added a comment - Hudson automated QA only handles patches against trunk. Turning off "Patch" to stop spurious tests on v22 patch.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #677 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/677/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #677 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/677/ )
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479971/hdfs-1952-0.22.patch
          against trunk revision 1125601.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/606//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479971/hdfs-1952-0.22.patch against trunk revision 1125601. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/606//console This message is automatically generated.
          Hide
          Andrew added a comment -

          Patch against 0.22 branch. Testcase is more ugly but idea is the same.

          Show
          Andrew added a comment - Patch against 0.22 branch. Testcase is more ugly but idea is the same.
          Hide
          Matt Foley added a comment -

          Andrew, if you want examples you can check out the diffs between v23 and v22 patches for HDFS-1505 and HDFS-1921.

          Show
          Matt Foley added a comment - Andrew, if you want examples you can check out the diffs between v23 and v22 patches for HDFS-1505 and HDFS-1921 .
          Hide
          Todd Lipcon added a comment -

          Committed to trunk. Congrats on your first patch, Andrew!

          I believe this bug also effects 0.22, but this patch doesn't apply as-is. Would you mind checking out the 0.22 branch and uploading a patch that applies against that code? (it's only slightly different)

          Show
          Todd Lipcon added a comment - Committed to trunk. Congrats on your first patch, Andrew! I believe this bug also effects 0.22, but this patch doesn't apply as-is. Would you mind checking out the 0.22 branch and uploading a patch that applies against that code? (it's only slightly different)
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479843/hdfs-1952.patch
          against trunk revision 1125057.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
          org.apache.hadoop.hdfs.TestFileConcurrentReader
          org.apache.hadoop.hdfs.TestHDFSTrash

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/592//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/592//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/592//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479843/hdfs-1952.patch against trunk revision 1125057. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.hdfs.TestHDFSTrash +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/592//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/592//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/592//console This message is automatically generated.
          Hide
          Matt Foley added a comment -

          Sorry I missed this the first time. It's minor, so you don't have to re-spin the patch just for this, but for future reference:
          Per the coding guidelines (http://wiki.apache.org/hadoop/HowToContribute#Making_Changes) please add { } after if statements, even single-line ones. Thanks.

          +1 pending Hudson test-patch results.

          Show
          Matt Foley added a comment - Sorry I missed this the first time. It's minor, so you don't have to re-spin the patch just for this, but for future reference: Per the coding guidelines ( http://wiki.apache.org/hadoop/HowToContribute#Making_Changes ) please add { } after if statements, even single-line ones. Thanks. +1 pending Hudson test-patch results.
          Hide
          Andrew added a comment -

          Used --strip-prefix this time. Tested application to trunk with "patch -p0 < hdfs-1952.patch", hopefully Hudson likes it.

          Show
          Andrew added a comment - Used --strip-prefix this time. Tested application to trunk with "patch -p0 < hdfs-1952.patch", hopefully Hudson likes it.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479839/hdfs-1952.patch
          against trunk revision 1125057.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/591//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479839/hdfs-1952.patch against trunk revision 1125057. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/591//console This message is automatically generated.
          Hide
          Andrew added a comment -

          Incorporated comments from Todd and Matt.

          Show
          Andrew added a comment - Incorporated comments from Todd and Matt.
          Hide
          Matt Foley added a comment -

          Agree. Maybe change the exception message to "Failed to initialize edits log in any storage directory."

          The test-patch failures are recurring issues unrelated to this patch.

          Show
          Matt Foley added a comment - Agree. Maybe change the exception message to "Failed to initialize edits log in any storage directory." The test-patch failures are recurring issues unrelated to this patch.
          Hide
          Todd Lipcon added a comment -

          Some nits:

          • the new closing brace '}' is indented incorrectly (before addNewEditLogStream in FSEditLog.java)
          • in testFailedOpen there are a few commented-out assignments of fileSys – please remove these lines

          Otherwise looks good to me!

          Show
          Todd Lipcon added a comment - Some nits: the new closing brace '}' is indented incorrectly (before addNewEditLogStream in FSEditLog.java) in testFailedOpen there are a few commented-out assignments of fileSys – please remove these lines Otherwise looks good to me!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479683/hdfs-1952.patch
          against trunk revision 1124459.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
          org.apache.hadoop.hdfs.TestFileConcurrentReader
          org.apache.hadoop.tools.TestJMXGet

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/570//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/570//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/570//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479683/hdfs-1952.patch against trunk revision 1124459. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.tools.TestJMXGet +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/570//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/570//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/570//console This message is automatically generated.

            People

            • Assignee:
              Andrew
              Reporter:
              Matt Foley
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development