Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8965

Harden edit log reading code against out of memory errors

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      We should harden the edit log reading code against out of memory errors. Now that each op has a length prefix and a checksum, we can validate the checksum before trying to load the Op data. This should avoid out of memory errors when trying to load garbage data as Op data.

      1. HDFS-8965.001.patch
        18 kB
        Colin P. McCabe
      2. HDFS-8965.002.patch
        18 kB
        Colin P. McCabe
      3. HDFS-8965.003.patch
        19 kB
        Colin P. McCabe
      4. HDFS-8965.004.patch
        24 kB
        Colin P. McCabe
      5. HDFS-8965.005.patch
        25 kB
        Colin P. McCabe
      6. HDFS-8965.006.patch
        25 kB
        Colin P. McCabe
      7. HDFS-8965.007.patch
        25 kB
        Colin P. McCabe

        Issue Links

          Activity

          Hide
          cmccabe Colin P. McCabe added a comment -

          The testFuzzSequences unit test fails without this patch when MAX_INVALID_SEQ = 50000

          Show
          cmccabe Colin P. McCabe added a comment - The testFuzzSequences unit test fails without this patch when MAX_INVALID_SEQ = 50000
          Hide
          cmccabe Colin P. McCabe added a comment -

          some minor fixes to v1

          Show
          cmccabe Colin P. McCabe added a comment - some minor fixes to v1
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 16m 8s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 2m 12s The patch appears to cause the build to fail.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752600/HDFS-8965.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f44b599
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12145/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 16m 8s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 2m 12s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752600/HDFS-8965.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f44b599 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12145/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 17m 42s Pre-patch trunk has 4 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 2m 13s The patch appears to cause the build to fail.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752600/HDFS-8965.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f44b599
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12147/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12147/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 42s Pre-patch trunk has 4 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 2m 13s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752600/HDFS-8965.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f44b599 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12147/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12147/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Fix compilation error

          Show
          cmccabe Colin P. McCabe added a comment - Fix compilation error
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Colin, thanks for working on this. Seems like the idea is to, for length-prefixed edit logs, to check the checksum before reading the fields. Sensible and good change, overall looks good. Mostly nit comments:

          • A comment mentions a 1-byte txid, I think you meant opid. Having the expected format in the comment is super helpful too, thanks. Mind replicating it for the other two readers too?
          • Why move the temp array into a member? It'll probably be stack allocated and thus no GC pressure, 4KB should fit too.
          • Great opportunity to fix the LayoutVersion.EDITS_CHESKUM typo
          • LengthPrefixedReader#scanOp, shouldn't this be basically readOp but without the second pass to readFields? Not sure why it doesn't validate the max edit size or the checksum. The TODO is relevant.
          • Can we get a unit test for this situation?
          • Is some code sharing is possible among the three readOp's for getting the FSEditLogOp, this bit?
          byte opCodeByte;
                try {
                  opCodeByte = in.readByte();
                } catch (EOFException eof) {
                  // EOF at an opcode boundary is expected.
                  return null;
                }
                FSEditLogOpCodes opCode = FSEditLogOpCodes.fromByte(opCodeByte);
                if (opCode == OP_INVALID) {
                  verifyTerminator();
                  return null;
                }
                FSEditLogOp op = cache.get(opCode);
                if (op == null) {
                  throw new IOException("Read invalid opcode " + opCode);
                }
          
          Show
          andrew.wang Andrew Wang added a comment - Hi Colin, thanks for working on this. Seems like the idea is to, for length-prefixed edit logs, to check the checksum before reading the fields. Sensible and good change, overall looks good. Mostly nit comments: A comment mentions a 1-byte txid, I think you meant opid. Having the expected format in the comment is super helpful too, thanks. Mind replicating it for the other two readers too? Why move the temp array into a member? It'll probably be stack allocated and thus no GC pressure, 4KB should fit too. Great opportunity to fix the LayoutVersion.EDITS_CHESKUM typo LengthPrefixedReader#scanOp, shouldn't this be basically readOp but without the second pass to readFields? Not sure why it doesn't validate the max edit size or the checksum. The TODO is relevant. Can we get a unit test for this situation? Is some code sharing is possible among the three readOp's for getting the FSEditLogOp, this bit? byte opCodeByte; try { opCodeByte = in.readByte(); } catch (EOFException eof) { // EOF at an opcode boundary is expected. return null; } FSEditLogOpCodes opCode = FSEditLogOpCodes.fromByte(opCodeByte); if (opCode == OP_INVALID) { verifyTerminator(); return null; } FSEditLogOp op = cache.get(opCode); if (op == null) { throw new IOException("Read invalid opcode " + opCode); }
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 18m 27s Pre-patch trunk has 4 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 52s There were no new javac warning messages.
          +1 javadoc 9m 59s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 36s The applied patch generated 12 new checkstyle issues (total was 385, now 392).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          -1 findbugs 3m 25s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 7s Pre-build of native portion
          -1 hdfs tests 160m 49s Tests failed in hadoop-hdfs.
          -1 hdfs tests 0m 14s Tests failed in bkjournal.
              208m 8s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.crypto.TestHdfsCryptoStreams
            hadoop.hdfs.server.namenode.TestFSNamesystem
            hadoop.hdfs.qjournal.server.TestJournal
          Failed build bkjournal



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752628/HDFS-8965.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4cbbfa2
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12151/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12151/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12151/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12151/artifact/patchprocess/testrun_hadoop-hdfs.txt
          bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12151/artifact/patchprocess/testrun_bkjournal.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12151/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12151/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 27s Pre-patch trunk has 4 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 52s There were no new javac warning messages. +1 javadoc 9m 59s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 36s The applied patch generated 12 new checkstyle issues (total was 385, now 392). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 3m 25s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 7s Pre-build of native portion -1 hdfs tests 160m 49s Tests failed in hadoop-hdfs. -1 hdfs tests 0m 14s Tests failed in bkjournal.     208m 8s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.crypto.TestHdfsCryptoStreams   hadoop.hdfs.server.namenode.TestFSNamesystem   hadoop.hdfs.qjournal.server.TestJournal Failed build bkjournal Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752628/HDFS-8965.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4cbbfa2 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12151/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12151/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12151/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12151/artifact/patchprocess/testrun_hadoop-hdfs.txt bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12151/artifact/patchprocess/testrun_bkjournal.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12151/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12151/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks for the review.

          A comment mentions a 1-byte txid, I think you meant opid. Having the expected format in the comment is super helpful too, thanks. Mind replicating it for the other two readers too?

          sure

          Why move the temp array into a member? It'll probably be stack allocated and thus no GC pressure, 4KB should fit too.

          Hmm. I'm not sure if it will be stack allocated or not. The compiler would have to do some pretty fancy escape analysis since we're passing the array to many functions, any of which might hold a reference. It seems easier just to keep the buffer around, since we're using it a lot.

          Great opportunity to fix the LayoutVersion.EDITS_CHESKUM typo

          ok

          LengthPrefixedReader#scanOp, shouldn't this be basically readOp but without the second pass to readFields? Not sure why it doesn't validate the max edit size or the checksum. The TODO is relevant.

          Yeah, you're right... we should be validating checksums here. Fixed.

          It looks like this was actually a regression in edit log validation introduced by HDFS-6038. Previously, the JN did validate checksums when attempting to skip to a specific op, and after that change, it didn't.

          Can we get a unit test for this situation?

          Yeah, let me add a unit test that scanOp validates checksums. It should be possible to do as a true unit test too.

          Is some code sharing is possible among the three readOp's for getting the FSEditLogOp, this bit?

          I'm on the fence about this since I think it might make it harder to read since the control flow would be passing from base->derived->base again.

          Show
          cmccabe Colin P. McCabe added a comment - Thanks for the review. A comment mentions a 1-byte txid, I think you meant opid. Having the expected format in the comment is super helpful too, thanks. Mind replicating it for the other two readers too? sure Why move the temp array into a member? It'll probably be stack allocated and thus no GC pressure, 4KB should fit too. Hmm. I'm not sure if it will be stack allocated or not. The compiler would have to do some pretty fancy escape analysis since we're passing the array to many functions, any of which might hold a reference. It seems easier just to keep the buffer around, since we're using it a lot. Great opportunity to fix the LayoutVersion.EDITS_CHESKUM typo ok LengthPrefixedReader#scanOp, shouldn't this be basically readOp but without the second pass to readFields? Not sure why it doesn't validate the max edit size or the checksum. The TODO is relevant. Yeah, you're right... we should be validating checksums here. Fixed. It looks like this was actually a regression in edit log validation introduced by HDFS-6038 . Previously, the JN did validate checksums when attempting to skip to a specific op, and after that change, it didn't. Can we get a unit test for this situation? Yeah, let me add a unit test that scanOp validates checksums. It should be possible to do as a true unit test too. Is some code sharing is possible among the three readOp's for getting the FSEditLogOp, this bit? I'm on the fence about this since I think it might make it harder to read since the control flow would be passing from base->derived->base again.
          Hide
          andrew.wang Andrew Wang added a comment -

          Hmm. I'm not sure if it will be stack allocated or not...

          I did a little background research about Java escape analysis and it's pretty fancy, it can even stack allocate across function calls once things are inlined. Our temp array is used for a read loop within a function, so the scope looks a lot like the test program on this page[1] which gets stack allocated. If we trust in the JVM for this, I'd prefer to see local variables rather than members.

          [1] http://www.burnison.ca/articles/escape-analysis-stack-allocated-objects

          Show
          andrew.wang Andrew Wang added a comment - Hmm. I'm not sure if it will be stack allocated or not... I did a little background research about Java escape analysis and it's pretty fancy, it can even stack allocate across function calls once things are inlined. Our temp array is used for a read loop within a function, so the scope looks a lot like the test program on this page [1] which gets stack allocated. If we trust in the JVM for this, I'd prefer to see local variables rather than members. [1] http://www.burnison.ca/articles/escape-analysis-stack-allocated-objects
          Hide
          cmccabe Colin P. McCabe added a comment - - edited

          Added checksumming for scanOp. I added a unit test that scanOp now verifies checksums, and verified that it failed on trunk but passed with the patch. It's a true unit test which doesn't start a miniDFSCluster. Fixed the typo in EDITS_CHEKSUM.

          I agree that there is a very good chance that a large array would be allocated on the stack. But there's also a chance that it wouldn't. Since the difference in verboseness is negligable (its a single extra line) it seems like we should just allocate it inside the Reader.

          Show
          cmccabe Colin P. McCabe added a comment - - edited Added checksumming for scanOp. I added a unit test that scanOp now verifies checksums, and verified that it failed on trunk but passed with the patch. It's a true unit test which doesn't start a miniDFSCluster. Fixed the typo in EDITS_CHEKSUM. I agree that there is a very good chance that a large array would be allocated on the stack. But there's also a chance that it wouldn't. Since the difference in verboseness is negligable (its a single extra line) it seems like we should just allocate it inside the Reader.
          Hide
          andrew.wang Andrew Wang added a comment -

          Reeeeeally sure that this would get stack allocated but I'll let it slide. Noticed the other two readers don't have a format comment, but that's not a big deal either I guess.

          Do we also already have tests for invalid op lengths (e.g. greater than max op size)? I see testFuzzSequences but that's not explicit.

          Otherwise LGTM +1 pending

          Show
          andrew.wang Andrew Wang added a comment - Reeeeeally sure that this would get stack allocated but I'll let it slide. Noticed the other two readers don't have a format comment, but that's not a big deal either I guess. Do we also already have tests for invalid op lengths (e.g. greater than max op size)? I see testFuzzSequences but that's not explicit. Otherwise LGTM +1 pending
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 18m 35s Pre-patch trunk has 2 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 54s There were no new javac warning messages.
          +1 javadoc 10m 4s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 34s The applied patch generated 12 new checkstyle issues (total was 401, now 407).
          +1 whitespace 0m 3s The patch has no lines that end in whitespace.
          +1 install 1m 27s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          -1 findbugs 3m 22s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 8s Pre-build of native portion
          -1 hdfs tests 177m 38s Tests failed in hadoop-hdfs.
          -1 hdfs tests 0m 14s Tests failed in bkjournal.
              225m 5s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.qjournal.client.TestQuorumJournalManager
            hadoop.hdfs.server.namenode.TestFSImageWithAcl
            hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.TestRenameWhileOpen
            hadoop.hdfs.qjournal.server.TestJournal
          Timed out tests org.apache.hadoop.hdfs.web.TestWebHDFSAcl
          Failed build bkjournal



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752906/HDFS-8965.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 035ed26
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12187/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12187/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12187/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12187/artifact/patchprocess/testrun_hadoop-hdfs.txt
          bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12187/artifact/patchprocess/testrun_bkjournal.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12187/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12187/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 35s Pre-patch trunk has 2 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 54s There were no new javac warning messages. +1 javadoc 10m 4s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 34s The applied patch generated 12 new checkstyle issues (total was 401, now 407). +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 27s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 3m 22s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 8s Pre-build of native portion -1 hdfs tests 177m 38s Tests failed in hadoop-hdfs. -1 hdfs tests 0m 14s Tests failed in bkjournal.     225m 5s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.qjournal.client.TestQuorumJournalManager   hadoop.hdfs.server.namenode.TestFSImageWithAcl   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestRenameWhileOpen   hadoop.hdfs.qjournal.server.TestJournal Timed out tests org.apache.hadoop.hdfs.web.TestWebHDFSAcl Failed build bkjournal Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752906/HDFS-8965.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 035ed26 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12187/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12187/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12187/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12187/artifact/patchprocess/testrun_hadoop-hdfs.txt bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12187/artifact/patchprocess/testrun_bkjournal.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12187/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12187/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 18m 32s Pre-patch trunk has 2 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 8m 6s There were no new javac warning messages.
          +1 javadoc 10m 23s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 39s The applied patch generated 12 new checkstyle issues (total was 401, now 407).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse.
          -1 findbugs 3m 27s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 22s Pre-build of native portion
          -1 hdfs tests 191m 46s Tests failed in hadoop-hdfs.
          +1 hdfs tests 6m 33s Tests passed in bkjournal.
              246m 37s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.qjournal.server.TestJournal



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752906/HDFS-8965.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 035ed26
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12184/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12184/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12184/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12184/artifact/patchprocess/testrun_hadoop-hdfs.txt
          bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12184/artifact/patchprocess/testrun_bkjournal.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12184/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12184/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 32s Pre-patch trunk has 2 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 6s There were no new javac warning messages. +1 javadoc 10m 23s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 39s The applied patch generated 12 new checkstyle issues (total was 401, now 407). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse. -1 findbugs 3m 27s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 22s Pre-build of native portion -1 hdfs tests 191m 46s Tests failed in hadoop-hdfs. +1 hdfs tests 6m 33s Tests passed in bkjournal.     246m 37s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.qjournal.server.TestJournal Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752906/HDFS-8965.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 035ed26 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12184/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12184/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12184/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12184/artifact/patchprocess/testrun_hadoop-hdfs.txt bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12184/artifact/patchprocess/testrun_bkjournal.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12184/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12184/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Do we also already have tests for invalid op lengths (e.g. greater than max op size)? I see testFuzzSequences but that's not explicit.

          TestNameNodeRecovery#testNonDefaultMaxOpSize tests maximum op sizes.

          The latest patch fixes the test failure in TestJournal. The issue was that we need to ensure that scanOp works when the edit log version is newer than the latest version.

          Show
          cmccabe Colin P. McCabe added a comment - Do we also already have tests for invalid op lengths (e.g. greater than max op size)? I see testFuzzSequences but that's not explicit. TestNameNodeRecovery#testNonDefaultMaxOpSize tests maximum op sizes. The latest patch fixes the test failure in TestJournal . The issue was that we need to ensure that scanOp works when the edit log version is newer than the latest version.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 46s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 55s There were no new javac warning messages.
          +1 javadoc 10m 7s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 33s The applied patch generated 7 new checkstyle issues (total was 401, now 402).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          -1 findbugs 3m 26s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 10s Pre-build of native portion
          -1 hdfs tests 146m 58s Tests failed in hadoop-hdfs.
          +1 hdfs tests 3m 56s Tests passed in bkjournal.
              198m 31s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Timed out tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics
            org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12753081/HDFS-8965.005.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / e2c9b28
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12207/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12207/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12207/artifact/patchprocess/testrun_hadoop-hdfs.txt
          bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12207/artifact/patchprocess/testrun_bkjournal.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12207/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12207/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 46s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 55s There were no new javac warning messages. +1 javadoc 10m 7s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 33s The applied patch generated 7 new checkstyle issues (total was 401, now 402). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 3m 26s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 10s Pre-build of native portion -1 hdfs tests 146m 58s Tests failed in hadoop-hdfs. +1 hdfs tests 3m 56s Tests passed in bkjournal.     198m 31s   Reason Tests FindBugs module:hadoop-hdfs Timed out tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics   org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12753081/HDFS-8965.005.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / e2c9b28 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12207/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12207/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12207/artifact/patchprocess/testrun_hadoop-hdfs.txt bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12207/artifact/patchprocess/testrun_bkjournal.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12207/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12207/console This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thanks Colin P. McCabe for working on this issue..Great work here!!!

          Some Minor nits :

          1)

          in.skip(MIN_OP_LENGTH - 1 - CHECKSUM_LENGTH); // skip length and txid

          I think it will be easy to understand code like in.skip (4+8) compared to current statement. what do you say..?

          2)

          -     * @param logVersion The version of the data coming from the stream.

          Seems to be by mistake you might have removed this..?

          3)

               FSIMAGE_CHECKSUM(-26, "Support checksum for fsimage"),
               REMOVE_REL13_DISK_LAYOUT_SUPPORT(-27, "Remove support for 0.13 disk layout"),
          -    EDITS_CHESKUM(-28, "Support checksum for editlog"),
          +    EDITS_CHEKSUM(-28, "Support checksum for editlog"),
          

          I feel, it can be EDITS_CHECKSUM where it will be similar to FSIMAGE_CHECKSUM and as Andrew Wang mentioned,it is good chance to correct this typo..

          4)

          +     * 1-byte opid

          how about change to opcode instead of opid which we did not use..?

          5)

          The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.


          Currently jenkins report will not show this findbug warning,once after committed this patch,we can see..can you take care..? had seen like HDFS-8969...

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks Colin P. McCabe for working on this issue..Great work here!!! Some Minor nits : 1) in.skip(MIN_OP_LENGTH - 1 - CHECKSUM_LENGTH); // skip length and txid I think it will be easy to understand code like in.skip (4+8) compared to current statement. what do you say..? 2) - * @param logVersion The version of the data coming from the stream. Seems to be by mistake you might have removed this..? 3) FSIMAGE_CHECKSUM(-26, "Support checksum for fsimage" ), REMOVE_REL13_DISK_LAYOUT_SUPPORT(-27, "Remove support for 0.13 disk layout" ), - EDITS_CHESKUM(-28, "Support checksum for editlog" ), + EDITS_CHEKSUM(-28, "Support checksum for editlog" ), I feel, it can be EDITS_CHE C KSUM where it will be similar to FSIMAGE_CHECKSUM and as Andrew Wang mentioned,it is good chance to correct this typo.. 4) + * 1- byte opid how about change to opcode instead of opid which we did not use..? 5) The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. Currently jenkins report will not show this findbug warning,once after committed this patch,we can see..can you take care..? had seen like HDFS-8969 ...
          Hide
          cmccabe Colin P. McCabe added a comment -

          I think it will be easy to understand code like in.skip (4+8) compared to current statement. what do you say..?

          ok

          Seems to be by mistake you might have removed [the logVersion JavaDoc]

          Fixed, thanks

          I feel, it can be EDITS_CHECKSUM where it will be similar to FSIMAGE_CHECKSUM and as Andrew Wang mentioned,it is good chance to correct this typo..

          Yes, it seems that I didn't actually correct the typo! Fixed.

          how about change [the comment about opid to opcode]

          Yes, we refer to "opcode" in other places, so why not here. Changed.

          Show
          cmccabe Colin P. McCabe added a comment - I think it will be easy to understand code like in.skip (4+8) compared to current statement. what do you say..? ok Seems to be by mistake you might have removed [the logVersion JavaDoc] Fixed, thanks I feel, it can be EDITS_CHECKSUM where it will be similar to FSIMAGE_CHECKSUM and as Andrew Wang mentioned,it is good chance to correct this typo.. Yes, it seems that I didn't actually correct the typo! Fixed. how about change [the comment about opid to opcode] Yes, we refer to "opcode" in other places, so why not here. Changed.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Currently jenkins report will not show this findbug warning,once after committed this patch,we can see..can you take care..? had seen like HDFS-8969...

          It looks like the findbugs warning was for Hadoop common, not Hadoop-HDFS, which was a bit confusing. I was able to find out that the warning was:

          org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$LengthPrefixedReader.decodeOp() ignores result of java.io.DataInputStream.skip(long)

          I have replaced the call to skip with a call to IOUtils#skipFully.

          Show
          cmccabe Colin P. McCabe added a comment - Currently jenkins report will not show this findbug warning,once after committed this patch,we can see..can you take care..? had seen like HDFS-8969 ... It looks like the findbugs warning was for Hadoop common, not Hadoop-HDFS, which was a bit confusing. I was able to find out that the warning was: org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$LengthPrefixedReader.decodeOp() ignores result of java.io.DataInputStream.skip(long) I have replaced the call to skip with a call to IOUtils#skipFully .
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 34s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 52s There were no new javac warning messages.
          +1 javadoc 10m 3s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 38s The applied patch generated 7 new checkstyle issues (total was 401, now 402).
          +1 whitespace 0m 3s The patch has no lines that end in whitespace.
          +1 install 1m 29s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 3m 21s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 15s Pre-build of native portion
          -1 hdfs tests 62m 54s Tests failed in hadoop-hdfs.
          -1 hdfs tests 0m 14s Tests failed in bkjournal.
              110m 32s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
          Timed out tests org.apache.hadoop.fs.contract.hdfs.TestHDFSContractDelete
            org.apache.hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA
          Failed build bkjournal



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12753347/HDFS-8965.006.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / caa04de
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12215/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12215/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12215/artifact/patchprocess/testrun_hadoop-hdfs.txt
          bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12215/artifact/patchprocess/testrun_bkjournal.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12215/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12215/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 34s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 52s There were no new javac warning messages. +1 javadoc 10m 3s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 38s The applied patch generated 7 new checkstyle issues (total was 401, now 402). +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 3m 21s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 15s Pre-build of native portion -1 hdfs tests 62m 54s Tests failed in hadoop-hdfs. -1 hdfs tests 0m 14s Tests failed in bkjournal.     110m 32s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints Timed out tests org.apache.hadoop.fs.contract.hdfs.TestHDFSContractDelete   org.apache.hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA Failed build bkjournal Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12753347/HDFS-8965.006.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / caa04de checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12215/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12215/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12215/artifact/patchprocess/testrun_hadoop-hdfs.txt bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12215/artifact/patchprocess/testrun_bkjournal.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12215/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12215/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          missed a skip()

          Show
          cmccabe Colin P. McCabe added a comment - missed a skip()
          Hide
          andrew.wang Andrew Wang added a comment -

          LGTM +1

          Show
          andrew.wang Andrew Wang added a comment - LGTM +1
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 23s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 53s There were no new javac warning messages.
          +1 javadoc 10m 1s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 34s The applied patch generated 7 new checkstyle issues (total was 401, now 402).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 20s Pre-build of native portion
          -1 hdfs tests 162m 51s Tests failed in hadoop-hdfs.
          -1 hdfs tests 0m 16s Tests failed in bkjournal.
              210m 18s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
          Failed build bkjournal



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12753375/HDFS-8965.007.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / caa04de
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12219/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12219/artifact/patchprocess/testrun_hadoop-hdfs.txt
          bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12219/artifact/patchprocess/testrun_bkjournal.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12219/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12219/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 23s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 53s There were no new javac warning messages. +1 javadoc 10m 1s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 34s The applied patch generated 7 new checkstyle issues (total was 401, now 402). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 20s Pre-build of native portion -1 hdfs tests 162m 51s Tests failed in hadoop-hdfs. -1 hdfs tests 0m 16s Tests failed in bkjournal.     210m 18s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication Failed build bkjournal Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12753375/HDFS-8965.007.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / caa04de checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12219/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12219/artifact/patchprocess/testrun_hadoop-hdfs.txt bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12219/artifact/patchprocess/testrun_bkjournal.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12219/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12219/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          TestDNFencingWithReplication and bkjm test failures are unrelated to this change and did not reproduce for me locally. Checkstyle issues are more whines about file length and complaining about package-private variables. Committed to 2.8. Thanks, all!

          Show
          cmccabe Colin P. McCabe added a comment - TestDNFencingWithReplication and bkjm test failures are unrelated to this change and did not reproduce for me locally. Checkstyle issues are more whines about file length and complaining about package-private variables. Committed to 2.8. Thanks, all!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8375 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8375/)
          HDFS-8965. Harden edit log reading code against out of memory errors (cmccabe) (cmccabe: rev 24f6a7c9563757234f53ca23e12f9c9208b53082)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8375 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8375/ ) HDFS-8965 . Harden edit log reading code against out of memory errors (cmccabe) (cmccabe: rev 24f6a7c9563757234f53ca23e12f9c9208b53082) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Colin P. McCabeThanks for addressing all the comments neatly...+1 ( non-binding) on latest patch..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Colin P. McCabe Thanks for addressing all the comments neatly...+1 ( non-binding) on latest patch..
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1059 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1059/)
          HDFS-8965. Harden edit log reading code against out of memory errors (cmccabe) (cmccabe: rev 24f6a7c9563757234f53ca23e12f9c9208b53082)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1059 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1059/ ) HDFS-8965 . Harden edit log reading code against out of memory errors (cmccabe) (cmccabe: rev 24f6a7c9563757234f53ca23e12f9c9208b53082) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #325 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/325/)
          HDFS-8965. Harden edit log reading code against out of memory errors (cmccabe) (cmccabe: rev 24f6a7c9563757234f53ca23e12f9c9208b53082)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #325 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/325/ ) HDFS-8965 . Harden edit log reading code against out of memory errors (cmccabe) (cmccabe: rev 24f6a7c9563757234f53ca23e12f9c9208b53082) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #332 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/332/)
          HDFS-8965. Harden edit log reading code against out of memory errors (cmccabe) (cmccabe: rev 24f6a7c9563757234f53ca23e12f9c9208b53082)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #332 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/332/ ) HDFS-8965 . Harden edit log reading code against out of memory errors (cmccabe) (cmccabe: rev 24f6a7c9563757234f53ca23e12f9c9208b53082) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2274 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2274/)
          HDFS-8965. Harden edit log reading code against out of memory errors (cmccabe) (cmccabe: rev 24f6a7c9563757234f53ca23e12f9c9208b53082)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2274 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2274/ ) HDFS-8965 . Harden edit log reading code against out of memory errors (cmccabe) (cmccabe: rev 24f6a7c9563757234f53ca23e12f9c9208b53082) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #316 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/316/)
          HDFS-8965. Harden edit log reading code against out of memory errors (cmccabe) (cmccabe: rev 24f6a7c9563757234f53ca23e12f9c9208b53082)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #316 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/316/ ) HDFS-8965 . Harden edit log reading code against out of memory errors (cmccabe) (cmccabe: rev 24f6a7c9563757234f53ca23e12f9c9208b53082) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2255 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2255/)
          HDFS-8965. Harden edit log reading code against out of memory errors (cmccabe) (cmccabe: rev 24f6a7c9563757234f53ca23e12f9c9208b53082)

          • hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileInputStream.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2255 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2255/ ) HDFS-8965 . Harden edit log reading code against out of memory errors (cmccabe) (cmccabe: rev 24f6a7c9563757234f53ca23e12f9c9208b53082) hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileInputStream.java
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Colin P. McCabe,

          Thanks for your earlier work on this issue. I have a question, sounds like that the edit log is somehow corrupted, and we tried to avoid out-of-memory issue when loading the edit log. Two questions:
          1. What do we do with the corrupted edit log entry? skip it?
          2. Do we have idea about whether it is a bug that corrupt edit log, or is it a disk/network problem that corrupt the edit log?

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Colin P. McCabe , Thanks for your earlier work on this issue. I have a question, sounds like that the edit log is somehow corrupted, and we tried to avoid out-of-memory issue when loading the edit log. Two questions: 1. What do we do with the corrupted edit log entry? skip it? 2. Do we have idea about whether it is a bug that corrupt edit log, or is it a disk/network problem that corrupt the edit log? Thanks.
          Hide
          cmccabe Colin P. McCabe added a comment -

          1. What do we do with the corrupted edit log entry? skip it?

          We only would attempt to skip corrupted edit log entries if recovery mode were turned on. Otherwise, an exception would be thrown, which would ordinarily lead to us trying to read a different copy of the edit log.

          2. Do we have idea about whether it is a bug that corrupt edit log, or is it a disk/network problem that corrupt the edit log?

          In general, I don't see how we could know the answer to that question without more information.

          Show
          cmccabe Colin P. McCabe added a comment - 1. What do we do with the corrupted edit log entry? skip it? We only would attempt to skip corrupted edit log entries if recovery mode were turned on. Otherwise, an exception would be thrown, which would ordinarily lead to us trying to read a different copy of the edit log. 2. Do we have idea about whether it is a bug that corrupt edit log, or is it a disk/network problem that corrupt the edit log? In general, I don't see how we could know the answer to that question without more information.

            People

            • Assignee:
              cmccabe Colin P. McCabe
              Reporter:
              cmccabe Colin P. McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development