Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8964

When validating the edit log, do not read at or beyond the file offset that is being written

    Details

    • Target Version/s:

      Description

      NN/JN validates in-progress edit log files in multiple scenarios, via EditLogFile#validateLog. The method scans through the edit log file to find the last transaction ID.

      However, an in-progress edit log file could be actively written to, which creates a race condition and causes incorrect data to be read (and later we attempt to interpret the data as ops). This causes problems for INotify, which reads edit log entries while the edit log is still being written.

      Currently validateLog is used in 3 places:

      1. NN getEditsFromTxid
      2. JN getEditLogManifest
      3. NN/JN recoverUnfinalizedSegments

      In the first two scenarios we should provide a maximum TxId to validate in the in-progress file. The 3rd scenario won't cause a race condition because only non-current in-progress edit log files are validated.

      validateLog is actually only used with in-progress files, and could use a better name and Javadoc.

      1. HDFS-8964.00.patch
        13 kB
        Zhe Zhang
      2. HDFS-8964.01.patch
        12 kB
        Zhe Zhang
      3. HDFS-8964.02.patch
        15 kB
        Zhe Zhang
      4. HDFS-8964.03.patch
        22 kB
        Zhe Zhang
      5. HDFS-8964.04.patch
        22 kB
        Zhe Zhang
      6. HDFS-8964.05.patch
        22 kB
        Zhe Zhang
      7. HDFS-8964.06.patch
        22 kB
        Zhe Zhang

        Activity

        Hide
        zhz Zhe Zhang added a comment -

        Initial patch to demonstrate the idea and trigger Jenkins. Will add a test to emulate the described read-being-written race condition and verify the fix addresses it.

        Show
        zhz Zhe Zhang added a comment - Initial patch to demonstrate the idea and trigger Jenkins. Will add a test to emulate the described read-being-written race condition and verify the fix addresses it.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 18m 45s Pre-patch trunk has 4 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
        +1 javac 7m 57s There were no new javac warning messages.
        +1 javadoc 10m 6s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 41s There were no new checkstyle issues.
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 32s mvn install still works.
        +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
        -1 findbugs 3m 25s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        +1 native 3m 11s Pre-build of native portion
        -1 hdfs tests 135m 4s Tests failed in hadoop-hdfs.
        +1 hdfs tests 6m 19s Tests passed in bkjournal.
            189m 2s  



        Reason Tests
        FindBugs module:hadoop-hdfs
        Failed unit tests hadoop.hdfs.server.blockmanagement.TestBlockManager
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.server.namenode.TestFileJournalManager
        Timed out tests org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          org.apache.hadoop.hdfs.server.namenode.TestEditLog



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12752674/HDFS-8964.00.patch
        Optional Tests javac unit findbugs checkstyle javadoc
        git revision trunk / 4cbbfa2
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12157/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12157/artifact/patchprocess/whitespace.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12157/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12157/artifact/patchprocess/testrun_hadoop-hdfs.txt
        bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12157/artifact/patchprocess/testrun_bkjournal.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12157/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12157/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 45s Pre-patch trunk has 4 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 7m 57s There were no new javac warning messages. +1 javadoc 10m 6s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 41s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. -1 findbugs 3m 25s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 11s Pre-build of native portion -1 hdfs tests 135m 4s Tests failed in hadoop-hdfs. +1 hdfs tests 6m 19s Tests passed in bkjournal.     189m 2s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.namenode.TestFileJournalManager Timed out tests org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   org.apache.hadoop.hdfs.server.namenode.TestEditLog Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752674/HDFS-8964.00.patch Optional Tests javac unit findbugs checkstyle javadoc git revision trunk / 4cbbfa2 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12157/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12157/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12157/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12157/artifact/patchprocess/testrun_hadoop-hdfs.txt bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12157/artifact/patchprocess/testrun_bkjournal.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12157/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12157/console This message was automatically generated.
        Hide
        cmccabe Colin P. McCabe added a comment -

        The overall idea looks good. Rather than special-casing negative values for maxTxId, we can just pass in Long.MAX_VALUE in cases where we really don't want a limit. Of course we also need to plug this into the relevant places in NameNode and JournalNode.

        Show
        cmccabe Colin P. McCabe added a comment - The overall idea looks good. Rather than special-casing negative values for maxTxId, we can just pass in Long.MAX_VALUE in cases where we really don't want a limit. Of course we also need to plug this into the relevant places in NameNode and JournalNode.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Colin for taking a look. Updating the patch to better handle cases with no provided maxTxId.

        I also found scanLog is identical to validateLog and removed it from two places.

        FileJournalManager#getRemoteEditLogs and selectInputStreams are already updated to provide maxTxId. Where else do you think we are trying to read an active in-progress edit file?

        Show
        zhz Zhe Zhang added a comment - Thanks Colin for taking a look. Updating the patch to better handle cases with no provided maxTxId . I also found scanLog is identical to validateLog and removed it from two places. FileJournalManager#getRemoteEditLogs and selectInputStreams are already updated to provide maxTxId . Where else do you think we are trying to read an active in-progress edit file?
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 53s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 52s There were no new javac warning messages.
        +1 javadoc 10m 7s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 22s The applied patch generated 2 new checkstyle issues (total was 162, now 162).
        -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 31s mvn install still works.
        +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
        -1 findbugs 2m 37s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        +1 native 3m 21s Pre-build of native portion
        -1 hdfs tests 189m 35s Tests failed in hadoop-hdfs.
            235m 24s  



        Reason Tests
        FindBugs module:hadoop-hdfs
        Failed unit tests hadoop.cli.TestHDFSCLI
          hadoop.hdfs.qjournal.server.TestJournal
          hadoop.hdfs.server.namenode.TestFSNamesystem
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.server.namenode.TestFileJournalManager



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12752950/HDFS-8964.01.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / e166c03
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12193/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12193/artifact/patchprocess/whitespace.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12193/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12193/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12193/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12193/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 53s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 52s There were no new javac warning messages. +1 javadoc 10m 7s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 22s The applied patch generated 2 new checkstyle issues (total was 162, now 162). -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. -1 findbugs 2m 37s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 21s Pre-build of native portion -1 hdfs tests 189m 35s Tests failed in hadoop-hdfs.     235m 24s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.cli.TestHDFSCLI   hadoop.hdfs.qjournal.server.TestJournal   hadoop.hdfs.server.namenode.TestFSNamesystem   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.namenode.TestFileJournalManager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752950/HDFS-8964.01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / e166c03 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12193/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12193/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12193/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12193/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12193/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12193/console This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Updating the patch with a test (which turns out to be the hardest part). Basically, we want to verify that NN doesn't try to read past the latest synced op when validating the edit log file. But the validateLog logic absorbs all exceptions.

        So I had to verify the NameNode log for the error message that is expected without the change.

        Show
        zhz Zhe Zhang added a comment - Updating the patch with a test (which turns out to be the hardest part). Basically, we want to verify that NN doesn't try to read past the latest synced op when validating the edit log file. But the validateLog logic absorbs all exceptions. So I had to verify the NameNode log for the error message that is expected without the change.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 24s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 5s There were no new javac warning messages.
        +1 javadoc 10m 13s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 20s The applied patch generated 2 new checkstyle issues (total was 162, now 162).
        -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 29s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 2m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 19s Pre-build of native portion
        -1 hdfs tests 190m 38s Tests failed in hadoop-hdfs.
            235m 58s  



        Reason Tests
        Failed unit tests hadoop.hdfs.TestDFSRollback
          hadoop.hdfs.qjournal.server.TestJournal
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
          hadoop.hdfs.qjournal.TestSecureNNWithQJM
          hadoop.hdfs.server.namenode.TestFileJournalManager



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12753121/HDFS-8964.02.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / e2c9b28
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12211/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12211/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12211/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12211/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12211/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 24s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 5s There were no new javac warning messages. +1 javadoc 10m 13s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 20s The applied patch generated 2 new checkstyle issues (total was 162, now 162). -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 19s Pre-build of native portion -1 hdfs tests 190m 38s Tests failed in hadoop-hdfs.     235m 58s   Reason Tests Failed unit tests hadoop.hdfs.TestDFSRollback   hadoop.hdfs.qjournal.server.TestJournal   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.TestFileJournalManager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12753121/HDFS-8964.02.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / e2c9b28 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12211/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12211/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12211/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12211/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12211/console This message was automatically generated.
        Hide
        cmccabe Colin P. McCabe added a comment -

        Thank you, Zhe Zhang.

        JNStorage: import com.google.common.annotations.VisibleForTesting; was added, but not used.

        I also found scanLog is identical to validateLog and removed it from two places.

        They're not quite identical. validateLog ends up repeatedly calling FSEditLogInputStream#readOp, whereas scanLog ends up repeatedly calling FSEditLogInputStream#scanNextOp. In the past, scanNextOp did not validate checksums, whereas readOp did. However, HDFS-8965 is changing FSEditLogInputStream#scanNextOp to validate checksums. So I agree that it now makes sense to merge these two.

        FileJournalManager#getRemoteEditLogs: this isn't going to work.

          public List<RemoteEditLog> getRemoteEditLogs(long firstTxId,
              boolean inProgressOk) throws IOException {
        ...
                  elf.validateLog(firstTxId);
        

        The problem is that by doing this, we set EditLogFile#lastTxId to firstTxid, and then all subsequent edits in the in-progress log are lost. Instead of doing this, you should pass in the known highest txid from Journal.java (taking care to pass Long.MAX_VALUE if this is not known).

        Same issue in FileJournalManager#addStreamsToCollectionFromFiles.

            /** 
             * Find out where the edit log ends.
             * This will update the lastTxId of the EditLogFile or
             * mark it as corrupt if it is.
             */
            public void validateLog() throws IOException {
              validateLog(Long.MAX_VALUE);
            }
        
            public void validateLog(long maxTxId) throws IOException {
        

        I would prefer not to have this kind of function overloading, since it tends to obscure what is going on. In that vein, I think there are more places where we are calling validateLog without a maxTxid, where we should actually have a max txid, like Journal#getSegmentInfo.

              if (maxTxId > 0 && lastTxId > maxTxId) {
                break;
              }
        

        Why is maxTxid == 0 special here? If we want to have no limit, we can just pass in Long.MAX_VALUE. 0 is a bad value to overload to mean "there is no limit" because it means you have to special-case it. I also don't think this function correctly handles the case where we don't expect any edits in the log currently. In this case, it is quite valid for the caller to pass 0 as maxTxid, and we should always return an EditLogValidate with maxTxid == startTxId == 0. The check should be at the start of the loop, not the end.

         
             File[] files = new File(f1, "current").listFiles(new FilenameFilter() {
        -        @Override
        -        public boolean accept(File dir, String name) {
        -          if (name.startsWith(NNStorage.getFinalizedEditsFileName(startErrorTxId, 
        -                                  endErrorTxId))) {
        -            return true;
        -          }
        -          return false;
        +      @Override
        +      public boolean accept(File dir, String name) {
        +        if (name.startsWith(NNStorage.getFinalizedEditsFileName(startErrorTxId,
        +            endErrorTxId))) {
        +          return true;
                 }
        -      });
        +        return false;
        +      }
        +    });
        

        Let's not mess with the whitespace here. I don't see anything wrong with the original whitespace, and it makes backports harder. Similar with writeInt

        The unit test looks good to me. Can you also test that if we call EditLogFileInputStream#validateEditLOg(editLogFile, finalTxId - 1) we don't get the last edit log op, but if we call validateEditLOg(editLogFile, finalTxId), we do? It should be easy to do by keeping track of the last edit log transaction id in the loop where we're writing transactions (probably can get it out of FSNamesystem).

        Show
        cmccabe Colin P. McCabe added a comment - Thank you, Zhe Zhang . JNStorage: import com.google.common.annotations.VisibleForTesting; was added, but not used. I also found scanLog is identical to validateLog and removed it from two places. They're not quite identical. validateLog ends up repeatedly calling FSEditLogInputStream#readOp , whereas scanLog ends up repeatedly calling FSEditLogInputStream#scanNextOp . In the past, scanNextOp did not validate checksums, whereas readOp did. However, HDFS-8965 is changing FSEditLogInputStream#scanNextOp to validate checksums. So I agree that it now makes sense to merge these two. FileJournalManager#getRemoteEditLogs: this isn't going to work. public List<RemoteEditLog> getRemoteEditLogs( long firstTxId, boolean inProgressOk) throws IOException { ... elf.validateLog(firstTxId); The problem is that by doing this, we set EditLogFile#lastTxId to firstTxid , and then all subsequent edits in the in-progress log are lost. Instead of doing this, you should pass in the known highest txid from Journal.java (taking care to pass Long.MAX_VALUE if this is not known). Same issue in FileJournalManager#addStreamsToCollectionFromFiles . /** * Find out where the edit log ends. * This will update the lastTxId of the EditLogFile or * mark it as corrupt if it is. */ public void validateLog() throws IOException { validateLog( Long .MAX_VALUE); } public void validateLog( long maxTxId) throws IOException { I would prefer not to have this kind of function overloading, since it tends to obscure what is going on. In that vein, I think there are more places where we are calling validateLog without a maxTxid, where we should actually have a max txid, like Journal#getSegmentInfo . if (maxTxId > 0 && lastTxId > maxTxId) { break ; } Why is maxTxid == 0 special here? If we want to have no limit, we can just pass in Long.MAX_VALUE . 0 is a bad value to overload to mean "there is no limit" because it means you have to special-case it. I also don't think this function correctly handles the case where we don't expect any edits in the log currently. In this case, it is quite valid for the caller to pass 0 as maxTxid, and we should always return an EditLogValidate with maxTxid == startTxId == 0 . The check should be at the start of the loop, not the end. File[] files = new File(f1, "current" ).listFiles( new FilenameFilter() { - @Override - public boolean accept(File dir, String name) { - if (name.startsWith(NNStorage.getFinalizedEditsFileName(startErrorTxId, - endErrorTxId))) { - return true ; - } - return false ; + @Override + public boolean accept(File dir, String name) { + if (name.startsWith(NNStorage.getFinalizedEditsFileName(startErrorTxId, + endErrorTxId))) { + return true ; } - }); + return false ; + } + }); Let's not mess with the whitespace here. I don't see anything wrong with the original whitespace, and it makes backports harder. Similar with writeInt The unit test looks good to me. Can you also test that if we call EditLogFileInputStream#validateEditLOg(editLogFile, finalTxId - 1) we don't get the last edit log op, but if we call validateEditLOg(editLogFile, finalTxId) , we do? It should be easy to do by keeping track of the last edit log transaction id in the loop where we're writing transactions (probably can get it out of FSNamesystem).
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Colin for the review! Updating the patch to add lastSyncTxId to FileJournalManager. The idea is that FJM should be aware of the latest Tx ID synced to its files.

        Please let me know if the structure looks OK. I'll beef up the test in the next rev.

        Show
        zhz Zhe Zhang added a comment - Thanks Colin for the review! Updating the patch to add lastSyncTxId to FileJournalManager . The idea is that FJM should be aware of the latest Tx ID synced to its files. Please let me know if the structure looks OK. I'll beef up the test in the next rev.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 18m 36s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
        +1 javac 7m 57s There were no new javac warning messages.
        +1 javadoc 11m 29s There were no new javadoc warning messages.
        +1 release audit 0m 28s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 45s The applied patch generated 1 new checkstyle issues (total was 344, now 345).
        -1 whitespace 0m 2s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 41s mvn install still works.
        +1 eclipse:eclipse 0m 44s The patch built with eclipse:eclipse.
        +1 findbugs 4m 9s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 38s Pre-build of native portion
        -1 hdfs tests 104m 30s Tests failed in hadoop-hdfs.
        -1 hdfs tests 0m 15s Tests failed in bkjournal.
            155m 29s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.datanode.TestFsDatasetCache
          hadoop.hdfs.server.datanode.TestDataNodeUUID
          hadoop.hdfs.TestLeaseRecovery2
        Timed out tests org.apache.hadoop.hdfs.qjournal.client.TestIPCLoggerChannel
        Failed build bkjournal



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12753394/HDFS-8964.03.patch
        Optional Tests javac unit findbugs checkstyle javadoc
        git revision trunk / 826ae1c
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12221/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12221/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12221/artifact/patchprocess/testrun_hadoop-hdfs.txt
        bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12221/artifact/patchprocess/testrun_bkjournal.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12221/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12221/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 36s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 7m 57s There were no new javac warning messages. +1 javadoc 11m 29s There were no new javadoc warning messages. +1 release audit 0m 28s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 45s The applied patch generated 1 new checkstyle issues (total was 344, now 345). -1 whitespace 0m 2s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 41s mvn install still works. +1 eclipse:eclipse 0m 44s The patch built with eclipse:eclipse. +1 findbugs 4m 9s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 38s Pre-build of native portion -1 hdfs tests 104m 30s Tests failed in hadoop-hdfs. -1 hdfs tests 0m 15s Tests failed in bkjournal.     155m 29s   Reason Tests Failed unit tests hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.TestLeaseRecovery2 Timed out tests org.apache.hadoop.hdfs.qjournal.client.TestIPCLoggerChannel Failed build bkjournal Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12753394/HDFS-8964.03.patch Optional Tests javac unit findbugs checkstyle javadoc git revision trunk / 826ae1c checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12221/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12221/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12221/artifact/patchprocess/testrun_hadoop-hdfs.txt bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12221/artifact/patchprocess/testrun_bkjournal.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12221/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12221/console This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        All reported failures pass locally. Updating the patch to fix checkstyle and whitespace issues.

        The added test is causing the following warning:

        SLF4J: Class path contains multiple SLF4J bindings.
        SLF4J: Found binding in [jar:file:/Users/zhezhang/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
        SLF4J: Found binding in [jar:file:/Users/zhezhang/.m2/repository/ch/qos/logback/logback-classic/1.1.2/logback-classic-1.1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
        SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
        SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
        

        Not sure if it's we are OK with it.

        Show
        zhz Zhe Zhang added a comment - All reported failures pass locally. Updating the patch to fix checkstyle and whitespace issues. The added test is causing the following warning: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/Users/zhezhang/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/Users/zhezhang/.m2/repository/ch/qos/logback/logback-classic/1.1.2/logback-classic-1.1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http: //www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Not sure if it's we are OK with it.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 18m 20s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
        +1 javac 7m 45s There were no new javac warning messages.
        +1 javadoc 9m 57s There were no new javadoc warning messages.
        +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 39s There were no new checkstyle issues.
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 26s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 3m 14s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 9s Pre-build of native portion
        +1 hdfs tests 162m 45s Tests passed in hadoop-hdfs.
        +1 hdfs tests 3m 57s Tests passed in bkjournal.
            213m 15s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12753479/HDFS-8964.04.patch
        Optional Tests javac unit findbugs checkstyle javadoc
        git revision trunk / 7ad3556
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12227/artifact/patchprocess/testrun_hadoop-hdfs.txt
        bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12227/artifact/patchprocess/testrun_bkjournal.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12227/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12227/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 20s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 7m 45s There were no new javac warning messages. +1 javadoc 9m 57s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 39s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 26s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 14s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 9s Pre-build of native portion +1 hdfs tests 162m 45s Tests passed in hadoop-hdfs. +1 hdfs tests 3m 57s Tests passed in bkjournal.     213m 15s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12753479/HDFS-8964.04.patch Optional Tests javac unit findbugs checkstyle javadoc git revision trunk / 7ad3556 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12227/artifact/patchprocess/testrun_hadoop-hdfs.txt bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12227/artifact/patchprocess/testrun_bkjournal.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12227/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12227/console This message was automatically generated.
        Hide
        cmccabe Colin P. McCabe added a comment -

        FileJournalManager: it seems like "lastReadableTxId" is a better name than "syncTxId". After all, the last txid may or may not have been synced. This also reflects the fact that this starts out as Long.MAX_VALUE, indicating that we can read all txids.

        FsEditLogLoader#validateEditLog: maxTxId seems like a confusing name here. It's not really the maximum, since if there is a transaction gap, we may end up reading a later txid (and returning the transaction!). How about finalTxId, plus a JavaDoc indicating that we will stop reading after reading this or a higher txid.

        Please remove the whitespace-only changes, including the one in testEditLogFailOverFromCorrupt (I mentioned this above)

        Looks good aside from that. Thanks, Zhe Zhang.

        Show
        cmccabe Colin P. McCabe added a comment - FileJournalManager : it seems like "lastReadableTxId" is a better name than "syncTxId". After all, the last txid may or may not have been synced. This also reflects the fact that this starts out as Long.MAX_VALUE, indicating that we can read all txids. FsEditLogLoader#validateEditLog : maxTxId seems like a confusing name here. It's not really the maximum, since if there is a transaction gap, we may end up reading a later txid (and returning the transaction!). How about finalTxId , plus a JavaDoc indicating that we will stop reading after reading this or a higher txid. Please remove the whitespace-only changes, including the one in testEditLogFailOverFromCorrupt (I mentioned this above) Looks good aside from that. Thanks, Zhe Zhang .
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Colin for reviewing the patch again. Updating the patch to address the comments. I like the name lastReadableTxId. W.r.t the other name, I changed it to maxTxIdToValidate because even finalTxId sounds like the variable is associated with the log file, instead of the validation. LMK how it looks.

        Show
        zhz Zhe Zhang added a comment - Thanks Colin for reviewing the patch again. Updating the patch to address the comments. I like the name lastReadableTxId . W.r.t the other name, I changed it to maxTxIdToValidate because even finalTxId sounds like the variable is associated with the log file, instead of the validation. LMK how it looks.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 18m 13s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
        +1 javac 7m 43s There were no new javac warning messages.
        +1 javadoc 9m 59s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 36s The applied patch generated 1 new checkstyle issues (total was 343, now 344).
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 26s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 3m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 15s Pre-build of native portion
        -1 hdfs tests 162m 46s Tests failed in hadoop-hdfs.
        -1 hdfs tests 0m 19s Tests failed in bkjournal.
            209m 48s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.namenode.TestNameNodeMetricsLogger
        Failed build bkjournal



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12753846/HDFS-8964.05.patch
        Optional Tests javac unit findbugs checkstyle javadoc
        git revision trunk / afc88b3
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12256/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12256/artifact/patchprocess/testrun_hadoop-hdfs.txt
        bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12256/artifact/patchprocess/testrun_bkjournal.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12256/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12256/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 13s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 7m 43s There were no new javac warning messages. +1 javadoc 9m 59s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 36s The applied patch generated 1 new checkstyle issues (total was 343, now 344). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 26s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 15s Pre-build of native portion -1 hdfs tests 162m 46s Tests failed in hadoop-hdfs. -1 hdfs tests 0m 19s Tests failed in bkjournal.     209m 48s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.TestNameNodeMetricsLogger Failed build bkjournal Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12753846/HDFS-8964.05.patch Optional Tests javac unit findbugs checkstyle javadoc git revision trunk / afc88b3 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12256/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12256/artifact/patchprocess/testrun_hadoop-hdfs.txt bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12256/artifact/patchprocess/testrun_bkjournal.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12256/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12256/console This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Updating the patch to fix a checkstyle issue (line too long).

        TestNameNodeMetricsLogger passes locally.

        Show
        zhz Zhe Zhang added a comment - Updating the patch to fix a checkstyle issue (line too long). TestNameNodeMetricsLogger passes locally.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 18m 29s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
        +1 javac 8m 0s There were no new javac warning messages.
        +1 javadoc 10m 13s There were no new javadoc warning messages.
        +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 47s There were no new checkstyle issues.
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 27s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 3m 19s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 12s Pre-build of native portion
        +1 hdfs tests 163m 36s Tests passed in hadoop-hdfs.
        +1 hdfs tests 4m 1s Tests passed in bkjournal.
            215m 2s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12753921/HDFS-8964.06.patch
        Optional Tests javac unit findbugs checkstyle javadoc
        git revision trunk / 09c64ba
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12272/artifact/patchprocess/testrun_hadoop-hdfs.txt
        bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12272/artifact/patchprocess/testrun_bkjournal.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12272/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12272/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 29s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 8m 0s There were no new javac warning messages. +1 javadoc 10m 13s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 47s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 27s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 19s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 12s Pre-build of native portion +1 hdfs tests 163m 36s Tests passed in hadoop-hdfs. +1 hdfs tests 4m 1s Tests passed in bkjournal.     215m 2s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12753921/HDFS-8964.06.patch Optional Tests javac unit findbugs checkstyle javadoc git revision trunk / 09c64ba hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12272/artifact/patchprocess/testrun_hadoop-hdfs.txt bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12272/artifact/patchprocess/testrun_bkjournal.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12272/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12272/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 18m 8s Findbugs (version ) appears to be broken on trunk.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
        +1 javac 7m 49s There were no new javac warning messages.
        +1 javadoc 10m 11s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 42s There were no new checkstyle issues.
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 31s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 3m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 15s Pre-build of native portion
        -1 hdfs tests 164m 23s Tests failed in hadoop-hdfs.
        +1 hdfs tests 3m 58s Tests passed in bkjournal.
            215m 20s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.blockmanagement.TestNodeCount



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12753921/HDFS-8964.06.patch
        Optional Tests javac unit findbugs checkstyle javadoc
        git revision trunk / 09c64ba
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12273/artifact/patchprocess/testrun_hadoop-hdfs.txt
        bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12273/artifact/patchprocess/testrun_bkjournal.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12273/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12273/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 8s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 7m 49s There were no new javac warning messages. +1 javadoc 10m 11s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 42s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 3m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 15s Pre-build of native portion -1 hdfs tests 164m 23s Tests failed in hadoop-hdfs. +1 hdfs tests 3m 58s Tests passed in bkjournal.     215m 20s   Reason Tests Failed unit tests hadoop.hdfs.server.blockmanagement.TestNodeCount Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12753921/HDFS-8964.06.patch Optional Tests javac unit findbugs checkstyle javadoc git revision trunk / 09c64ba hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12273/artifact/patchprocess/testrun_hadoop-hdfs.txt bkjournal test log https://builds.apache.org/job/PreCommit-HDFS-Build/12273/artifact/patchprocess/testrun_bkjournal.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12273/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12273/console This message was automatically generated.
        Hide
        cmccabe Colin P. McCabe added a comment -

        +1. Thanks, Zhe Zhang.

        Show
        cmccabe Colin P. McCabe added a comment - +1. Thanks, Zhe Zhang .
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8399 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8399/)
        HDFS-8964. When validating the edit log, do not read at or beyond the file offset that is being written (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 53c38cc89ab979ec47557dcfa7affbad20578c0a)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckPointForSecurityTokens.java
        • hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8399 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8399/ ) HDFS-8964 . When validating the edit log, do not read at or beyond the file offset that is being written (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 53c38cc89ab979ec47557dcfa7affbad20578c0a) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckPointForSecurityTokens.java hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        zhz Zhe Zhang added a comment -

        Thanks Colin for reviewing!

        Show
        zhz Zhe Zhang added a comment - Thanks Colin for reviewing!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #348 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/348/)
        HDFS-8964. When validating the edit log, do not read at or beyond the file offset that is being written (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 53c38cc89ab979ec47557dcfa7affbad20578c0a)

        • hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckPointForSecurityTokens.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #348 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/348/ ) HDFS-8964 . When validating the edit log, do not read at or beyond the file offset that is being written (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 53c38cc89ab979ec47557dcfa7affbad20578c0a) hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckPointForSecurityTokens.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #342 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/342/)
        HDFS-8964. When validating the edit log, do not read at or beyond the file offset that is being written (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 53c38cc89ab979ec47557dcfa7affbad20578c0a)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckPointForSecurityTokens.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #342 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/342/ ) HDFS-8964 . When validating the edit log, do not read at or beyond the file offset that is being written (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 53c38cc89ab979ec47557dcfa7affbad20578c0a) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckPointForSecurityTokens.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1079 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1079/)
        HDFS-8964. When validating the edit log, do not read at or beyond the file offset that is being written (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 53c38cc89ab979ec47557dcfa7affbad20578c0a)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckPointForSecurityTokens.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1079 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1079/ ) HDFS-8964 . When validating the edit log, do not read at or beyond the file offset that is being written (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 53c38cc89ab979ec47557dcfa7affbad20578c0a) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckPointForSecurityTokens.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2290 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2290/)
        HDFS-8964. When validating the edit log, do not read at or beyond the file offset that is being written (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 53c38cc89ab979ec47557dcfa7affbad20578c0a)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckPointForSecurityTokens.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2290 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2290/ ) HDFS-8964 . When validating the edit log, do not read at or beyond the file offset that is being written (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 53c38cc89ab979ec47557dcfa7affbad20578c0a) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckPointForSecurityTokens.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #331 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/331/)
        HDFS-8964. When validating the edit log, do not read at or beyond the file offset that is being written (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 53c38cc89ab979ec47557dcfa7affbad20578c0a)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java
        • hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckPointForSecurityTokens.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #331 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/331/ ) HDFS-8964 . When validating the edit log, do not read at or beyond the file offset that is being written (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 53c38cc89ab979ec47557dcfa7affbad20578c0a) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckPointForSecurityTokens.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #2269 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2269/)
        HDFS-8964. When validating the edit log, do not read at or beyond the file offset that is being written (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 53c38cc89ab979ec47557dcfa7affbad20578c0a)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        • hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckPointForSecurityTokens.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2269 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2269/ ) HDFS-8964 . When validating the edit log, do not read at or beyond the file offset that is being written (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 53c38cc89ab979ec47557dcfa7affbad20578c0a) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckPointForSecurityTokens.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java

          People

          • Assignee:
            zhz Zhe Zhang
            Reporter:
            zhz Zhe Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development