Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7501

TransactionsSinceLastCheckpoint can be negative on SBNs

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed a bug where the StandbyNameNode's TransactionsSinceLastCheckpoint metric may slide into a negative number after every subsequent checkpoint.

      Description

      The metric TransactionsSinceLastCheckpoint is derived as FSEditLog.txid minus NNStorage.mostRecentCheckpointTxId.

      In Standby mode, the former does not increment beyond the loaded or last-when-active value, but the latter does change due to checkpoints done regularly in this mode. Thereby, the SBN will eventually end up showing negative values for TransactionsSinceLastCheckpoint.

      This is not an issue as the metric only makes sense to be monitored on the Active NameNode, but we should perhaps just show the value 0 by detecting if the NN is in SBN form, as allowing a negative number is confusing to view within a chart that tracks it.

      1. HDFS-7501-3.patch
        7 kB
        Harsh J
      2. HDFS-7501-2.patch
        6 kB
        Gautam Gopalakrishnan
      3. HDFS-7501.patch
        3 kB
        Gautam Gopalakrishnan

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2079 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2079/)
          HDFS-7501. TransactionsSinceLastCheckpoint can be negative on SBNs. Contributed by Gautam Gopalakrishnan. (harsh: rev 3d9132d434c39e9b6e142e5cf9fd7a8afa4190a6)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2079 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2079/ ) HDFS-7501 . TransactionsSinceLastCheckpoint can be negative on SBNs. Contributed by Gautam Gopalakrishnan. (harsh: rev 3d9132d434c39e9b6e142e5cf9fd7a8afa4190a6) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2097 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2097/)
          HDFS-7501. TransactionsSinceLastCheckpoint can be negative on SBNs. Contributed by Gautam Gopalakrishnan. (harsh: rev 3d9132d434c39e9b6e142e5cf9fd7a8afa4190a6)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2097 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2097/ ) HDFS-7501 . TransactionsSinceLastCheckpoint can be negative on SBNs. Contributed by Gautam Gopalakrishnan. (harsh: rev 3d9132d434c39e9b6e142e5cf9fd7a8afa4190a6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #147 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/147/)
          HDFS-7501. TransactionsSinceLastCheckpoint can be negative on SBNs. Contributed by Gautam Gopalakrishnan. (harsh: rev 3d9132d434c39e9b6e142e5cf9fd7a8afa4190a6)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #147 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/147/ ) HDFS-7501 . TransactionsSinceLastCheckpoint can be negative on SBNs. Contributed by Gautam Gopalakrishnan. (harsh: rev 3d9132d434c39e9b6e142e5cf9fd7a8afa4190a6) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #138 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/138/)
          HDFS-7501. TransactionsSinceLastCheckpoint can be negative on SBNs. Contributed by Gautam Gopalakrishnan. (harsh: rev 3d9132d434c39e9b6e142e5cf9fd7a8afa4190a6)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #138 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/138/ ) HDFS-7501 . TransactionsSinceLastCheckpoint can be negative on SBNs. Contributed by Gautam Gopalakrishnan. (harsh: rev 3d9132d434c39e9b6e142e5cf9fd7a8afa4190a6) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #147 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/147/)
          HDFS-7501. TransactionsSinceLastCheckpoint can be negative on SBNs. Contributed by Gautam Gopalakrishnan. (harsh: rev 3d9132d434c39e9b6e142e5cf9fd7a8afa4190a6)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #147 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/147/ ) HDFS-7501 . TransactionsSinceLastCheckpoint can be negative on SBNs. Contributed by Gautam Gopalakrishnan. (harsh: rev 3d9132d434c39e9b6e142e5cf9fd7a8afa4190a6) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #881 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/881/)
          HDFS-7501. TransactionsSinceLastCheckpoint can be negative on SBNs. Contributed by Gautam Gopalakrishnan. (harsh: rev 3d9132d434c39e9b6e142e5cf9fd7a8afa4190a6)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #881 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/881/ ) HDFS-7501 . TransactionsSinceLastCheckpoint can be negative on SBNs. Contributed by Gautam Gopalakrishnan. (harsh: rev 3d9132d434c39e9b6e142e5cf9fd7a8afa4190a6) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7454 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7454/)
          HDFS-7501. TransactionsSinceLastCheckpoint can be negative on SBNs. Contributed by Gautam Gopalakrishnan. (harsh: rev 3d9132d434c39e9b6e142e5cf9fd7a8afa4190a6)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7454 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7454/ ) HDFS-7501 . TransactionsSinceLastCheckpoint can be negative on SBNs. Contributed by Gautam Gopalakrishnan. (harsh: rev 3d9132d434c39e9b6e142e5cf9fd7a8afa4190a6) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          qwertymaniac Harsh J added a comment -

          Thanks Akira Ajisaka for the additional review! Thanks Gautam for the fix and tests!

          Pushed to branch-2 and trunk.

          Show
          qwertymaniac Harsh J added a comment - Thanks Akira Ajisaka for the additional review! Thanks Gautam for the fix and tests! Pushed to branch-2 and trunk.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Looks good to me, +1. Thanks Harsh J and Gautam Gopalakrishnan.

          Show
          ajisakaa Akira Ajisaka added a comment - Looks good to me, +1. Thanks Harsh J and Gautam Gopalakrishnan .
          Hide
          qwertymaniac Harsh J added a comment -

          The TestDFSClientRetries failure seems to be a test timeout w.r.t. starting some mini cluster internally. It is unrelated to this fix/change.

          Daryn Sharp - Can you take a look at this again? This should resolve your raised point earlier.

          Show
          qwertymaniac Harsh J added a comment - The TestDFSClientRetries failure seems to be a test timeout w.r.t. starting some mini cluster internally. It is unrelated to this fix/change. Daryn Sharp - Can you take a look at this again? This should resolve your raised point earlier.
          Hide
          ggop Gautam Gopalakrishnan added a comment -

          Thanks Harsh, sorry I couldn't get to this earlier.

          Show
          ggop Gautam Gopalakrishnan added a comment - Thanks Harsh, sorry I couldn't get to this earlier.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12706977/HDFS-7501-3.patch
          against trunk revision a16bfff.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestDFSClientRetries
          org.apache.hadoop.tracing.TestTracing

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10054//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10054//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12706977/HDFS-7501-3.patch against trunk revision a16bfff. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestDFSClientRetries org.apache.hadoop.tracing.TestTracing Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10054//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10054//console This message is automatically generated.
          Hide
          qwertymaniac Harsh J added a comment -

          I went ahead and modified the patch with the suggested change above, and also added some indent cleanup + comments to the great test written there (aside of the fix to the test as required for the new changes).

          New test fails as expected without the fix, and passes all 3 criterion with the fix.

          Attaching and marking as Patch Available to run full module tests via jenkins.

          Show
          qwertymaniac Harsh J added a comment - I went ahead and modified the patch with the suggested change above, and also added some indent cleanup + comments to the great test written there (aside of the fix to the test as required for the new changes). New test fails as expected without the fix, and passes all 3 criterion with the fix. Attaching and marking as Patch Available to run full module tests via jenkins.
          Hide
          qwertymaniac Harsh J added a comment -

          Hey Gautam Gopalakrishnan, can you switch the method to simply make use of FSImage.getLastAppliedOrWrittenTxId(…) method? That should work just fine for both ANN and SBN modes, as it appears to track both written mode edits and tailed mode edits.

          The test can be modified to be a test of "> 0" as opposed to "= 0" as we previously targeted.

          Show
          qwertymaniac Harsh J added a comment - Hey Gautam Gopalakrishnan , can you switch the method to simply make use of FSImage.getLastAppliedOrWrittenTxId(…) method? That should work just fine for both ANN and SBN modes, as it appears to track both written mode edits and tailed mode edits. The test can be modified to be a test of "> 0" as opposed to "= 0" as we previously targeted.
          Hide
          qwertymaniac Harsh J added a comment -

          Daryn Sharp - Just wanted to check if exposing out and using the lastLoadedTxnId from the EditLogTailer instead in StandbyNN mode would be OK to do instead?

          Show
          qwertymaniac Harsh J added a comment - Daryn Sharp - Just wanted to check if exposing out and using the lastLoadedTxnId from the EditLogTailer instead in StandbyNN mode would be OK to do instead?
          Hide
          qwertymaniac Harsh J added a comment -

          Daryn Sharp - The metric goes negative at the standby after the first checkpoint and continues going negative until it is in active mode again. The reason is that the getEditLog().getLastWrittenTxId() freezes in standby mode where no local edit logs are written anymore, and only the edit log tailer has the txid tracking info. We could switch to querying that - would that make sense to do just when we are in standby mode.

          We could expose lastLoadedTxnId in the EditLogTailer, for example.

          Sorry on delay in responding.

          Show
          qwertymaniac Harsh J added a comment - Daryn Sharp - The metric goes negative at the standby after the first checkpoint and continues going negative until it is in active mode again. The reason is that the getEditLog().getLastWrittenTxId() freezes in standby mode where no local edit logs are written anymore, and only the edit log tailer has the txid tracking info. We could switch to querying that - would that make sense to do just when we are in standby mode. We could expose lastLoadedTxnId in the EditLogTailer, for example. Sorry on delay in responding.
          Hide
          daryn Daryn Sharp added a comment -

          Here's the use case I have in mind. Let's say my cluster looks fine. However the active thinks there are 1 million txns since checkpoint. The standby thinks there are only 100 hundred because of a problem propagating edits. I'd like to know that rather than see 0 on the standby.

          I quickly glanced at the existing code. When does the metric go negative? Only briefly during the actual checkpoint operation? Is it when the edit log is null in the image?

          Show
          daryn Daryn Sharp added a comment - Here's the use case I have in mind. Let's say my cluster looks fine. However the active thinks there are 1 million txns since checkpoint. The standby thinks there are only 100 hundred because of a problem propagating edits. I'd like to know that rather than see 0 on the standby. I quickly glanced at the existing code. When does the metric go negative? Only briefly during the actual checkpoint operation? Is it when the edit log is null in the image?
          Hide
          ggop Gautam Gopalakrishnan added a comment -

          Daryn Sharp any thoughts?

          Show
          ggop Gautam Gopalakrishnan added a comment - Daryn Sharp any thoughts?
          Hide
          qwertymaniac Harsh J added a comment -

          Daryn Sharp - Won't the metric lag at the StandBy even if we were to correct things up (for that metric) during checkpoints? Is a laggy metric OK to display (better than negatives, but still)?

          Show
          qwertymaniac Harsh J added a comment - Daryn Sharp - Won't the metric lag at the StandBy even if we were to correct things up (for that metric) during checkpoints? Is a laggy metric OK to display (better than negatives, but still)?
          Hide
          daryn Daryn Sharp added a comment -

          I don't agree with returning a hardcoded 0 on the standby. I'd like to see the correct metric returned on both active and standby.

          Show
          daryn Daryn Sharp added a comment - I don't agree with returning a hardcoded 0 on the standby. I'd like to see the correct metric returned on both active and standby.
          Hide
          ggop Gautam Gopalakrishnan added a comment -

          The warning is triggered from code outside the patch.

          Exceptional return value of java.io.File.delete() ignored in org.apache.hadoop.hdfs.server.namenode.TransferFsImage.deleteTmpFiles(List)
          
          Bug type RV_RETURN_VALUE_IGNORED_BAD_PRACTICE (click for details)
          In class org.apache.hadoop.hdfs.server.namenode.TransferFsImage
          In method org.apache.hadoop.hdfs.server.namenode.TransferFsImage.deleteTmpFiles(List)
          Called method java.io.File.delete()
          At TransferFsImage.java:[line 577]
          
          Show
          ggop Gautam Gopalakrishnan added a comment - The warning is triggered from code outside the patch. Exceptional return value of java.io.File.delete() ignored in org.apache.hadoop.hdfs.server.namenode.TransferFsImage.deleteTmpFiles(List) Bug type RV_RETURN_VALUE_IGNORED_BAD_PRACTICE (click for details) In class org.apache.hadoop.hdfs.server.namenode.TransferFsImage In method org.apache.hadoop.hdfs.server.namenode.TransferFsImage.deleteTmpFiles(List) Called method java.io.File.delete() At TransferFsImage.java:[line 577]
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12689917/HDFS-7501-2.patch
          against trunk revision 947578c.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9139//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9139//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9139//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12689917/HDFS-7501-2.patch against trunk revision 947578c. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9139//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9139//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9139//console This message is automatically generated.
          Hide
          ggop Gautam Gopalakrishnan added a comment -

          Improved the test case.

          Show
          ggop Gautam Gopalakrishnan added a comment - Improved the test case.
          Hide
          qwertymaniac Harsh J added a comment -

          Resetting state awaiting improved test.

          Show
          qwertymaniac Harsh J added a comment - Resetting state awaiting improved test.
          Hide
          qwertymaniac Harsh J added a comment -

          Failing test appears unrelated to change proposed here.

          Show
          qwertymaniac Harsh J added a comment - Failing test appears unrelated to change proposed here.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12687689/HDFS-7501.patch
          against trunk revision e996a1b.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9057//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9057//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12687689/HDFS-7501.patch against trunk revision e996a1b. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9057//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9057//console This message is automatically generated.
          Hide
          qwertymaniac Harsh J added a comment -

          Thanks Gautam, patch looks mostly good and many thanks for writing a test as well! I think the test could be improved a bit further to be a regression-catching one for future, by doing something like:

          1. Run NNs, preferably with low checkpoint transactions limit threshold
          2. Make enough transactions to make it ready for checkpoint
          3. Invoke or wait for a checkpoint on SBN
          4. Measure metric for an assertion of not < 0 (this assertion will/must fail without the fix)

          Many thanks again!

          Show
          qwertymaniac Harsh J added a comment - Thanks Gautam, patch looks mostly good and many thanks for writing a test as well! I think the test could be improved a bit further to be a regression-catching one for future, by doing something like: 1. Run NNs, preferably with low checkpoint transactions limit threshold 2. Make enough transactions to make it ready for checkpoint 3. Invoke or wait for a checkpoint on SBN 4. Measure metric for an assertion of not < 0 (this assertion will/must fail without the fix) Many thanks again!

            People

            • Assignee:
              ggop Gautam Gopalakrishnan
              Reporter:
              qwertymaniac Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development