Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3391

TestPipelinesFailover#testLeaseRecoveryAfterFailover is failing

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Running org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
      Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.208 sec <<< FAILURE!

      Running org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
      Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 81.195 sec <<< FAILURE!

      1. hdfs-3391.txt
        12 kB
        Todd Lipcon
      2. hdfs-3391.txt
        13 kB
        Todd Lipcon
      3. hdfs-3391.txt
        14 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          Are you seeing these fail regularly? I haven't seen either fail in recent months.

          Show
          Todd Lipcon added a comment - Are you seeing these fail regularly? I haven't seen either fail in recent months.
          Hide
          Eli Collins added a comment -

          These are both of these are failing due to java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.addINode(Lorg/apache/hadoop/hdfs/server/blockmanagement/BlockInfo;Lorg/apache/hadoop/hdfs/server/namenode/INodeFile;)Lorg/apache/hadoop/hdfs/server/blockmanagement/BlockInfo; at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:349)

          Nicholas, John - due to HDFS-3363?

          Show
          Eli Collins added a comment - These are both of these are failing due to java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.addINode(Lorg/apache/hadoop/hdfs/server/blockmanagement/BlockInfo;Lorg/apache/hadoop/hdfs/server/namenode/INodeFile;)Lorg/apache/hadoop/hdfs/server/blockmanagement/BlockInfo; at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:349) Nicholas, John - due to HDFS-3363 ?
          Hide
          Eli Collins added a comment -

          I think this is a stale jar in mvn, re-running after mvn clean works and there is such a method.

          Show
          Eli Collins added a comment - I think this is a stale jar in mvn, re-running after mvn clean works and there is such a method.
          Hide
          Eli Collins added a comment -

          After mvn clean both tests pass for me, and this error looks like a stale jar. Closing.

          Show
          Eli Collins added a comment - After mvn clean both tests pass for me, and this error looks like a stale jar. Closing.
          Hide
          Arun C Murthy added a comment -

          TestRBWBlockInvalidation fails consistently with:

          Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.827 sec <<< FAILURE!
          testBlockInvalidationWhenRBWReplicaMissedInDN(org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation)  Time elapsed: 6.725 sec  <<< FAILURE!
          java.lang.AssertionError: There should be three live replicas expected:<3> but was:<2>
            at org.junit.Assert.fail(Assert.java:91)
          

          TestPipelinesFailover fails sporadically.

          Show
          Arun C Murthy added a comment - TestRBWBlockInvalidation fails consistently with: Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.827 sec <<< FAILURE! testBlockInvalidationWhenRBWReplicaMissedInDN(org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation) Time elapsed: 6.725 sec <<< FAILURE! java.lang.AssertionError: There should be three live replicas expected:<3> but was:<2> at org.junit.Assert.fail(Assert.java:91) TestPipelinesFailover fails sporadically.
          Hide
          Eli Collins added a comment -

          testBlockInvalidationWhenRBWReplicaMissedInDN passes consistently for me in a fresh branch-2 tree. Does your tree have HDFS-3157?

          I looped TestPipelinesFailover and sometimes it fails with:

          testLeaseRecoveryAfterFailover(org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover)  Time elapsed: 18.19 sec  <<< ERROR!
          java.io.IOException: p=/test-file, length=6144, i=4096
                  at org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:135)
                  at org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testLeaseRecoveryAfterFailover(TestPipelinesFailover.java:250)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                  at java.lang.reflect.Method.invoke(Method.java:597)
                  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
                  at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
                  at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
                  at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
                  at org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
          Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1782491480-192.168.1.113-1336537917180:blk_-3060921670577866018_1006 file=/test-file
                  at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:644)
                  at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:437)
                  at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:577)
                  at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:500)
                  at java.io.FilterInputStream.read(FilterInputStream.java:66)
                  at org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129)
                  ... 10 more
          
          Show
          Eli Collins added a comment - testBlockInvalidationWhenRBWReplicaMissedInDN passes consistently for me in a fresh branch-2 tree. Does your tree have HDFS-3157 ? I looped TestPipelinesFailover and sometimes it fails with: testLeaseRecoveryAfterFailover(org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover) Time elapsed: 18.19 sec <<< ERROR! java.io.IOException: p=/test-file, length=6144, i=4096 at org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:135) at org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testLeaseRecoveryAfterFailover(TestPipelinesFailover.java:250) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28) Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1782491480-192.168.1.113-1336537917180:blk_-3060921670577866018_1006 file=/test-file at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:644) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:437) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:577) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:500) at java.io.FilterInputStream.read(FilterInputStream.java:66) at org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) ... 10 more
          Hide
          Arun C Murthy added a comment -

          testBlockInvalidationWhenRBWReplicaMissedInDN passes consistently for me in a fresh branch-2 tree. Does your tree have HDFS-3157?

          Yep, I do have HDFS-3157.

          Can you pls check on branch-2.0.0-alpha? Tx.

          Show
          Arun C Murthy added a comment - testBlockInvalidationWhenRBWReplicaMissedInDN passes consistently for me in a fresh branch-2 tree. Does your tree have HDFS-3157 ? Yep, I do have HDFS-3157 . Can you pls check on branch-2.0.0-alpha? Tx.
          Hide
          Eli Collins added a comment -

          Passes for me consistently in a fresh branch-2.0.0-alpha tree. Trying TestPipelinesFailover now.

          Show
          Eli Collins added a comment - Passes for me consistently in a fresh branch-2.0.0-alpha tree. Trying TestPipelinesFailover now.
          Hide
          Eli Collins added a comment -

          Same failures with TestPipelinesFailover on branch-2.0.0-alpha.

          Show
          Eli Collins added a comment - Same failures with TestPipelinesFailover on branch-2.0.0-alpha.
          Hide
          John George added a comment -

          TestPipelinesFailover passes for me consistently on trunk - ran 5 times. Checking branch-2.0.0-alpha. Ideally failures related to HDFS-3363 would be more consistent since it was mostly name change and package location change.

          Show
          John George added a comment - TestPipelinesFailover passes for me consistently on trunk - ran 5 times. Checking branch-2.0.0-alpha. Ideally failures related to HDFS-3363 would be more consistent since it was mostly name change and package location change.
          Hide
          John George added a comment -

          I was wrong about trunk - I had not updated to the latest trunk and once I did, I could reproduce this consistently. It is easily reproducible on branch-2.0.0-alpha as well.
          I have closed it down to the changes in HDFS-3157. Without HDFS-3157, the test passes consistently and with the patch, it fails. I am going to unassign myself from the JIRA and going to ping HDFS-3157

          Show
          John George added a comment - I was wrong about trunk - I had not updated to the latest trunk and once I did, I could reproduce this consistently. It is easily reproducible on branch-2.0.0-alpha as well. I have closed it down to the changes in HDFS-3157 . Without HDFS-3157 , the test passes consistently and with the patch, it fails. I am going to unassign myself from the JIRA and going to ping HDFS-3157
          Hide
          Tsz Wo Nicholas Sze added a comment -

          For TestPipelinesFailover, there are also many "THIS IS NOT SUPPOSED TO HAPPEN" as in build #2397.

          2012-05-09 23:50:32,074 INFO  ipc.Server (Server.java:run(1711)) - IPC Server handler 2 on 46044,
           call org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol.initReplicaRecovery from 127.0.0.1:33818:
           error: java.io.IOException: THIS IS NOT SUPPOSED TO HAPPEN: replica.getGenerationStamp() >= recoveryId = 1006,
           block=blk_-3039116449792967513_1003, replica=FinalizedReplica, blk_-3039116449792967513_1006, FINALIZED
            getNumBytes()     = 2048
            getBytesOnDisk()  = 2048
            getVisibleLength()= 2048
            getVolume()       = /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2/current
            getBlockFile()    = /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2/current/BP-1244283508-67.195.138.20-1336607428359/current/finalized/blk_-3039116449792967513
            unlinked          =false
          
          Show
          Tsz Wo Nicholas Sze added a comment - For TestPipelinesFailover, there are also many "THIS IS NOT SUPPOSED TO HAPPEN" as in build #2397 . 2012-05-09 23:50:32,074 INFO ipc.Server (Server.java:run(1711)) - IPC Server handler 2 on 46044, call org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol.initReplicaRecovery from 127.0.0.1:33818: error: java.io.IOException: THIS IS NOT SUPPOSED TO HAPPEN: replica.getGenerationStamp() >= recoveryId = 1006, block=blk_-3039116449792967513_1003, replica=FinalizedReplica, blk_-3039116449792967513_1006, FINALIZED getNumBytes() = 2048 getBytesOnDisk() = 2048 getVisibleLength()= 2048 getVolume() = /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2/current getBlockFile() = /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2/current/BP-1244283508-67.195.138.20-1336607428359/current/finalized/blk_-3039116449792967513 unlinked =false
          Hide
          Tsz Wo Nicholas Sze added a comment -

          HDFS-3157 is reverted. So TestRBWBlockInvalidation is no longer a problem.

          Not sure if TestPipelinesFailover is related to HDFS-3157. Could any of you try to reproduce the failure? I actually never see it failing in my machine.

          Show
          Tsz Wo Nicholas Sze added a comment - HDFS-3157 is reverted. So TestRBWBlockInvalidation is no longer a problem. Not sure if TestPipelinesFailover is related to HDFS-3157 . Could any of you try to reproduce the failure? I actually never see it failing in my machine.
          Hide
          Eli Collins added a comment -

          Yea, TestPipelinesFailover fails for me when I loop it, eg on the 3rd or 4th iteration.

          Show
          Eli Collins added a comment - Yea, TestPipelinesFailover fails for me when I loop it, eg on the 3rd or 4th iteration.
          Hide
          Todd Lipcon added a comment -

          I'll investigate TestPipelinesFailover, since I wrote it.

          Show
          Todd Lipcon added a comment - I'll investigate TestPipelinesFailover, since I wrote it.
          Hide
          Eli Collins added a comment -

          Forgot to mention, I only see TestPipelinesFailover fail on branch-2-alpha, not trunk.

          Show
          Eli Collins added a comment - Forgot to mention, I only see TestPipelinesFailover fail on branch-2-alpha, not trunk.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          It does fail in trunk as in build #2397. The error is the same as Eli got.

          Show
          Tsz Wo Nicholas Sze added a comment - It does fail in trunk as in build #2397 . The error is the same as Eli got.
          Hide
          Todd Lipcon added a comment -

          I looped TestPipelinesFailover for quite some time and could not get a failure. In the logs you pointed to on build #2397, I traced the issue to the following:

          2012-05-09 23:50:33,074 DEBUG namenode.FSNamesystem (FSEditLogLoader.java:applyEditLogOp(296)) - OP_CLOSE: /test-file numblocks : 2 clientHolder  clientMachine
          2012-05-09 23:50:33,074 DEBUG blockmanagement.BlockManager (BlockManager.java:processQueuedMessages(1760)) - Processing previouly queued message ReportedBlockInfo [block=blk_-3039116449792967513_1005, dn=127.0.0.1:45674, reportedState=FINALIZED]
          2012-05-09 23:50:33,074 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1660)) - Reported block blk_-3039116449792967513_1005 on 127.0.0.1:45674 size 2048 replicaState = FINALIZED
          2012-05-09 23:50:33,074 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1684)) - In memory blockUCState = COMPLETE
          2012-05-09 23:50:33,075 INFO  hdfs.StateChange (BlockManager.java:markBlockAsCorrupt(926)) - BLOCK markBlockAsCorrupt: block blk_-3039116449792967513_1005 could not be marked as corrupt as it does not belong to any file
          2012-05-09 23:50:33,076 INFO  hdfs.StateChange (InvalidateBlocks.java:add(77)) - BLOCK* InvalidateBlocks: add blk_-3039116449792967513_1005 to 127.0.0.1:45674
          2012-05-09 23:50:33,076 DEBUG blockmanagement.BlockManager (BlockManager.java:processQueuedMessages(1760)) - Processing previouly queued message ReportedBlockInfo [block=blk_-3039116449792967513_1005, dn=127.0.0.1:35659, reportedState=FINALIZED]
          2012-05-09 23:50:33,076 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1660)) - Reported block blk_-3039116449792967513_1005 on 127.0.0.1:35659 size 2048 replicaState = FINALIZED
          2012-05-09 23:50:33,076 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1684)) - In memory blockUCState = COMPLETE
          2012-05-09 23:50:33,077 INFO  hdfs.StateChange (BlockManager.java:markBlockAsCorrupt(926)) - BLOCK markBlockAsCorrupt: block blk_-3039116449792967513_1005 could not be marked as corrupt as it does not belong to any file
          2012-05-09 23:50:33,077 INFO  hdfs.StateChange (InvalidateBlocks.java:add(77)) - BLOCK* InvalidateBlocks: add blk_-3039116449792967513_1005 to 127.0.0.1:35659
          2012-05-09 23:50:33,077 DEBUG blockmanagement.BlockManager (BlockManager.java:processQueuedMessages(1760)) - Processing previouly queued message ReportedBlockInfo [block=blk_-3039116449792967513_1005, dn=127.0.0.1:59499, reportedState=FINALIZED]
          2012-05-09 23:50:33,077 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1660)) - Reported block blk_-3039116449792967513_1005 on 127.0.0.1:59499 size 2048 replicaState = FINALIZED
          2012-05-09 23:50:33,078 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1684)) - In memory blockUCState = COMPLETE
          2012-05-09 23:50:33,078 INFO  hdfs.StateChange (BlockManager.java:markBlockAsCorrupt(926)) - BLOCK markBlockAsCorrupt: block blk_-3039116449792967513_1005 could not be marked as corrupt as it does not belong to any file
          2012-05-09 23:50:33,078 INFO  hdfs.StateChange (InvalidateBlocks.java:add(77)) - BLOCK* InvalidateBlocks: add blk_-3039116449792967513_1005 to 127.0.0.1:59499
          2012-05-09 23:50:33,078 DEBUG blockmanagement.BlockManager (BlockManager.java:processQueuedMessages(1760)) - Processing previouly queued message ReportedBlockInfo [block=blk_-3039116449792967513_1006, dn=127.0.0.1:45674, reportedState=FINALIZED]
          2012-05-09 23:50:33,079 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1660)) - Reported block blk_-3039116449792967513_1006 on 127.0.0.1:45674 size 2048 replicaState = FINALIZED
          2012-05-09 23:50:33,079 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1684)) - In memory blockUCState = COMPLETE
          2012-05-09 23:50:33,079 DEBUG blockmanagement.BlockManager (BlockManager.java:processQueuedMessages(1760)) - Processing previouly queued message ReportedBlockInfo [block=blk_-3039116449792967513_1006, dn=127.0.0.1:59499, reportedState=FINALIZED]
          2012-05-09 23:50:33,079 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1660)) - Reported block blk_-3039116449792967513_1006 on 127.0.0.1:59499 size 2048 replicaState = FINALIZED
          2012-05-09 23:50:33,080 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1684)) - In memory blockUCState = COMPLETE
          2012-05-09 23:50:33,080 DEBUG blockmanagement.BlockManager (BlockManager.java:processQueuedMessages(1760)) - Processing previouly queued message ReportedBlockInfo [block=blk_-3039116449792967513_1006, dn=127.0.0.1:35659, reportedState=FINALIZED]
          2012-05-09 23:50:33,080 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1660)) - Reported block blk_-3039116449792967513_1006 on 127.0.0.1:35659 size 2048 replicaState = FINALIZED
          2012-05-09 23:50:33,080 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1684)) - In memory blockUCState = COMPLETE
          

          In this run, the block synchronization ended up bumping the genstamp twice (once to 1005 and then again to 1006). When the standby is becoming active, it replays first the BlockReceived for 1005 (out of date genstamp) and then the BlockReceived for 1006 (which should succeed in adding it to the block map). But something fishy happens: it doesn't find that block in the block map at all.

          Given that HDFS-3157 was committed when build #2397 ran (it ran at r1336399 which was before the revert), and HDFS-3157 touches exactly those lines of code which decide whether to mark out-of-date genstamp blocks as corrupt, I really think it was related.

          I will try reapplying HDFS-3157 locally and seeing if I can repro the issue.

          Show
          Todd Lipcon added a comment - I looped TestPipelinesFailover for quite some time and could not get a failure. In the logs you pointed to on build #2397, I traced the issue to the following: 2012-05-09 23:50:33,074 DEBUG namenode.FSNamesystem (FSEditLogLoader.java:applyEditLogOp(296)) - OP_CLOSE: /test-file numblocks : 2 clientHolder clientMachine 2012-05-09 23:50:33,074 DEBUG blockmanagement.BlockManager (BlockManager.java:processQueuedMessages(1760)) - Processing previouly queued message ReportedBlockInfo [block=blk_-3039116449792967513_1005, dn=127.0.0.1:45674, reportedState=FINALIZED] 2012-05-09 23:50:33,074 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1660)) - Reported block blk_-3039116449792967513_1005 on 127.0.0.1:45674 size 2048 replicaState = FINALIZED 2012-05-09 23:50:33,074 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1684)) - In memory blockUCState = COMPLETE 2012-05-09 23:50:33,075 INFO hdfs.StateChange (BlockManager.java:markBlockAsCorrupt(926)) - BLOCK markBlockAsCorrupt: block blk_-3039116449792967513_1005 could not be marked as corrupt as it does not belong to any file 2012-05-09 23:50:33,076 INFO hdfs.StateChange (InvalidateBlocks.java:add(77)) - BLOCK* InvalidateBlocks: add blk_-3039116449792967513_1005 to 127.0.0.1:45674 2012-05-09 23:50:33,076 DEBUG blockmanagement.BlockManager (BlockManager.java:processQueuedMessages(1760)) - Processing previouly queued message ReportedBlockInfo [block=blk_-3039116449792967513_1005, dn=127.0.0.1:35659, reportedState=FINALIZED] 2012-05-09 23:50:33,076 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1660)) - Reported block blk_-3039116449792967513_1005 on 127.0.0.1:35659 size 2048 replicaState = FINALIZED 2012-05-09 23:50:33,076 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1684)) - In memory blockUCState = COMPLETE 2012-05-09 23:50:33,077 INFO hdfs.StateChange (BlockManager.java:markBlockAsCorrupt(926)) - BLOCK markBlockAsCorrupt: block blk_-3039116449792967513_1005 could not be marked as corrupt as it does not belong to any file 2012-05-09 23:50:33,077 INFO hdfs.StateChange (InvalidateBlocks.java:add(77)) - BLOCK* InvalidateBlocks: add blk_-3039116449792967513_1005 to 127.0.0.1:35659 2012-05-09 23:50:33,077 DEBUG blockmanagement.BlockManager (BlockManager.java:processQueuedMessages(1760)) - Processing previouly queued message ReportedBlockInfo [block=blk_-3039116449792967513_1005, dn=127.0.0.1:59499, reportedState=FINALIZED] 2012-05-09 23:50:33,077 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1660)) - Reported block blk_-3039116449792967513_1005 on 127.0.0.1:59499 size 2048 replicaState = FINALIZED 2012-05-09 23:50:33,078 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1684)) - In memory blockUCState = COMPLETE 2012-05-09 23:50:33,078 INFO hdfs.StateChange (BlockManager.java:markBlockAsCorrupt(926)) - BLOCK markBlockAsCorrupt: block blk_-3039116449792967513_1005 could not be marked as corrupt as it does not belong to any file 2012-05-09 23:50:33,078 INFO hdfs.StateChange (InvalidateBlocks.java:add(77)) - BLOCK* InvalidateBlocks: add blk_-3039116449792967513_1005 to 127.0.0.1:59499 2012-05-09 23:50:33,078 DEBUG blockmanagement.BlockManager (BlockManager.java:processQueuedMessages(1760)) - Processing previouly queued message ReportedBlockInfo [block=blk_-3039116449792967513_1006, dn=127.0.0.1:45674, reportedState=FINALIZED] 2012-05-09 23:50:33,079 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1660)) - Reported block blk_-3039116449792967513_1006 on 127.0.0.1:45674 size 2048 replicaState = FINALIZED 2012-05-09 23:50:33,079 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1684)) - In memory blockUCState = COMPLETE 2012-05-09 23:50:33,079 DEBUG blockmanagement.BlockManager (BlockManager.java:processQueuedMessages(1760)) - Processing previouly queued message ReportedBlockInfo [block=blk_-3039116449792967513_1006, dn=127.0.0.1:59499, reportedState=FINALIZED] 2012-05-09 23:50:33,079 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1660)) - Reported block blk_-3039116449792967513_1006 on 127.0.0.1:59499 size 2048 replicaState = FINALIZED 2012-05-09 23:50:33,080 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1684)) - In memory blockUCState = COMPLETE 2012-05-09 23:50:33,080 DEBUG blockmanagement.BlockManager (BlockManager.java:processQueuedMessages(1760)) - Processing previouly queued message ReportedBlockInfo [block=blk_-3039116449792967513_1006, dn=127.0.0.1:35659, reportedState=FINALIZED] 2012-05-09 23:50:33,080 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1660)) - Reported block blk_-3039116449792967513_1006 on 127.0.0.1:35659 size 2048 replicaState = FINALIZED 2012-05-09 23:50:33,080 DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(1684)) - In memory blockUCState = COMPLETE In this run, the block synchronization ended up bumping the genstamp twice (once to 1005 and then again to 1006). When the standby is becoming active, it replays first the BlockReceived for 1005 (out of date genstamp) and then the BlockReceived for 1006 (which should succeed in adding it to the block map). But something fishy happens: it doesn't find that block in the block map at all. Given that HDFS-3157 was committed when build #2397 ran (it ran at r1336399 which was before the revert), and HDFS-3157 touches exactly those lines of code which decide whether to mark out-of-date genstamp blocks as corrupt, I really think it was related. I will try reapplying HDFS-3157 locally and seeing if I can repro the issue.
          Hide
          Todd Lipcon added a comment -

          I was able to reproduce this by reapplying HDFS-3157 and adding the following in DataNode.java:

          --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          @@ -1983,6 +1983,13 @@ public class DataNode extends Configured
                 datanodes[i] = r.id;
                 storages[i] = r.storageID;
               }
          +    if (newBlock.getGenerationStamp() == 1005) {
          +      try {
          +        Thread.sleep(1500);
          +      } catch (InterruptedException ie) {
          +        Thread.currentThread().interrupt();
          +      }
          +    }
               nn.commitBlockSynchronization(block,
                   newBlock.getGenerationStamp(), newBlock.getNumBytes(), true, false,
                   datanodes, storages);
          

          I have to think through whether this is a bug which we've had for a while which is uncovered by HDFS-3157, or if HDFS-3157 itself was incorrect.

          Show
          Todd Lipcon added a comment - I was able to reproduce this by reapplying HDFS-3157 and adding the following in DataNode.java: --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java @@ -1983,6 +1983,13 @@ public class DataNode extends Configured datanodes[i] = r.id; storages[i] = r.storageID; } + if (newBlock.getGenerationStamp() == 1005) { + try { + Thread .sleep(1500); + } catch (InterruptedException ie) { + Thread .currentThread().interrupt(); + } + } nn.commitBlockSynchronization(block, newBlock.getGenerationStamp(), newBlock.getNumBytes(), true , false , datanodes, storages); I have to think through whether this is a bug which we've had for a while which is uncovered by HDFS-3157 , or if HDFS-3157 itself was incorrect.
          Hide
          Todd Lipcon added a comment -

          This attached patch seems to fix the issue, even with HDFS-3157 and the above troublesome sleep() call in place.

          I think what was happening here was the following:

          • in some cases, the block synchronization path can run twice, if the first attempt is slow. This ends up first finalizing the block at genstamp 1005, and then again at 1006 or 1007.
          • for each of those genstamps, the DNs report FINALIZED replicas to both NNs.
          • When the new NN becomes active, then, it replays the block reports – first FINALIZED for blk_N_1005, and then FINALIZED for blk_N_1006.
          • When it sees the blk_N_1005 genstamp, it already knows that 1006 is the "correct" latest genstamp for the block, so it wants to mark it as corrupt.

          Here is where the behavior differs:

          Prior to HDFS-3157, it was marking blk_N_1006 as corrupt instead of blk_N_1005. Thus the markBlockAsCorrupt() call would succeed. When processing the FINALIZED blk_N_1006, it would remove it from the corrupt list, and everything would be fine.

          With HDFS-3157 in place, it instead marks blk_N_1005 as corrupt. However, the BlockInfo object it creates to do so has no attached inode (BlockCollection in new parlance). So, markBlockAsCorrupt immediately enqueued the replica for invalidation, rather than treating it like a normal corrupt replica. Then, upon seeing the report of the blk_N_1006 FINALIZED replica, the check against invalidateBlocks.contains(block) caused it to be skipped, and thus addStoredBlock() didn't get called.

          The fix in this patch is to change invalidateBlocks so that its contains() call can check for genstamp match as well. So, even though blk_N_1005 has been enqueued for deletion, we should still accept a block report for blk_N_1006.

          Show
          Todd Lipcon added a comment - This attached patch seems to fix the issue, even with HDFS-3157 and the above troublesome sleep() call in place. I think what was happening here was the following: in some cases, the block synchronization path can run twice, if the first attempt is slow. This ends up first finalizing the block at genstamp 1005, and then again at 1006 or 1007. for each of those genstamps, the DNs report FINALIZED replicas to both NNs. When the new NN becomes active, then, it replays the block reports – first FINALIZED for blk_N_1005, and then FINALIZED for blk_N_1006. When it sees the blk_N_1005 genstamp, it already knows that 1006 is the "correct" latest genstamp for the block, so it wants to mark it as corrupt. Here is where the behavior differs: Prior to HDFS-3157 , it was marking blk_N_1006 as corrupt instead of blk_N_1005. Thus the markBlockAsCorrupt() call would succeed. When processing the FINALIZED blk_N_1006, it would remove it from the corrupt list, and everything would be fine. With HDFS-3157 in place, it instead marks blk_N_1005 as corrupt. However, the BlockInfo object it creates to do so has no attached inode (BlockCollection in new parlance). So, markBlockAsCorrupt immediately enqueued the replica for invalidation, rather than treating it like a normal corrupt replica. Then, upon seeing the report of the blk_N_1006 FINALIZED replica, the check against invalidateBlocks.contains(block) caused it to be skipped, and thus addStoredBlock() didn't get called. The fix in this patch is to change invalidateBlocks so that its contains() call can check for genstamp match as well. So, even though blk_N_1005 has been enqueued for deletion, we should still accept a block report for blk_N_1006.
          Hide
          Uma Maheswara Rao G added a comment -

          In one way HDFS-3157 is incorrectly handled. Because, It was creating new block info. But unfortunately new blockInfo ctor sets the inode as null. When we are marking it corrupt, that will just invalidate the blocks and will say block does not belongs any file. When we set the inode from storedBlock to newly created BlockInfo also doesn't help, strangely I have seen triplets does not contain that block info. Now it is able add to corrupt replicas, but nodeIterator for BlockMap does not have information about this block.

          2012-05-10 21:30:04,378 WARN  blockmanagement.BlockManager (BlockManager.java:createLocatedBlock(666)) - Inconsistent number of corrupt replicas for blk_-6411755644530997250_1003 blockMap has 0 but corrupt replicas map has 1
          2012-05-10 21:30:04,381 WARN  blockmanagement.BlockManager (BlockManager.java:createLocatedBlock(666)) - Inconsistent number of corrupt replicas for blk_-6411755644530997250_1003 blockMap has 0 but corrupt replicas map has 1

          Let me dig into it. Is there any other bug exist in this lines which we did not notice.

          Show
          Uma Maheswara Rao G added a comment - In one way HDFS-3157 is incorrectly handled. Because, It was creating new block info. But unfortunately new blockInfo ctor sets the inode as null. When we are marking it corrupt, that will just invalidate the blocks and will say block does not belongs any file. When we set the inode from storedBlock to newly created BlockInfo also doesn't help, strangely I have seen triplets does not contain that block info. Now it is able add to corrupt replicas, but nodeIterator for BlockMap does not have information about this block. 2012-05-10 21:30:04,378 WARN blockmanagement.BlockManager (BlockManager.java:createLocatedBlock(666)) - Inconsistent number of corrupt replicas for blk_-6411755644530997250_1003 blockMap has 0 but corrupt replicas map has 1 2012-05-10 21:30:04,381 WARN blockmanagement.BlockManager (BlockManager.java:createLocatedBlock(666)) - Inconsistent number of corrupt replicas for blk_-6411755644530997250_1003 blockMap has 0 but corrupt replicas map has 1 Let me dig into it. Is there any other bug exist in this lines which we did not notice.
          Hide
          Todd Lipcon added a comment -

          Hi Uma. I commented on HDFS-3157 as well, so let's continue that discussion there.

          On this JIRA let's discuss the improvement to InvalidateBlocks – I think this bug fix is a good improvement regardless of whether 3157 is in.

          Show
          Todd Lipcon added a comment - Hi Uma. I commented on HDFS-3157 as well, so let's continue that discussion there. On this JIRA let's discuss the improvement to InvalidateBlocks – I think this bug fix is a good improvement regardless of whether 3157 is in.
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks Todd, We can disscuss in HDFS-3157.

          Show
          Uma Maheswara Rao G added a comment - Thanks Todd, We can disscuss in HDFS-3157 .
          Hide
          Todd Lipcon added a comment -

          Tsz Wo Nicholas Sze - does this patch look reasonable to you?

          I will file another JIRA for the "THIS IS NOT SUPPOSED TO HAPPEN" warnings, which are as far as I can tell unrelated to the problem with InvalidateBlocks.

          Show
          Todd Lipcon added a comment - Tsz Wo Nicholas Sze - does this patch look reasonable to you? I will file another JIRA for the "THIS IS NOT SUPPOSED TO HAPPEN" warnings, which are as far as I can tell unrelated to the problem with InvalidateBlocks.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12526482/hdfs-3391.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2451//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526482/hdfs-3391.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2451//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Rebased on trunk

          Show
          Todd Lipcon added a comment - Rebased on trunk
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12527764/hdfs-3391.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2456//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2456//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527764/hdfs-3391.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2456//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2456//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Todd, thanks for posting a patch. I will review it later today.

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Todd, thanks for posting a patch. I will review it later today.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The patch looks good. Some comments:

          • All calls of invalidateBlocks.contains(..) have matchGenStamp == true. How about remove matchGenStamp from the parameters?
          • Remove the TODO below.
            -    if(invalidateBlocks.contains(dn.getStorageID(), block)) {
            +    if(invalidateBlocks.contains(dn.getStorageID(), block, true)) {
             /*  TODO: following assertion is incorrect, see HDFS-2668
             assert storedBlock.findDatanode(dn) < 0 : "Block " + block
                     + " in recentInvalidatesSet should not appear in DN " + dn; */
            
          • In LightWeightHashSet.getEqualElement(..), since the key will be cast to T, change the type to T and remove @SuppressWarnings("unchecked"). Then, we need to cast the key to T in contains(..). Add @Override to contains(..). Also, how about renaming getEqualElement to getElement?
          Show
          Tsz Wo Nicholas Sze added a comment - The patch looks good. Some comments: All calls of invalidateBlocks.contains(..) have matchGenStamp == true. How about remove matchGenStamp from the parameters? Remove the TODO below. - if (invalidateBlocks.contains(dn.getStorageID(), block)) { + if (invalidateBlocks.contains(dn.getStorageID(), block, true )) { /* TODO: following assertion is incorrect, see HDFS-2668 assert storedBlock.findDatanode(dn) < 0 : "Block " + block + " in recentInvalidatesSet should not appear in DN " + dn; */ In LightWeightHashSet.getEqualElement(..), since the key will be cast to T, change the type to T and remove @SuppressWarnings("unchecked"). Then, we need to cast the key to T in contains(..). Add @Override to contains(..). Also, how about renaming getEqualElement to getElement?
          Hide
          Todd Lipcon added a comment -

          Attached patch addresses Nicholas's comments above. The one I did not address was to remove the TODO that references HDFS-2668. Since that TODO is not addressed by this JIRA, I think it's better to address it in HDFS-2668 itself.

          Show
          Todd Lipcon added a comment - Attached patch addresses Nicholas's comments above. The one I did not address was to remove the TODO that references HDFS-2668 . Since that TODO is not addressed by this JIRA, I think it's better to address it in HDFS-2668 itself.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good. Thanks.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good. Thanks.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12527912/hdfs-3391.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2463//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2463//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527912/hdfs-3391.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2463//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2463//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Thanks Nicholas. The two javadoc warnings above are due to gridmix, so unrelated to this patch. I'll commit this momentarily.

          Show
          Todd Lipcon added a comment - Thanks Nicholas. The two javadoc warnings above are due to gridmix, so unrelated to this patch. I'll commit this momentarily.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2337 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2337/)
          HDFS-3391. Fix InvalidateBlocks to compare blocks including their generation stamps. Contributed by Todd Lipcon. (Revision 1339897)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339897
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightLinkedSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2337 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2337/ ) HDFS-3391 . Fix InvalidateBlocks to compare blocks including their generation stamps. Contributed by Todd Lipcon. (Revision 1339897) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339897 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightLinkedSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2264 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2264/)
          HDFS-3391. Fix InvalidateBlocks to compare blocks including their generation stamps. Contributed by Todd Lipcon. (Revision 1339897)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339897
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightLinkedSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2264 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2264/ ) HDFS-3391 . Fix InvalidateBlocks to compare blocks including their generation stamps. Contributed by Todd Lipcon. (Revision 1339897) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339897 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightLinkedSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2282 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2282/)
          HDFS-3391. Fix InvalidateBlocks to compare blocks including their generation stamps. Contributed by Todd Lipcon. (Revision 1339897)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339897
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightLinkedSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2282 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2282/ ) HDFS-3391 . Fix InvalidateBlocks to compare blocks including their generation stamps. Contributed by Todd Lipcon. (Revision 1339897) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339897 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightLinkedSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1049 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1049/)
          HDFS-3391. Fix InvalidateBlocks to compare blocks including their generation stamps. Contributed by Todd Lipcon. (Revision 1339897)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339897
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightLinkedSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1049 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1049/ ) HDFS-3391 . Fix InvalidateBlocks to compare blocks including their generation stamps. Contributed by Todd Lipcon. (Revision 1339897) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339897 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightLinkedSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1083 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1083/)
          HDFS-3391. Fix InvalidateBlocks to compare blocks including their generation stamps. Contributed by Todd Lipcon. (Revision 1339897)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339897
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightLinkedSet.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1083 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1083/ ) HDFS-3391 . Fix InvalidateBlocks to compare blocks including their generation stamps. Contributed by Todd Lipcon. (Revision 1339897) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339897 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightHashSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightLinkedSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestLightWeightHashSet.java

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Arun C Murthy
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development