Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3605

Block mistakenly marked corrupt during edit log catchup phase of failover

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 3.0.0, 2.0.2-alpha
    • Component/s: ha, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Open file for append
      Write data and sync.
      After next log roll and editlog tailing in standbyNN close the append stream.
      Call append multiple times on the same file, before next editlog roll.
      Now abruptly kill the current active namenode.

      Here block is missed..

      this may be because of All latest blocks were queued in StandBy Namenode.
      During failover, first OP_CLOSE was processing the pending queue and adding the block to corrupted block.

      1. HDFS-3605.patch
        12 kB
        Uma Maheswara Rao G
      2. hdfs-3605.txt
        10 kB
        Todd Lipcon
      3. hdfs-3605.txt
        11 kB
        Todd Lipcon
      4. hdfs-3605.txt
        11 kB
        Todd Lipcon
      5. TestAppendBlockMiss.java
        4 kB
        Brahma Reddy Battula

        Activity

        Hide
        Uma Maheswara Rao G added a comment -

        Great catch Brahma.

        Here I have one question, why we are keeping all the blocks which are having the same blockID and different genstamps due to append recovery etc.? I think we should maintain only the latest block which is reported recently. Mostly this block will have the higher genstamp.

        Other part is:

        if (changeMade) {
                // The state or gen-stamp of the block has changed. So, we may be
                // able to process some messages from datanodes that we previously
                // were unable to process.
                fsNamesys.getBlockManager().processQueuedMessagesForBlock(newBlock);
              }
        

        In updateBlocks we have done this, because of optimizing the queued block processing?
        Due to this, it may mark block as corrupt right if have queued older genstamp block?

        What if we maintain only recently reported genstamp block in postPonedDNMessages and do only after loading complete edits?

        Show
        Uma Maheswara Rao G added a comment - Great catch Brahma. Here I have one question, why we are keeping all the blocks which are having the same blockID and different genstamps due to append recovery etc.? I think we should maintain only the latest block which is reported recently. Mostly this block will have the higher genstamp. Other part is: if (changeMade) { // The state or gen-stamp of the block has changed. So, we may be // able to process some messages from datanodes that we previously // were unable to process. fsNamesys.getBlockManager().processQueuedMessagesForBlock(newBlock); } In updateBlocks we have done this, because of optimizing the queued block processing? Due to this, it may mark block as corrupt right if have queued older genstamp block? What if we maintain only recently reported genstamp block in postPonedDNMessages and do only after loading complete edits?
        Hide
        Todd Lipcon added a comment -

        I'm sorry, I'm not entirely understanding the description. Can you post a unit test which reproduces the issue?

        Show
        Todd Lipcon added a comment - I'm sorry, I'm not entirely understanding the description. Can you post a unit test which reproduces the issue?
        Hide
        Brahma Reddy Battula added a comment -

        Hi Todd,
        Thanks for taking a look..Attaching unit test to reproduce this issue.

        Show
        Brahma Reddy Battula added a comment - Hi Todd, Thanks for taking a look..Attaching unit test to reproduce this issue.
        Hide
        Todd Lipcon added a comment -

        Thanks for the test. Very helpful. I'll take a look at this.

        Show
        Todd Lipcon added a comment - Thanks for the test. Very helpful. I'll take a look at this.
        Hide
        Uma Maheswara Rao G added a comment -
        public void processQueuedMessagesForBlock(Block b) throws IOException {
            Queue<ReportedBlockInfo> queue = pendingDNMessages.takeBlockQueue(b);
            if (queue == null) {
              // Nothing to re-process
              return;
            }
            processQueuedMessages(queue);
          }
        

        I think here, on first OP_CLOSE edit processing it is trying to process the QueuedMessagesForBlock. But here queued messages may contains the more future block as well, because duw to many append calls, SNN queued that messages.
        Instead processing all the queued messages for that block, it is make sense to process that current block(current OP_CLOSE genstamp block)?

        pendingDNMessages.takeBlockQueue(b); will give set of blocks which are matching to the blockID, because it was queuse by block ID. did not consider genstamp. But, by considering current case, do we need to consider getstamp also, while getting the blcok from queued message and process. Because there moght be some more OPs which will come with that block as part of furtheer appends. At that time, anyway that respective genstamp blocks will get processed right?

        Show
        Uma Maheswara Rao G added a comment - public void processQueuedMessagesForBlock(Block b) throws IOException { Queue<ReportedBlockInfo> queue = pendingDNMessages.takeBlockQueue(b); if (queue == null ) { // Nothing to re-process return ; } processQueuedMessages(queue); } I think here, on first OP_CLOSE edit processing it is trying to process the QueuedMessagesForBlock. But here queued messages may contains the more future block as well, because duw to many append calls, SNN queued that messages. Instead processing all the queued messages for that block, it is make sense to process that current block(current OP_CLOSE genstamp block)? pendingDNMessages.takeBlockQueue(b); will give set of blocks which are matching to the blockID, because it was queuse by block ID. did not consider genstamp. But, by considering current case, do we need to consider getstamp also, while getting the blcok from queued message and process. Because there moght be some more OPs which will come with that block as part of furtheer appends. At that time, anyway that respective genstamp blocks will get processed right?
        Hide
        Todd Lipcon added a comment -

        The design of the code should be such that, it will re-process those "future" events, but they'll get re-postponed at that point. Maybe the issue is specifically in the case where these opcodes get read during the "catchup" during transition to active?

        Show
        Todd Lipcon added a comment - The design of the code should be such that, it will re-process those "future" events, but they'll get re-postponed at that point. Maybe the issue is specifically in the case where these opcodes get read during the "catchup" during transition to active?
        Hide
        Uma Maheswara Rao G added a comment -

        The design of the code should be such that, it will re-process those "future" events, but they'll get re-postponed at that point

        This is what I mean, if i understand you intent correctly here. leave the future genstamps here for processing. Once all the OP codes read and processed anyway it is processing all quequed messages again if anything left I remeber. So, this should help us in this case.

        Maybe the issue is specifically in the case where these opcodes get read during the "catchup" during transition to active?

        What issue you are pointing here. Edits are getting read in correct order only right?

        Show
        Uma Maheswara Rao G added a comment - The design of the code should be such that, it will re-process those "future" events, but they'll get re-postponed at that point This is what I mean, if i understand you intent correctly here. leave the future genstamps here for processing. Once all the OP codes read and processed anyway it is processing all quequed messages again if anything left I remeber. So, this should help us in this case. Maybe the issue is specifically in the case where these opcodes get read during the "catchup" during transition to active? What issue you are pointing here. Edits are getting read in correct order only right?
        Hide
        Vinayakumar B added a comment -

        The design of the code should be such that, it will re-process those "future" events, but they'll get re-postponed at that point. Maybe the issue is specifically in the case where these opcodes get read during the "catchup" during transition to active?

        You are correct Todd. this problem will come in case of catch up during failover.

        if (namesystem.isInStandbyState() &&
                namesystem.isGenStampInFuture(block.getGenerationStamp())) {
              queueReportedBlock(dn, block, reportedState,
                  QUEUE_REASON_FUTURE_GENSTAMP);
              return null;
            }

        Since during failover the state is already changed to ACTIVE, block will not be added again to queue even though it is in future.

        Show
        Vinayakumar B added a comment - The design of the code should be such that, it will re-process those "future" events, but they'll get re-postponed at that point. Maybe the issue is specifically in the case where these opcodes get read during the "catchup" during transition to active? You are correct Todd. this problem will come in case of catch up during failover. if (namesystem.isInStandbyState() && namesystem.isGenStampInFuture(block.getGenerationStamp())) { queueReportedBlock(dn, block, reportedState, QUEUE_REASON_FUTURE_GENSTAMP); return null ; } Since during failover the state is already changed to ACTIVE, block will not be added again to queue even though it is in future.
        Hide
        Vinayakumar B added a comment -

        @Uma

        Here I have one question, why we are keeping all the blocks which are having the same blockID and different genstamps due to append recovery etc.? I think we should maintain only the latest block which is reported recently. Mostly this block will have the higher genstamp.

        I agree with your point, we can keep only the latest reported state of the block regardless of genstamp from each datanode instead of keeping all previous states in queue which may be outdated by the time those are processed.

        Show
        Vinayakumar B added a comment - @Uma Here I have one question, why we are keeping all the blocks which are having the same blockID and different genstamps due to append recovery etc.? I think we should maintain only the latest block which is reported recently. Mostly this block will have the higher genstamp. I agree with your point, we can keep only the latest reported state of the block regardless of genstamp from each datanode instead of keeping all previous states in queue which may be outdated by the time those are processed.
        Hide
        Todd Lipcon added a comment -

        So, I think the fix needs to be that, while we're doing the "catch-up" tailing, we continue to treat everything as if it were still in standby state, from the block management perspective. Then process any remaining queued messages only after becoming active. Right?

        Show
        Todd Lipcon added a comment - So, I think the fix needs to be that, while we're doing the "catch-up" tailing, we continue to treat everything as if it were still in standby state, from the block management perspective. Then process any remaining queued messages only after becoming active. Right?
        Hide
        Uma Maheswara Rao G added a comment -

        Exactly Todd, This is what we have done in our internal branch as a work around for this fix. Still we can discuss about the optimization as we discussed in above comment, like we can process only the current genstamp blkID, and postpone remaining stuff. Other case could be that maintain only, renecnt block report with the greater genstamp. Do you feel any issues with this?

        Show
        Uma Maheswara Rao G added a comment - Exactly Todd, This is what we have done in our internal branch as a work around for this fix. Still we can discuss about the optimization as we discussed in above comment, like we can process only the current genstamp blkID, and postpone remaining stuff. Other case could be that maintain only, renecnt block report with the greater genstamp. Do you feel any issues with this?
        Hide
        Uma Maheswara Rao G added a comment -

        Note that, In this case we have 2 options

        • we have to maintain only recently reoprted blocks on QueuedDNMessages(with latest genstamp). we have to think the impacts of this, if any scenario we are missing. Currently we have opted this for our work around.
          (or)
        • process the current genstamp then and there and postpone higher genstamp blcoks again here. Otherwise if we just postpone everything till all edits loading, then the older genstamp blocks will create issue as genstamp already might have updated to higher in blocksMap while loading all edits, So, still block may get marked as corrupt.
        Show
        Uma Maheswara Rao G added a comment - Note that, In this case we have 2 options we have to maintain only recently reoprted blocks on QueuedDNMessages(with latest genstamp). we have to think the impacts of this, if any scenario we are missing. Currently we have opted this for our work around. (or) process the current genstamp then and there and postpone higher genstamp blcoks again here. Otherwise if we just postpone everything till all edits loading, then the older genstamp blocks will create issue as genstamp already might have updated to higher in blocksMap while loading all edits, So, still block may get marked as corrupt.
        Hide
        Todd Lipcon added a comment -

        Hey Uma. I think we should separate the discussion of a potential optimization from the discussion of fixing this bug.

        Do you already have a patch for the bug? If not, I'll make one following the approach described above.

        Once the bug fix is in, we can talk about how to optimize the memory usage.

        Show
        Todd Lipcon added a comment - Hey Uma. I think we should separate the discussion of a potential optimization from the discussion of fixing this bug. Do you already have a patch for the bug? If not, I'll make one following the approach described above. Once the bug fix is in, we can talk about how to optimize the memory usage.
        Hide
        Uma Maheswara Rao G added a comment -

        Hi Todd,

        Attached a patch which I am thiking of currently.

        I think we should separate the discussion of a potential optimization from the discussion of fixing this bug.

        Sure. Infact that is not an optimization, its required in the approach which I am thinking.

        Please take a look and correct me, if I miss something in the patch as you and Jitendra involved in that changes mainly.

        Thanks a lot,

        Uma

        Show
        Uma Maheswara Rao G added a comment - Hi Todd, Attached a patch which I am thiking of currently. I think we should separate the discussion of a potential optimization from the discussion of fixing this bug. Sure. Infact that is not an optimization, its required in the approach which I am thinking. Please take a look and correct me, if I miss something in the patch as you and Jitendra involved in that changes mainly. Thanks a lot, Uma
        Hide
        Todd Lipcon added a comment -

        Hey Uma. I took your unit test (thanks) and modified it to be minimal and remove sleeps. Then I prepared a patch with a slightly different approach: I now use a boolean inside BlockManager to determine whether to do the block postponement. I think this is a bit simpler, and still fixes the issue.

        Am I missing another case with this fix? The optimization you did might be useful but per above I think we can make this minimal and optimize separately. I don't think it's required for the bugfix.

        This patch isn't quite final - I want to add a few javadocs, etc.

        Show
        Todd Lipcon added a comment - Hey Uma. I took your unit test (thanks) and modified it to be minimal and remove sleeps. Then I prepared a patch with a slightly different approach: I now use a boolean inside BlockManager to determine whether to do the block postponement. I think this is a bit simpler, and still fixes the issue. Am I missing another case with this fix? The optimization you did might be useful but per above I think we can make this minimal and optimize separately. I don't think it's required for the bugfix. This patch isn't quite final - I want to add a few javadocs, etc.
        Hide
        Todd Lipcon added a comment -

        submitting patch to make sure it doesn't break other tests

        Show
        Todd Lipcon added a comment - submitting patch to make sure it doesn't break other tests
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12536481/hdfs-3605.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.TestBackupNode
        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2821//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2821//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536481/hdfs-3605.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2821//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2821//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        TestPipelinesFailover is probably related. I'll look into that.

        Show
        Todd Lipcon added a comment - TestPipelinesFailover is probably related. I'll look into that.
        Hide
        Uma Maheswara Rao G added a comment -

        Thanks a lot, Todd for the patch.

        I have taken a quick look on the patch. Yes, this approach should work as well.
        Blocks will get processed for all the ops, so, the matching to current genStamp will get processed in current iteration and future ones will get postponed again.

        A few comments on patch: Did not check for any javadoc issues as you mentioned already, i.e, will work on javadocs.

        + out.writeBytes("/data");
        +
        + // TODO: why do we need an hflush for this test case to fail?

        I remember, this is just added to ensure tthat the current packet will be en-queued and block will get allocated.
        Other wise less than 64K content may not be flushed at that time.

        DFSTestUtil.appendFile(fs, fileToAppend, "data");

        Having the multiple append calls can give the regression for the case where we have many genstamp and they got processed in order and future ones will get postponed.

        // Wait till DN reports blocks
        + cluster.triggerBlockReports();

        comment need to update?

        shouldPostponeInvalidBlocks

        do we need to change the variable name? Since blocks are not declared as invalid yet.

        Will take a look deeply on the patch tomorrow again. ( not able to concentrate much, as I am traveling today)

        Show
        Uma Maheswara Rao G added a comment - Thanks a lot, Todd for the patch. I have taken a quick look on the patch. Yes, this approach should work as well. Blocks will get processed for all the ops, so, the matching to current genStamp will get processed in current iteration and future ones will get postponed again. A few comments on patch: Did not check for any javadoc issues as you mentioned already, i.e, will work on javadocs. + out.writeBytes("/data"); + + // TODO: why do we need an hflush for this test case to fail? I remember, this is just added to ensure tthat the current packet will be en-queued and block will get allocated. Other wise less than 64K content may not be flushed at that time. DFSTestUtil.appendFile(fs, fileToAppend, "data"); Having the multiple append calls can give the regression for the case where we have many genstamp and they got processed in order and future ones will get postponed. // Wait till DN reports blocks + cluster.triggerBlockReports(); comment need to update? shouldPostponeInvalidBlocks do we need to change the variable name? Since blocks are not declared as invalid yet. Will take a look deeply on the patch tomorrow again. ( not able to concentrate much, as I am traveling today)
        Hide
        Todd Lipcon added a comment -

        I looked into the TestPipelinesFailover issue, and saw a "Too many open files" in the log. But I looked for leaks using MAT and some lsofing, and couldn't figure out what the leak issue is. Resubmitting an updated patch, which also addresses Uma's comments above.

        Show
        Todd Lipcon added a comment - I looked into the TestPipelinesFailover issue, and saw a "Too many open files" in the log. But I looked for leaks using MAT and some lsofing, and couldn't figure out what the leak issue is. Resubmitting an updated patch, which also addresses Uma's comments above.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12536725/hdfs-3605.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2835//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2835//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536725/hdfs-3605.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2835//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2835//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        What do you think, Uma. Does this look good to you?

        Show
        Todd Lipcon added a comment - What do you think, Uma. Does this look good to you?
        Hide
        Uma Maheswara Rao G added a comment -

        Hi Todd,

        sorry for the late reply on this. Stucked in someother work yesterday.

        Patch looks great to me.

        +1 on addressing a small nit.

        +  public void setPostponeInvalidBlockReports(boolean postpone) {
        +    this.shouldPostponeBlocksFromFuture  = postpone;
        +  }
        

        forgot to update method name? variable and method names looks different.

        I also ran some tests with this change by adding some debug points. Worked well for me.

        Thanks,
        Uma

        Show
        Uma Maheswara Rao G added a comment - Hi Todd, sorry for the late reply on this. Stucked in someother work yesterday. Patch looks great to me. +1 on addressing a small nit. + public void setPostponeInvalidBlockReports( boolean postpone) { + this .shouldPostponeBlocksFromFuture = postpone; + } forgot to update method name? variable and method names looks different. I also ran some tests with this change by adding some debug points. Worked well for me. Thanks, Uma
        Hide
        Todd Lipcon added a comment -

        Addresses Uma's nit above

        Show
        Todd Lipcon added a comment - Addresses Uma's nit above
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12536961/hdfs-3605.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2853//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2853//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536961/hdfs-3605.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2853//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2853//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        I'll commit this momentarily based on Uma's +1 above, since it was only a small nit.

        Show
        Todd Lipcon added a comment - I'll commit this momentarily based on Uma's +1 above, since it was only a small nit.
        Hide
        Todd Lipcon added a comment -

        Committed to branch-2 and trunk. Thanks Brahma for reporting the bug and writing a test case, and thanks Uma for the reviews and discussion above!

        Show
        Todd Lipcon added a comment - Committed to branch-2 and trunk. Thanks Brahma for reporting the bug and writing a test case, and thanks Uma for the reviews and discussion above!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2520 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2520/)
        HDFS-3605. Block mistakenly marked corrupt during edit log catchup phase of failover. Contributed by Todd Lipcon and Brahma Reddy Battula. (Revision 1363175)

        Result = FAILURE
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363175
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAAppend.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2520 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2520/ ) HDFS-3605 . Block mistakenly marked corrupt during edit log catchup phase of failover. Contributed by Todd Lipcon and Brahma Reddy Battula. (Revision 1363175) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363175 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAAppend.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2563 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2563/)
        HDFS-3605. Block mistakenly marked corrupt during edit log catchup phase of failover. Contributed by Todd Lipcon and Brahma Reddy Battula. (Revision 1363175)

        Result = SUCCESS
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363175
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAAppend.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2563 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2563/ ) HDFS-3605 . Block mistakenly marked corrupt during edit log catchup phase of failover. Contributed by Todd Lipcon and Brahma Reddy Battula. (Revision 1363175) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363175 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAAppend.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2498 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2498/)
        HDFS-3605. Block mistakenly marked corrupt during edit log catchup phase of failover. Contributed by Todd Lipcon and Brahma Reddy Battula. (Revision 1363175)

        Result = SUCCESS
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363175
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAAppend.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2498 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2498/ ) HDFS-3605 . Block mistakenly marked corrupt during edit log catchup phase of failover. Contributed by Todd Lipcon and Brahma Reddy Battula. (Revision 1363175) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363175 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAAppend.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1108 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1108/)
        HDFS-3605. Block mistakenly marked corrupt during edit log catchup phase of failover. Contributed by Todd Lipcon and Brahma Reddy Battula. (Revision 1363175)

        Result = FAILURE
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363175
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAAppend.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1108 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1108/ ) HDFS-3605 . Block mistakenly marked corrupt during edit log catchup phase of failover. Contributed by Todd Lipcon and Brahma Reddy Battula. (Revision 1363175) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363175 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAAppend.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1141 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1141/)
        HDFS-3605. Block mistakenly marked corrupt during edit log catchup phase of failover. Contributed by Todd Lipcon and Brahma Reddy Battula. (Revision 1363175)

        Result = FAILURE
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363175
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAAppend.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1141 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1141/ ) HDFS-3605 . Block mistakenly marked corrupt during edit log catchup phase of failover. Contributed by Todd Lipcon and Brahma Reddy Battula. (Revision 1363175) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1363175 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAAppend.java

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Brahma Reddy Battula
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development