Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1779

After NameNode restart , Clients can not read partial files even after client invokes Sync.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20-append
    • Fix Version/s: 0.20-append, 0.20.205.0
    • Component/s: datanode, namenode
    • Labels:
      None
    • Environment:

      Linux

    • Hadoop Flags:
      Reviewed

      Description

      In Append HDFS-200 issue,
      If file has 10 blocks and after writing 5 blocks if client invokes sync method then NN will persist the blocks information in edits.
      After this if we restart the NN, All the DataNodes will reregister with NN. But DataNodes are not sending the blocks being written information to NN. DNs are sending the blocksBeingWritten information in DN startup. So, here NameNode can not find that the 5 persisted blocks belongs to which datanodes. This information can build based on block reports from DN. Otherwise we will loose this 5 blocks information even NN persisted that block information in edits.

      1. HDFS-1779-20.security.3-fixes.patch
        1 kB
        Eli Collins
      2. HDFS-1779-20.security.3.patch
        22 kB
        Jitendra Nath Pandey
      3. HDFS-1779.patch
        8 kB
        Uma Maheswara Rao G
      4. HDFS-1779.2a.patch
        22 kB
        Uma Maheswara Rao G
      5. HDFS-1779.20Append.fix.patch
        1 kB
        Uma Maheswara Rao G
      6. HDFS-1779.2.patch
        20 kB
        Uma Maheswara Rao G
      7. HDFS-1779.1.patch
        10 kB
        Uma Maheswara Rao G
      8. bbwReportAppend.patch
        15 kB
        Hairong Kuang

        Issue Links

          Activity

          Hide
          Eli Collins added a comment -

          Thanks Uma. I committed the fix to allow tests to run with append enabled to branch-20-security.

          Show
          Eli Collins added a comment - Thanks Uma. I committed the fix to allow tests to run with append enabled to branch-20-security.
          Hide
          Matt Foley added a comment -

          Closed upon release of 0.20.205.0

          Show
          Matt Foley added a comment - Closed upon release of 0.20.205.0
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks a lot Eli for updating the patch.
          Even though we are running the tests with append off, better to handle it. In future if we enable append by default, again we need to fix this Test Code. Good point, lets handle now itself.
          I have updated the patch for 20Append as well.

          Patch looks good to me.
          +1 from my side.

          Thanks
          Uma

          Show
          Uma Maheswara Rao G added a comment - Thanks a lot Eli for updating the patch. Even though we are running the tests with append off, better to handle it. In future if we enable append by default, again we need to fix this Test Code. Good point, lets handle now itself. I have updated the patch for 20Append as well. Patch looks good to me. +1 from my side. Thanks Uma
          Hide
          Eli Collins added a comment -

          Patch attached with fixes for branch 20 security.

          Show
          Eli Collins added a comment - Patch attached with fixes for branch 20 security.
          Hide
          Eli Collins added a comment -

          Also, SimulatedFSDataset#getBlocksBeingWrittenReport should return new Block[0] not null, otherwise Datanode#register will pass null to BlockListAsLongs#convertToArrayLongs causing an NPE. You probably didn't see this on branch 20 security because it's not running with append support defaulted to true.

          Show
          Eli Collins added a comment - Also, SimulatedFSDataset#getBlocksBeingWrittenReport should return new Block [0] not null, otherwise Datanode#register will pass null to BlockListAsLongs#convertToArrayLongs causing an NPE. You probably didn't see this on branch 20 security because it's not running with append support defaulted to true.
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks Eli, I will take a look on this.

          Thanks
          Uma

          Show
          Uma Maheswara Rao G added a comment - Thanks Eli, I will take a look on this. Thanks Uma
          Hide
          Eli Collins added a comment - - edited

          Nit, in recoverBlocksBeingWritten we should remove the comment wrt sending block received messages since this was removed.

          -     * ongoingCreates. Also, send a blockreceived message to the NN
          -     * for each of these blocks because these are not part of a 
          -     * block report.
          +     * ongoingCreates.
          
          Show
          Eli Collins added a comment - - edited Nit, in recoverBlocksBeingWritten we should remove the comment wrt sending block received messages since this was removed. - * ongoingCreates. Also, send a blockreceived message to the NN - * for each of these blocks because these are not part of a - * block report. + * ongoingCreates.
          Hide
          Jitendra Nath Pandey added a comment -

          Committed to both append and security branches.

          Show
          Jitendra Nath Pandey added a comment - Committed to both append and security branches.
          Hide
          Uma Maheswara Rao G added a comment -

          Hi Todd/Jitendra, Can you please take a look for 20Append?

          Show
          Uma Maheswara Rao G added a comment - Hi Todd/Jitendra, Can you please take a look for 20Append?
          Hide
          Uma Maheswara Rao G added a comment -

          Can some one commit on 20Append also?

          Thanks
          Uma

          Show
          Uma Maheswara Rao G added a comment - Can some one commit on 20Append also? Thanks Uma
          Hide
          Jitendra Nath Pandey added a comment -

          Commited to branch-0.20-security. Thanks to Uma!

          Show
          Jitendra Nath Pandey added a comment - Commited to branch-0.20-security. Thanks to Uma!
          Hide
          Suresh Srinivas added a comment -

          +1 for the patch.

          Show
          Suresh Srinivas added a comment - +1 for the patch.
          Hide
          Uma Maheswara Rao G added a comment -

          Hi Jitendra,

          Thanks for the Review and security Patch!

          I just gone through the patch!
          +1 from my side. It looks good.

          Show
          Uma Maheswara Rao G added a comment - Hi Jitendra, Thanks for the Review and security Patch! I just gone through the patch! +1 from my side. It looks good.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493838/HDFS-1779-20.security.3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1226//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493838/HDFS-1779-20.security.3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1226//console This message is automatically generated.
          Hide
          Jitendra Nath Pandey added a comment -

          Patch for branch-0.20-security.

          Show
          Jitendra Nath Pandey added a comment - Patch for branch-0.20-security.
          Hide
          Jitendra Nath Pandey added a comment -

          +1 for the patch. This needs to be ported to 20-security branch as well.

          Show
          Jitendra Nath Pandey added a comment - +1 for the patch. This needs to be ported to 20-security branch as well.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493354/HDFS-1779.2a.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1219//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493354/HDFS-1779.2a.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1219//console This message is automatically generated.
          Hide
          Uma Maheswara Rao G added a comment -

          Hi Jitendra,

          Thanks a lot for taking a look!

          2. Please use cluster.restartNameNode() instead of private restartNamenode.
          4. In tearDown, namenode.stop is not required because cluster.shutdown takes care of it.

          Actually in 20Append, there is no restartNameNode api in MiniDFsCluster.
          That is the reason i provided my own implementation. Now i have back ported that API in MiniDFSCluster.

          Addressed the all remaining comments as well.

          Thanks
          Uma

          Show
          Uma Maheswara Rao G added a comment - Hi Jitendra, Thanks a lot for taking a look! 2. Please use cluster.restartNameNode() instead of private restartNamenode. 4. In tearDown, namenode.stop is not required because cluster.shutdown takes care of it. Actually in 20Append, there is no restartNameNode api in MiniDFsCluster. That is the reason i provided my own implementation. Now i have back ported that API in MiniDFSCluster. Addressed the all remaining comments as well. Thanks Uma
          Hide
          Jitendra Nath Pandey added a comment -

          It seems this will partly fix the issues in HDFS-1197. Since bbw are only added to targets, the premature completeFile should not happen due to addStoredBlocks.

          Show
          Jitendra Nath Pandey added a comment - It seems this will partly fix the issues in HDFS-1197 . Since bbw are only added to targets, the premature completeFile should not happen due to addStoredBlocks.
          Hide
          Jitendra Nath Pandey added a comment -

          A few comments:
          TestAppend.java:

          1. I will recommend to rename the class to something like TestBBWBlockReport.
          2. Please use cluster.restartNameNode() instead of private restartNamenode.
          3. Can we avoid making dfs, namenode, outStream etc as class fields.
          4. In tearDown, namenode.stop is not required because cluster.shutdown takes care of it.
          5. outStream.close will likely fail after testDNShouldNotSendBBWReportIfAppendOff because namenode will be in safemode.
          6. waitFor method can be replaced with Thread.sleep.
          Show
          Jitendra Nath Pandey added a comment - A few comments: TestAppend.java: I will recommend to rename the class to something like TestBBWBlockReport. Please use cluster.restartNameNode() instead of private restartNamenode. Can we avoid making dfs, namenode, outStream etc as class fields. In tearDown, namenode.stop is not required because cluster.shutdown takes care of it. outStream.close will likely fail after testDNShouldNotSendBBWReportIfAppendOff because namenode will be in safemode. waitFor method can be replaced with Thread.sleep.
          Hide
          Hairong Kuang added a comment -

          With this patch, blockReceived does not get sent for bbw blocks. But my patch does not clean up the code for blockReceived.

          Show
          Hairong Kuang added a comment - With this patch, blockReceived does not get sent for bbw blocks. But my patch does not clean up the code for blockReceived.
          Hide
          Sanjay Radia added a comment -

          Since this patch sends a separate BBW-report on each DN registration, one should not need the "block received" that are being sent by DN introduced as part of hdfs-142 patch.

          Show
          Sanjay Radia added a comment - Since this patch sends a separate BBW-report on each DN registration, one should not need the "block received" that are being sent by DN introduced as part of hdfs-142 patch.
          Hide
          Hairong Kuang added a comment -

          Uma, thanks for addressing Todd's review comments and added unit tests.

          Todd, could you please review this one more time? Thanks!

          Show
          Hairong Kuang added a comment - Uma, thanks for addressing Todd's review comments and added unit tests. Todd, could you please review this one more time? Thanks!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493045/HDFS-1779.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1195//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493045/HDFS-1779.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1195//console This message is automatically generated.
          Hide
          Uma Maheswara Rao G added a comment -

          Hi Todd & Hairong,

          Added the additional tests to reproduce this scenario!

          Patch contains: Hairong code changes
          + Todd review comments fixes
          + Added additional testcases for reproducing this scenario.
          + made the supportAppends variable to private in datanode
          and added javadoc for processBlocksBeingWritten api.

          One more scenario is, if more number of blocks are under construction and also sync has been called for blocks .After that NN restart happened. Then if DNs didn't send the BBW report to NN, it will never become out of safemode because NN will expect some threshold of blocks to report to come out of safemode.This condition also added in tests.

          Thanks
          Uma

          Show
          Uma Maheswara Rao G added a comment - Hi Todd & Hairong, Added the additional tests to reproduce this scenario! Patch contains: Hairong code changes + Todd review comments fixes + Added additional testcases for reproducing this scenario. + made the supportAppends variable to private in datanode and added javadoc for processBlocksBeingWritten api. One more scenario is, if more number of blocks are under construction and also sync has been called for blocks .After that NN restart happened. Then if DNs didn't send the BBW report to NN, it will never become out of safemode because NN will expect some threshold of blocks to report to come out of safemode.This condition also added in tests. Thanks Uma
          Hide
          dhruba borthakur added a comment -

          Maybe we need to include this for the upcoming 0.20.205+ release that has append support.

          Show
          dhruba borthakur added a comment - Maybe we need to include this for the upcoming 0.20.205+ release that has append support.
          Hide
          Todd Lipcon added a comment -

          Mostly looks great. Small nits:

          • indentation in SimulatedFSDataset
          • there are some hard tabs in rejectAddStoredBlock
          • typo: 'unregisterted'

          Hairong, do you have a unit test? I have half of one here, similar to Uma's.

          Show
          Todd Lipcon added a comment - Mostly looks great. Small nits: indentation in SimulatedFSDataset there are some hard tabs in rejectAddStoredBlock typo: 'unregisterted' Hairong, do you have a unit test? I have half of one here, similar to Uma's.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12492778/bbwReportAppend.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1188//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12492778/bbwReportAppend.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1188//console This message is automatically generated.
          Hide
          Hairong Kuang added a comment -

          Here comes the patch:
          1. It adds a new RPC blocksBeingWrittenReport that allows datanode to send a report of bbw blocks.
          2. DataNode sends a bbw block report when registering with NameNode
          3. FSDatasetInterface is enhanced with getBlocksBeingWrittenReport API.

          Show
          Hairong Kuang added a comment - Here comes the patch: 1. It adds a new RPC blocksBeingWrittenReport that allows datanode to send a report of bbw blocks. 2. DataNode sends a bbw block report when registering with NameNode 3. FSDatasetInterface is enhanced with getBlocksBeingWrittenReport API.
          Hide
          Uma Maheswara Rao G added a comment -

          Hi Hairong,
          Actually i was also started looking into this. Since you have patch ready, I will test your patch.
          Thanks a lot Hairong for the help

          Thanks
          Uma

          Show
          Uma Maheswara Rao G added a comment - Hi Hairong, Actually i was also started looking into this. Since you have patch ready, I will test your patch. Thanks a lot Hairong for the help Thanks Uma
          Hide
          Hairong Kuang added a comment -

          Sure, I can upload a patch for supporting bbw block report.

          Show
          Hairong Kuang added a comment - Sure, I can upload a patch for supporting bbw block report.
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks a lot to Todd and Hairong for review!

          since that puts it into the blockmap but not the "targets" array. Wouldn't it be more correct to put it into the INodeFileUnderConstruction's targets array?

          Good point. I will take a look.
          I will update the patch as per your comments. or Hairong, do you have patch ready?

          Thanks,
          Uma

          Show
          Uma Maheswara Rao G added a comment - Thanks a lot to Todd and Hairong for review! since that puts it into the blockmap but not the "targets" array. Wouldn't it be more correct to put it into the INodeFileUnderConstruction's targets array? Good point. I will take a look. I will update the patch as per your comments. or Hairong, do you have patch ready? Thanks, Uma
          Hide
          Hairong Kuang added a comment -

          Yes, when processing a bbw block report, we put bbw replicas into tagets arrays but not in blocksMap.

          Basically with a separate bbw block report, addStoredBlock could be made much cleaner in append 0.20 and a bbw replica can be handled more semantic correctly.

          Show
          Hairong Kuang added a comment - Yes, when processing a bbw block report, we put bbw replicas into tagets arrays but not in blocksMap. Basically with a separate bbw block report, addStoredBlock could be made much cleaner in append 0.20 and a bbw replica can be handled more semantic correctly.
          Hide
          Todd Lipcon added a comment -

          Hi Uma, Hairong. I've been looking at this a bit here as well. I think it's not quite right to use blockReceived here - since that puts it into the blockmap but not the "targets" array. Wouldn't it be more correct to put it into the INodeFileUnderConstruction's targets array? Hairong: is that what your change does?

          Show
          Todd Lipcon added a comment - Hi Uma, Hairong. I've been looking at this a bit here as well. I think it's not quite right to use blockReceived here - since that puts it into the blockmap but not the "targets" array. Wouldn't it be more correct to put it into the INodeFileUnderConstruction's targets array? Hairong: is that what your change does?
          Hide
          Uma Maheswara Rao G added a comment -

          why do we need separate report for bbw?
          I think recoverBlcoksBeingWritten call will be enough upon registration.. no?
          Can you please just review the patch?
          If any suggestion from you, i will update the patch accordingly.

          Show
          Uma Maheswara Rao G added a comment - why do we need separate report for bbw? I think recoverBlcoksBeingWritten call will be enough upon registration.. no? Can you please just review the patch? If any suggestion from you, i will update the patch accordingly.
          Hide
          Uma Maheswara Rao G added a comment -

          Hi Hairong,
          I did not get what change your are suggesting here.
          Can you please give hint, what change you are expecting in this patch?
          Anyway i need to rebase it on trunk. If suggest the change i will be happy to do that

          Thanks
          Uma

          Show
          Uma Maheswara Rao G added a comment - Hi Hairong, I did not get what change your are suggesting here. Can you please give hint, what change you are expecting in this patch? Anyway i need to rebase it on trunk. If suggest the change i will be happy to do that Thanks Uma
          Hide
          Hairong Kuang added a comment -

          In our fb internal branch, we fixed the problem by sending a separate bbw report upon registration. Our solution is cleaner but this patch requires less code change.

          Show
          Hairong Kuang added a comment - In our fb internal branch, we fixed the problem by sending a separate bbw report upon registration. Our solution is cleaner but this patch requires less code change.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12489473/HDFS-1779.1.patch
          against trunk revision 1155998.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1070//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12489473/HDFS-1779.1.patch against trunk revision 1155998. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1070//console This message is automatically generated.
          Hide
          Uma Maheswara Rao G added a comment -

          Converted tests to Junit4.
          Updated patch for 20Append branch!

          Show
          Uma Maheswara Rao G added a comment - Converted tests to Junit4. Updated patch for 20Append branch!
          Hide
          Uma Maheswara Rao G added a comment -

          Yes Dhruba.

          The patch does the same thing that you have mentioned.
          DN sends the blocks being written to the NN on restart scenario only.

          Show
          Uma Maheswara Rao G added a comment - Yes Dhruba. The patch does the same thing that you have mentioned. DN sends the blocks being written to the NN on restart scenario only.
          Hide
          dhruba borthakur added a comment -

          if the namenode restarts, the datanode should invoke FSDataSet.recoverBlocksBeingWritten(). This will send all blocks in the block being written directory to the NN.

          Show
          dhruba borthakur added a comment - if the namenode restarts, the datanode should invoke FSDataSet.recoverBlocksBeingWritten(). This will send all blocks in the block being written directory to the NN.
          Hide
          Uma Maheswara Rao G added a comment -

          This patch is for 0.20-Append branch

          Show
          Uma Maheswara Rao G added a comment - This patch is for 0.20-Append branch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479917/HDFS-1779.patch
          against trunk revision 1125217.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/602//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479917/HDFS-1779.patch against trunk revision 1125217. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/602//console This message is automatically generated.
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks to Druba and Todd for spending time and giving comments.

          Datanodes already sends their blockRecieved command to NN successfull blocks. After this, NN restart happend then this problem problem can come. Because DataNode will not send any information about bbw.So NN can not find this partial blcok presents in which node.

          My proposal would be , When namenode is restarted, the datanode will re-register with the namenode. During datanode registration, can we recover the blocks in blocksBeingWritten directory ( sending bbw details to NN)? I verified this with two DNs, it works for normal scenarios. Do you think any impact with this ?

          Show
          Uma Maheswara Rao G added a comment - Thanks to Druba and Todd for spending time and giving comments. Datanodes already sends their blockRecieved command to NN successfull blocks. After this, NN restart happend then this problem problem can come. Because DataNode will not send any information about bbw.So NN can not find this partial blcok presents in which node. My proposal would be , When namenode is restarted, the datanode will re-register with the namenode. During datanode registration, can we recover the blocks in blocksBeingWritten directory ( sending bbw details to NN)? I verified this with two DNs, it works for normal scenarios. Do you think any impact with this ?
          Hide
          dhruba borthakur added a comment -

          If the app has invoked sync/hflush (as specified in the jira description), then the block is persisted in the fsimage. Lease recovery won't be successful until the time the datanode sends those blocks as part of a block report, So this is a bug and the workaround to get the missing blocks back would be to restart the datanodes?

          Show
          dhruba borthakur added a comment - If the app has invoked sync/hflush (as specified in the jira description), then the block is persisted in the fsimage. Lease recovery won't be successful until the time the datanode sends those blocks as part of a block report, So this is a bug and the workaround to get the missing blocks back would be to restart the datanodes?
          Hide
          Todd Lipcon added a comment -

          Dhruba – since we don't serialize the targets[] array with the lease, I don't think the NN could actually trigger block recovery properly. Am I missing something?

          Show
          Todd Lipcon added a comment - Dhruba – since we don't serialize the targets[] array with the lease, I don't think the NN could actually trigger block recovery properly. Am I missing something?
          Hide
          dhruba borthakur added a comment -

          Wait a minute, can you pl provide some more clarification? If you restart only the NN and not the DN, then the DN still has a reference to the blocks in the bbw directory. When the client closes the file (assuming that the client survived the NN restart), those blocks will send a blockReceived to the NN. If the client died, then the NN will start lease-recovery at some future point in time and that should make these datanodes send blockReceived messages to the NN. This sequence of events should make the blocks reappear in the NN. Are you seeing something different?

          Show
          dhruba borthakur added a comment - Wait a minute, can you pl provide some more clarification? If you restart only the NN and not the DN, then the DN still has a reference to the blocks in the bbw directory. When the client closes the file (assuming that the client survived the NN restart), those blocks will send a blockReceived to the NN. If the client died, then the NN will start lease-recovery at some future point in time and that should make these datanodes send blockReceived messages to the NN. This sequence of events should make the blocks reappear in the NN. Are you seeing something different?
          Hide
          dhruba borthakur added a comment -

          This seems to be a valid bug, would you have a patch for it?

          Show
          dhruba borthakur added a comment - This seems to be a valid bug, would you have a patch for it?

            People

            • Assignee:
              Uma Maheswara Rao G
              Reporter:
              Uma Maheswara Rao G
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development