Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-29

In Datanode, update block may fail due to length inconsistency

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When a primary datanode tries to recover a block. It calls getBlockMetaDataInfo(..) to obtains information like block length from each datanode. Then, it calls updateBlock(..).

      The block length returned in getBlockMetaDataInfo(..) may be obtained from a unclosed local block file F. However, in updateBlock(..), it first closes F (if F is open) and then gets the length. These two lengths may be different. In such case, updateBlock(..) throws an exception.

      1. h29_20091012.patch
        4 kB
        Tsz Wo Nicholas Sze
      2. h29_20091012b.patch
        7 kB
        Tsz Wo Nicholas Sze
      3. h29_20091012c.patch
        7 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          dhruba borthakur added a comment -

          The getBlockMetaDataInfo() should first terminate all open connections to the block and then return the size of the block. Then it is guaranteed that nobody can change the length of the block between the getBlockMetaDataInfo() and updateBlock() calls. This is documented in Step 5 in https://issues.apache.org/jira/browse/HADOOP-4663?focusedCommentId=12674490&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12674490

          will this solve your problem?

          Show
          dhruba borthakur added a comment - The getBlockMetaDataInfo() should first terminate all open connections to the block and then return the size of the block. Then it is guaranteed that nobody can change the length of the block between the getBlockMetaDataInfo() and updateBlock() calls. This is documented in Step 5 in https://issues.apache.org/jira/browse/HADOOP-4663?focusedCommentId=12674490&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12674490 will this solve your problem?
          Hide
          Hairong Kuang added a comment -

          In this case, there is no on-going writes. The problem is that data is not flushed to disk when getBlockMetaDataInfo is called. But updateBlock flushes and closes the file. Therefore, there is an inconsistency.

          Show
          Hairong Kuang added a comment - In this case, there is no on-going writes. The problem is that data is not flushed to disk when getBlockMetaDataInfo is called. But updateBlock flushes and closes the file. Therefore, there is an inconsistency.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          After a series of HDFS-265 sub-task patches, getBlockMetaDataInfo(..) and updateBlock(..) were replaced by initReplicaRecovery(..) and updateReplicaUnderRecovery(..), respectively. The stop writer logic was moved from updateBlock(..) to initReplicaRecovery(..) so that the writer is stopped before getting the replica length. So this problem disappears in the new codes.

          I suggest that we add more validation to check the replica and its file lengths, more specifically, to check

          • file length and replica's visible length after stopping a writer, and
          • replica's original visible length obtained from initReplicaRecovery(..) and its current visible length in updateReplicaUnderRecovery(..).
          Show
          Tsz Wo Nicholas Sze added a comment - After a series of HDFS-265 sub-task patches, getBlockMetaDataInfo(..) and updateBlock(..) were replaced by initReplicaRecovery(..) and updateReplicaUnderRecovery(..), respectively. The stop writer logic was moved from updateBlock(..) to initReplicaRecovery(..) so that the writer is stopped before getting the replica length. So this problem disappears in the new codes. I suggest that we add more validation to check the replica and its file lengths, more specifically, to check file length and replica's visible length after stopping a writer, and replica's original visible length obtained from initReplicaRecovery(..) and its current visible length in updateReplicaUnderRecovery(..).
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h29_20091012.patch: added more validation as described before.

          Show
          Tsz Wo Nicholas Sze added a comment - h29_20091012.patch: added more validation as described before.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h29_20091012b.patch:

          • fixed some bugs in TestInterDatanodeProtocol
          • also changed TestInterDatanodeProtocol to use junit 4.
          Show
          Tsz Wo Nicholas Sze added a comment - h29_20091012b.patch: fixed some bugs in TestInterDatanodeProtocol also changed TestInterDatanodeProtocol to use junit 4.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h29_20091012c.patch:

          • replaced ReplicaBeingWritten by FinalizedReplica in the test since the test only checks the logic and does not create files for ReplicaBeingWritten;
          • fixed some comments in the test.
          Show
          Tsz Wo Nicholas Sze added a comment - h29_20091012c.patch: replaced ReplicaBeingWritten by FinalizedReplica in the test since the test only checks the logic and does not create files for ReplicaBeingWritten; fixed some comments in the test.
          Hide
          Tsz Wo Nicholas Sze added a comment -
               [exec] +1 overall.
               [exec]
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec]
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec]
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          

          Also passed all unit tests locally.

          Show
          Tsz Wo Nicholas Sze added a comment - [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Also passed all unit tests locally.
          Hide
          Hairong Kuang added a comment -

          +1. The patch looks good.

          Show
          Hairong Kuang added a comment - +1. The patch looks good.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hairong, thank you for reviewing it.

          I have committed this to 0.21 and above.

          Show
          Tsz Wo Nicholas Sze added a comment - Hairong, thank you for reviewing it. I have committed this to 0.21 and above.
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #47 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/47/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #47 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/47/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #79 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/79/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #79 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/79/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #78 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/78/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #78 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/78/ )
          Hide
          Uma Maheswara Rao G added a comment -

          Looks this scenario applicable for 20version also.
          patch available for 20 version?

          Show
          Uma Maheswara Rao G added a comment - Looks this scenario applicable for 20version also. patch available for 20 version?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          By 0.20, I guess you mean 0.20-append. Since 0.20-append and 0.21 have different append design and codes, the patch here won't apply without a significant change. We also have to carefully verify the design difference. So, how about we create another jira for 0.20-append if a similar problem exist?

          Show
          Tsz Wo Nicholas Sze added a comment - By 0.20, I guess you mean 0.20-append. Since 0.20-append and 0.21 have different append design and codes, the patch here won't apply without a significant change. We also have to carefully verify the design difference. So, how about we create another jira for 0.20-append if a similar problem exist?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > patch available for 20 version?
          Forgot to answer you question: I don't have a patch for 0.20-append.

          Show
          Tsz Wo Nicholas Sze added a comment - > patch available for 20 version? Forgot to answer you question: I don't have a patch for 0.20-append.

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development