Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15875

Check whether file is being truncated before truncate

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0, 3.1.4, 3.2.2
    • 3.3.1, 3.4.0, 3.2.3
    • None

    Description

      We have got this problem.

      • A job sends truncate to namenode, and the block recovery goes.
      • DataNode D is timeout while it connects another datanode (60s), so block recovery costs 60+s
      • A job tails, and B job starts and it sends truncate to namenode. New recoveryId generates during recovery lease.
      • DataNode D commitBlockSynchronization and get errors "does not match current recovery id"

      So truncate will not complete forever. Datanode D has replica with new length and two other datanodes have replica old length. 

      DN has the error messages "Inconsistent size of finalized replicas"

      the related code is in BlockRecoveryWorker.java

      
      for (BlockRecord r : syncList) {
       assert r.rInfo.getNumBytes() > 0 : "zero length replica";
       ReplicaState rState = r.rInfo.getOriginalReplicaState();
       if (rState.getValue() < bestState.getValue()) {
       bestState = rState;
       }
       if(rState == ReplicaState.FINALIZED) {
       if (finalizedLength > 0 && finalizedLength != r.rInfo.getNumBytes()) {
       throw new IOException("Inconsistent size of finalized replicas. " +
       "Replica " + r.rInfo + " expected size: " + finalizedLength);
       }
       finalizedLength = r.rInfo.getNumBytes();
       }
      }
      
      

       

       

       

       

       

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ferhui Hui Fei
            ferhui Hui Fei
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 50m
                1h 50m

                Slack

                  Issue deployment