Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15875

Check whether file is being truncated before truncate

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.3.0, 3.1.4, 3.2.2
    • Fix Version/s: 3.3.1, 3.4.0, 3.2.3
    • Component/s: None

      Description

      We have got this problem.

      • A job sends truncate to namenode, and the block recovery goes.
      • DataNode D is timeout while it connects another datanode (60s), so block recovery costs 60+s
      • A job tails, and B job starts and it sends truncate to namenode. New recoveryId generates during recovery lease.
      • DataNode D commitBlockSynchronization and get errors "does not match current recovery id"

      So truncate will not complete forever. Datanode D has replica with new length and two other datanodes have replica old length. 

      DN has the error messages "Inconsistent size of finalized replicas"

      the related code is in BlockRecoveryWorker.java

      
      for (BlockRecord r : syncList) {
       assert r.rInfo.getNumBytes() > 0 : "zero length replica";
       ReplicaState rState = r.rInfo.getOriginalReplicaState();
       if (rState.getValue() < bestState.getValue()) {
       bestState = rState;
       }
       if(rState == ReplicaState.FINALIZED) {
       if (finalizedLength > 0 && finalizedLength != r.rInfo.getNumBytes()) {
       throw new IOException("Inconsistent size of finalized replicas. " +
       "Replica " + r.rInfo + " expected size: " + finalizedLength);
       }
       finalizedLength = r.rInfo.getNumBytes();
       }
      }
      
      

       

       

       

       

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ferhui Hui Fei
                Reporter:
                ferhui Hui Fei
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 50m
                  1h 50m