Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-16806

ec data balancer block blk_id The index error ,Data cannot be moved

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 3.1.0
    • None
    • hdfs
    • None
    • Reviewed

    Description

      ec data balancer block blk_id The index error ,Data cannot be moved

      dn->10.12.15.149 use disk 100%

       

      echo 10.12.15.149>sorucehost
      balancer  -fs hdfs://xxcluster06  -threshold 10 -source -f sorucehost   2>>~/balancer.log &  

       

      datanode logs 
      A lot of this log output  

      datanode logs
      ...
      2022-10-19 14:43:02,031 ERROR datanode.DataNode (DataXceiver.java:run(321)) - fs-hiido-dn-12-15-149.xx.com:1019:DataXceiver error processing COPY_BLOCK operation  src: /10.12.65.216:58214 dst: /10.12.15.149:1019
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-1822992414-10.12.65.48-1660893388633:blk_-9223372036799576592_4218617
              at org.apache.hadoop.hdfs.server.datanode.BlockSender.getReplica(BlockSender.java:492)
              at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:256)
              at org.apache.hadoop.hdfs.server.datanode.DataXceiver.copyBlock(DataXceiver.java:1089)
              at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opCopyBlock(Receiver.java:291)
              at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:113)
              at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290)
              at java.lang.Thread.run(Thread.java:748)
      ...    
          
      hdfs fsck -fs hdfs://xxcluster06 -blockId blk_-9223372036799576592 
      Connecting to namenode via http://fs-hiido-xxcluster06-yynn2.xx.com:50070/fsck?ugi=hdfs&blockId=blk_-9223372036799576592+&path=%2F
      FSCK started by hdfs (auth:KERBEROS_SSL) from /10.12.19.4 at Wed Oct 19 14:47:15 CST 2022Block Id: blk_-9223372036799576592
      Block belongs to: /hive_warehouse/warehouse_old_snapshots/yy_mbsdkevent_original/dt=20210505/post_202105052129_33.log.gz
      No. of Expected Replica: 5
      No. of live Replica: 5
      No. of excess Replica: 0
      No. of stale Replica: 5
      No. of decommissioned Replica: 0
      No. of decommissioning Replica: 0
      No. of corrupted Replica: 0
      Block replica on datanode/rack: fs-hiido-dn-12-66-4.xx.com/4F08-01-09 is HEALTHY
      Block replica on datanode/rack: fs-hiido-dn-12-65-244.xx.com/4F08-01-08 is HEALTHY
      Block replica on datanode/rack: fs-hiido-dn-12-15-149.xx.com/4F08-05-13 is HEALTHY
      Block replica on datanode/rack: fs-hiido-dn-12-65-218.xx.com/4F08-12-04 is HEALTHY
      Block replica on datanode/rack: fs-hiido-dn-12-17-35.xx.com/4F08-03-03 is HEALTHY
      
      
      
      hdfs fsck -fs hdfs://xxcluster06 /hive_warehouse/warehouse_old_snapshots/yy_mbsdkevent_original/dt=20210505/post_202105052129_33.log.gz -files -blocks -locations
      Connecting to namenode via http://xx.com:50070/fsck?ugi=hdfs&files=1&blocks=1&locations=1&path=%2Fhive_warehouse%2Fwarehouse_old_snapshots%2Fyy_mbsdkevent_original%2Fdt%3D20210505%2Fpost_202105052129_33.log.gz
      FSCK started by hdfs (auth:KERBEROS_SSL) from /10.12.19.4 for path /hive_warehouse/warehouse_old_snapshots/yy_mbsdkevent_original/dt=20210505/post_202105052129_33.log.gz at Wed Oct 19 14:48:42 CST 2022
      /hive_warehouse/warehouse_old_snapshots/yy_mbsdkevent_original/dt=20210505/post_202105052129_33.log.gz 500582412 bytes, erasure-coded: policy=RS-3-2-1024k, 1 block(s):  OK
      0. BP-1822992414-10.12.65.48-1660893388633:blk_-9223372036799576592_4218617 len=500582412 Live_repl=5  
      [blk_-9223372036799576592:DatanodeInfoWithStorage[10.12.17.35:1019,DS-3ccebf8d-5f05-45b5-ac7f-96d1cfb48608,DISK], 
      blk_-9223372036799576591:DatanodeInfoWithStorage[10.12.65.218:1019,DS-4f8e3114-7566-4cf1-ad5a-e454c8ea8805,DISK], 
      blk_-9223372036799576590:DatanodeInfoWithStorage[10.12.15.149:1019,DS-1dd55c27-8f47-46a6-935b-1d9024ca9188,DISK], 
      blk_-9223372036799576589:DatanodeInfoWithStorage[10.12.65.244:1019,DS-a9ffd747-c427-4aaa-8559-04cded7d9d5f,DISK], 
      blk_-9223372036799576588:DatanodeInfoWithStorage[10.12.66.4:1019,DS-d88f94db-6db1-4753-a652-780d7cd7f081,DISK]]
      Status: HEALTHY
       Number of data-nodes:  62
       Number of racks:               19
       Total dirs:                    0
       Total symlinks:                0Replicated Blocks:
       Total size:    0 B
       Total files:   0
       Total blocks (validated):      0
       Minimally replicated blocks:   0
       Over-replicated blocks:        0
       Under-replicated blocks:       0
       Mis-replicated blocks:         0
       Default replication factor:    3
       Average block replication:     0.0
       Missing blocks:                0
       Corrupt blocks:                0
       Missing replicas:              0Erasure Coded Block Groups:
       Total size:    500582412 B
       Total files:   1
       Total block groups (validated):        1 (avg. block group size 500582412 B)
       Minimally erasure-coded block groups:  1 (100.0 %)
       Over-erasure-coded block groups:       0 (0.0 %)
       Under-erasure-coded block groups:      0 (0.0 %)
       Unsatisfactory placement block groups: 0 (0.0 %)
       Average block group size:      5.0
       Missing block groups:          0
       Corrupt block groups:          0
       Missing internal blocks:       0 (0.0 %)
      FSCK ended at Wed Oct 19 14:48:42 CST 2022 in 1 milliseconds
      The filesystem under path '/hive_warehouse/warehouse_old_snapshots/yy_mbsdkevent_original/dt=20210505/post_202105052129_33.log.gz' is HEALTHY
      
      
      
       
      ssh 10.12.15.149 
      ls -alh  /data*/hadoop/dfs/data/current/*/current/finalized/subdir*/subdir*/blk_-9223372036799576592 
      ls: cannot access '/data*/hadoop/dfs/data/current/*/current/finalized/subdir*/subdir*/blk_-9223372036799576592': No such file or directory
      
      ls -alh  /data*/hadoop/dfs/data/current/*/current/finalized/subdir*/subdir*/blk_-9223372036799576590
       -rw-r--r-- 1 hdfs hdfs 159M Oct 18 20:02 /data7/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir10/subdir5/blk_-9223372036799576590 

      Should move blk_-9223372036799576590

      Attachments

        Activity

          People

            Unassigned Unassigned
            ruilaing ruiliang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: