[HDFS-16806] ec data balancer block blk_id The index error ,Data cannot be moved

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Component/s: hdfs
• Hadoop Flags: Reviewed
• Assignee: Unassigned
• Reporter: ruiliang
• Created: 19/Oct/22 07:02
• Updated: 20/Oct/22 06:14
• Resolved: 20/Oct/22 06:14

Description

ec data balancer block blk_id The index error ,Data cannot be moved

dn->10.12.15.149 use disk 100%

 

echo 10.12.15.149>sorucehost
balancer  -fs hdfs://xxcluster06  -threshold 10 -source -f sorucehost   2>>~/balancer.log &  

 

datanode logs 
A lot of this log output  

datanode logs
...
2022-10-19 14:43:02,031 ERROR datanode.DataNode (DataXceiver.java:run(321)) - fs-hiido-dn-12-15-149.xx.com:1019:DataXceiver error processing COPY_BLOCK operation  src: /10.12.65.216:58214 dst: /10.12.15.149:1019
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-1822992414-10.12.65.48-1660893388633:blk_-9223372036799576592_4218617
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.getReplica(BlockSender.java:492)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:256)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.copyBlock(DataXceiver.java:1089)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opCopyBlock(Receiver.java:291)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:113)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290)
        at java.lang.Thread.run(Thread.java:748)
...    
    
hdfs fsck -fs hdfs://xxcluster06 -blockId blk_-9223372036799576592 
Connecting to namenode via http://fs-hiido-xxcluster06-yynn2.xx.com:50070/fsck?ugi=hdfs&blockId=blk_-9223372036799576592+&path=%2F
FSCK started by hdfs (auth:KERBEROS_SSL) from /10.12.19.4 at Wed Oct 19 14:47:15 CST 2022Block Id: blk_-9223372036799576592
Block belongs to: /hive_warehouse/warehouse_old_snapshots/yy_mbsdkevent_original/dt=20210505/post_202105052129_33.log.gz
No. of Expected Replica: 5
No. of live Replica: 5
No. of excess Replica: 0
No. of stale Replica: 5
No. of decommissioned Replica: 0
No. of decommissioning Replica: 0
No. of corrupted Replica: 0
Block replica on datanode/rack: fs-hiido-dn-12-66-4.xx.com/4F08-01-09 is HEALTHY
Block replica on datanode/rack: fs-hiido-dn-12-65-244.xx.com/4F08-01-08 is HEALTHY
Block replica on datanode/rack: fs-hiido-dn-12-15-149.xx.com/4F08-05-13 is HEALTHY
Block replica on datanode/rack: fs-hiido-dn-12-65-218.xx.com/4F08-12-04 is HEALTHY
Block replica on datanode/rack: fs-hiido-dn-12-17-35.xx.com/4F08-03-03 is HEALTHY



hdfs fsck -fs hdfs://xxcluster06 /hive_warehouse/warehouse_old_snapshots/yy_mbsdkevent_original/dt=20210505/post_202105052129_33.log.gz -files -blocks -locations
Connecting to namenode via http://xx.com:50070/fsck?ugi=hdfs&files=1&blocks=1&locations=1&path=%2Fhive_warehouse%2Fwarehouse_old_snapshots%2Fyy_mbsdkevent_original%2Fdt%3D20210505%2Fpost_202105052129_33.log.gz
FSCK started by hdfs (auth:KERBEROS_SSL) from /10.12.19.4 for path /hive_warehouse/warehouse_old_snapshots/yy_mbsdkevent_original/dt=20210505/post_202105052129_33.log.gz at Wed Oct 19 14:48:42 CST 2022
/hive_warehouse/warehouse_old_snapshots/yy_mbsdkevent_original/dt=20210505/post_202105052129_33.log.gz 500582412 bytes, erasure-coded: policy=RS-3-2-1024k, 1 block(s):  OK
0. BP-1822992414-10.12.65.48-1660893388633:blk_-9223372036799576592_4218617 len=500582412 Live_repl=5  
[blk_-9223372036799576592:DatanodeInfoWithStorage[10.12.17.35:1019,DS-3ccebf8d-5f05-45b5-ac7f-96d1cfb48608,DISK], 
blk_-9223372036799576591:DatanodeInfoWithStorage[10.12.65.218:1019,DS-4f8e3114-7566-4cf1-ad5a-e454c8ea8805,DISK], 
blk_-9223372036799576590:DatanodeInfoWithStorage[10.12.15.149:1019,DS-1dd55c27-8f47-46a6-935b-1d9024ca9188,DISK], 
blk_-9223372036799576589:DatanodeInfoWithStorage[10.12.65.244:1019,DS-a9ffd747-c427-4aaa-8559-04cded7d9d5f,DISK], 
blk_-9223372036799576588:DatanodeInfoWithStorage[10.12.66.4:1019,DS-d88f94db-6db1-4753-a652-780d7cd7f081,DISK]]
Status: HEALTHY
 Number of data-nodes:  62
 Number of racks:               19
 Total dirs:                    0
 Total symlinks:                0Replicated Blocks:
 Total size:    0 B
 Total files:   0
 Total blocks (validated):      0
 Minimally replicated blocks:   0
 Over-replicated blocks:        0
 Under-replicated blocks:       0
 Mis-replicated blocks:         0
 Default replication factor:    3
 Average block replication:     0.0
 Missing blocks:                0
 Corrupt blocks:                0
 Missing replicas:              0Erasure Coded Block Groups:
 Total size:    500582412 B
 Total files:   1
 Total block groups (validated):        1 (avg. block group size 500582412 B)
 Minimally erasure-coded block groups:  1 (100.0 %)
 Over-erasure-coded block groups:       0 (0.0 %)
 Under-erasure-coded block groups:      0 (0.0 %)
 Unsatisfactory placement block groups: 0 (0.0 %)
 Average block group size:      5.0
 Missing block groups:          0
 Corrupt block groups:          0
 Missing internal blocks:       0 (0.0 %)
FSCK ended at Wed Oct 19 14:48:42 CST 2022 in 1 milliseconds
The filesystem under path '/hive_warehouse/warehouse_old_snapshots/yy_mbsdkevent_original/dt=20210505/post_202105052129_33.log.gz' is HEALTHY



 

ssh 10.12.15.149 
ls -alh  /data*/hadoop/dfs/data/current/*/current/finalized/subdir*/subdir*/blk_-9223372036799576592 
ls: cannot access '/data*/hadoop/dfs/data/current/*/current/finalized/subdir*/subdir*/blk_-9223372036799576592': No such file or directory

ls -alh  /data*/hadoop/dfs/data/current/*/current/finalized/subdir*/subdir*/blk_-9223372036799576590
 -rw-r--r-- 1 hdfs hdfs 159M Oct 18 20:02 /data7/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir10/subdir5/blk_-9223372036799576590 

Should move blk_-9223372036799576590