[HDFS-17547] debug verifyEC check error

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: common
• Assignee: Unassigned
• Reporter: ruiliang
• Created: 07/Jun/24 08:00
• Updated: 16/Oct/24 18:32

Description

When I validate a block that has been corrupted many times, does it appear normal?

 

hdfs  debug verifyEC  -file /file.orc
24/06/07 15:40:29 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
Checking EC block group: blk_-9223372036492703744
Status: OK

 

 

ByteBuffer hb show [0......] [0......]

buffers = {ByteBuffer[5]@3270} 
 0 = {HeapByteBuffer@3430} "java.nio.HeapByteBuffer[pos=65536 lim=65536 cap=65536]"
 1 = {HeapByteBuffer@3434} "java.nio.HeapByteBuffer[pos=65536 lim=65536 cap=65536]"
 2 = {HeapByteBuffer@3438} "java.nio.HeapByteBuffer[pos=65536 lim=65536 cap=65536]"
 3 = {HeapByteBuffer@3504} "java.nio.HeapByteBuffer[pos=0 lim=65536 cap=65536]"
  hb = {byte[65536]@3511} [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, +65,436 more]

buffers[this.dataBlkNum + ixx].equals(outputs[ixx] =true ?

outputs = {ByteBuffer[2]@3271} 
 0 = {HeapByteBuffer@3455} "java.nio.HeapByteBuffer[pos=0 lim=65536 cap=65536]"
  hb = {byte[65536]@3459} [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, +65,436 more]

Can this situation be judged as an anomaly?

 

check orc file

Structure for skip_ip/_skip_file File Version: 0.12 with ORC_517 by ORC Java  Exception in thread "main" java.io.IOException: Problem opening stripe 0 footer in skip_ip/_skip_file.         at org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:360)         at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:879)         at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:873)         at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:345)         at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:276)         at org.apache.orc.tools.FileDump.main(FileDump.java:137)         at org.apache.orc.tools.Driver.main(Driver.java:124) Caused by: java.lang.IllegalArgumentException: Buffer size too small. size = 131072 needed = 7752508 in column 3 kind LENGTH         at org.apache.orc.impl.InStream$CompressedStream.readHeader(InStream.java:481)         at org.apache.orc.impl.InStream$CompressedStream.ensureUncompressed(InStream.java:528)         at org.apache.orc.impl.InStream$CompressedStream.read(InStream.java:507)         at org.apache.orc.impl.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:59)         at org.apache.orc.impl.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:333)         at org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.readDictionaryLengthStream(TreeReaderFactory.java:2221)         at org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.startStripe(TreeReaderFactory.java:2201)         at org.apache.orc.impl.TreeReaderFactory$StringTreeReader.startStripe(TreeReaderFactory.java:1943)         at org.apache.orc.impl.reader.tree.StructBatchReader.startStripe(StructBatchReader.java:112)         at org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1251)         at org.apache.orc.impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1290)         at org.apache.orc.impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1333)         at org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:355)         ... 6 more